Static Allocation, Constant Work

In reply to this email:

Yes, object pools are an interesting case to think about, as they clarify the relation between memory safety and more general correctness.

First, consider the case where no object pool is used, and we malloc and free order objects. In this case, the logical error of use-after-free turns into physical type confusion, and can easily lead to arbitrary code execution and the like. If you have two objects of different types sharing the same memory location, a user-controlled integer in one object might be a function pointer in the other: an exploitable goto primitive

Now, what happens if we introduce an object pool which stores a list of “dead” objects of type T? Logical use-after-free is still possible, but its physical effect is now different — we still get aliasing of memory, but there’s no type confusion. You can’t necessarily fiddle with an integer and change a function pointer, unless you additionally hit the hard case, where the object in question stores an inline enum. Assuming that doesn’t hapen, you get a perfectly defined, deterministic behavior, even if you are not happy about the result.

This suggests an interesting solution for hardening code, which I’ve learned from Fil. If your allocation function is typed (it takes a T comptime parameter or runtime type witness, rather than a runtime type-erased size and alignment), you can write an allocator that uses type-segregated pools internally. This will be somewhat less memory efficient, as the allocator won’t be able to re-use freed memory of objects of type U for objects of type T, but the memory overhead will probably be small (rare object types do not matter, popular object types will have a lot of intra-type re-use), you might actually gain in memory locality, and solve most of type confusions. Again, inline enums break this, but, curiously, if you always heap allocate enum variants, then this works again. Fil-C can’t use this, because C allocator’s interface is untyped, but someone else could :P

But this is academic. How do we avoid the bugs? Generational indexes are a popular remedy, but I have never used them, so I don’t have any non-common knowledge insights about this pattern. Instead, I will share another pair of tricks from TigerStyle. I have only a vague understanding of what an order matching engine is, but I suspect these tricks might help there

$ order-engine --orders-max=1_000_000

const orders: []Order = try gpa.alloc(Order, cli_args.orders_max);

const OrderPool = struct {
    orders: []Order,
    free: DynamicBitSet,

    fn acquire(pool: *OrderPool) ?*Order { ... }
    fn release(pool: *OrderPool, order: *Order) { ... }
};

const OrderPool = struct {
    orders: []union {
        order: Order,
        next_free: ?u32,
    },
    first_free: ?u32,
};

const Order = {
    id: u128,
    price: u32,
    count: u32,

    tag: enum { bid, ask, reserved },

    pub const reserved: Order = .{
        .id = 0,
        .price = 0,
        .count = 0,
        .tag = .reserved,
    };
};

for (orders) |order| {
    process(order)
}

for (orders_active) |order_index| {
    const order = orders[order_index];
    process(order);
}

const item = for (items) |item| {
    if (predicate(item)) break item;
} else null;

Cancelation Terminology

A short note explaining the difference between synchronous cancelation, asynchronous cancelation, and graceful shutdown. I am not too attached to these specific three terms, but I want to call your attention to the three things behind them, which are important not to confuse with each other.

synchronous cancelation is an (often implicit) control flow structure. It unwinds the stack and looks like this:

task.cancel();
// The task will have finished by this point.

Synchronous cancelation is a bit like Molière’s prose — we do it all the time, but not necessarily in full consciousness. The primary source of synchronous cancelation is error handling — every time an Exception is thrown or an error returned, the code promptly breaks out of all the loops, ifs, and blocks, invoking the necessary cleanup actions via RAII, finally, with/try with resources or defer.

asynchronous cancelation is a communication protocol between two parties. One party requests cancelation (synchronously), but then it has to wait until the other party acknowledges it and winds down. It looks like this:

task.request_cancelation();
// The task could still be running here.
task.join().await;
// After the requisite wait, the task is finished.

Like synchronous cancelation, this is a relatively low-level concern when implementing a concurrent program in a way that doesn’t crash or hang. I know two central example where an asynchronous cancelation is required.

First is the CPU thread pool. Imagine you have offloaded encrypting a buffer to a separate thread as a part of handling user’s request. Some time later, you learn that the request must be canceled (perhaps the user had left). You can’t just abandon the encrypting thread. First, it would be smart not to waste CPU cycles for useless work, but, more importantly, the underlying buffer must remain tied up. If it were to be freed as a result of request cancelation, something else might re-use that memory, leading to data races.

But you also can’t just cancel that thread synchronously! It’s in the middle of a hyper-optimized SIMD loop, and you really don’t want it to check the cancelation flag before reading every byte. What you’d want is to split the buffer into reasonably-sized chunks, and check the cancelation status after every chunk. But that means that the party that requested the cancelation must wait for at least one chunk’s worth of work!

Another example here is io_uring. It has exactly the same shape: if you submit a write with a buffer to the kernel, that buffer must remain tied up until the write finishes (and you can cancel the write to make it finish faster). While io_uring is still at least a somewhat exotic technology (though, arguably, it’s the interfaces we have had before which are byzantine), the thread pool example demonstrates that the phenomenon of asynchronous cancelation itself is rather mundane.

Asynchronous cancelation comes up all the time when writing concurrent software. Because it affects the overall shape of the code, it’s useful to identify it early. Conversely, it is useful to ask yourself whether you need asynchronous cancelation at all, and whether synchronous one can be made to work. This is especially important in Rust, which makes synchronous cancelation too easy, and doesn’t provide great mechanisms for asynchronous one.

Finally, graceful shutdown is an application programming pattern for handling connections. It lives on a higher level of abstraction than the two cancelations. If you are implementing a web service, you can implement shutdown by stopping your accept loop (rejecting new connections), but continuing to serve all existing connections until their respective clients disconnect. If the load balancer is configured to route new connection requests to different instances of the service, this pattern allows you to do rolling upgrades without service disruptions.

As a bonus point, a related idea is that of crash-only software. Cancelation is all good, but your entire program can get SIGKILLed arbitrarily by an OOM killer, and the entire computer might get rebooted on powerloss. Reliable software has to handle ungraceful shutdown without losing data. But, if you can survive powerloss, you might as well implement the Quit button by SIGKILLing yourself, simultaneously simplifying the implementation and increasing testing coverage for powerloss scenarios.

To give some examples from TigerBeetle, Grid.cancel is an asynchronous cancelation. It takes a callback to notify the caller when the cancelation is done. This API is used during state sync. When a replica determines that that cluster is so far ahead that event based transfer doesn’t work, and that a state transfer is required to catch up, it must cancel all outstanding grid read operations. A read can be backed either by replica’s local disk, or by transparent fetch of the data from a neighboring replica. In the first case, we have to wait until the read is done. In the second case, we need to abandon the read — remote read getting stuck is probably the reason for us to state sync in the first place.

StateMachine.reset is an example of a synchronous cancelation. This is the part of the same flow as Grid.cancel, and is an example of how you can simplify the code if you think clearly about asynchronous vs synchronous cancelation. Ultimately, StateMachine sits on top of the Grid, but there’s a bunch of intermediate layers (Forest, Tree, Compaction, Scan, etc). A naive approach would be to notice that Grid requires asynchronous cancelation and propagate asynchrony throughout the stack. What we do instead is asynchronously canceling just the Grid directly, and then synchronously reseting everything else.

Another example of asynchronous cancelation is Client.shutdown. When an application using TigerBeetle “drops” the Client object, we need to free all OS resources. Our client also uses io_uring, so we must first wait for all outstanding syscalls to complete. In the comment, we call it “graceful shutdown”, but I think this is wrong, and this is the motivation for writing down this article. We don’t do graceful shutdown at TigerBeetle — it’s crash only all the way. Tail latency tolerance (asking several nodes for an answer and picking the fastest one) is a more general solution, as it handles not only crash faults, but also gray failures. In a distributed system, a very slow node looks exactly the same as a crashed one. A crash is just a degree of slowness.

Take aways:

• Synchronous cancelation is control flow operator
• Asynchronous cancelation is a communication protocol
• Graceful shutdown is an application-level design pattern

Rust Glancer

Rust Glancer, a functional LSP server for Rust which uses two orders of magnitude less RAM, is incredibly cool. Go check it out! This post started as a comment on lobste.rs, but I figured it out that it’s better to publish it somewhat more prominently. Don’t expect polished writing though!

Some thoughts:

Yeah, rowan is garbage :P I was really thinking about

• incremental parsing,
• incremental, DOM-mutation style refactorings,

And Rowan is pretty good for that. But that’s 1% use case. The 99% use case is all the code in your 6666 dependencies which you won’t ever look at, but which needs to be at least shallowly analyzed. Even for incremental tool whose main goal is refactoring, the primary AST structure should be just a list of arrays. There might be a real post about that at some point, see https://youtu.be/G93oYL1ry70 as a teaser.

If I understand correctly, Rust Glancer wants to process each function body. I think that part can perhaps be made lazy (but not incremental!) with little overhead? Index all items, but, for functions, do only the currently opened file? This might combine some of the better parts of both worlds.

Would be interesting to compare memory usage with Rust Rover. Net of the IDE GUI itself, I would expect RR to be more compact.

I might be rationalizing/misremembering things, but IIRC it’s exactly around adding proc macros that the thing began to feel unreasonably bulky. Expanding proc macros is slow as we are running real code, we can’t really do normal IDE cheats. And proc macros generate a lot of code. At one point I measured, it was like 30% of rust-analyzer binary size was attributed to JSON parsing code. If no one sees the code, it can’t harm anybody, right?

One potential approach here is to pull the Sorbet trick, where you don’t run meta programming at all, and instead have a plugin interface to “explain” the effects of what that would have done. Instead of running serde, we just add a shim that injects imp Serialize for T {} with an empty body.

Rust analyzer’s core data model is very pedantic about always observing consistent snapshots of the code, and does its best to ensure that the language client and server have a shared, strictly serializable view of the world. It’s a shame that LSP doesn’t allow that to be correct, only heuristically right, unlike the older Dart Analyzer protocol, which has sound data synchronization.

However our implementation of file watching is sketchy! First, there are two backends: we can ask the editor to do watching for us, or we can use server side watching. Try changing this option and see if it helps? But then, yeah, my recollection is that our native watcher’s API was fundamentally racy, and I didn’t do the messy platform-specific work of making it correct.

But the main thing I want to write, and why I moved from the cozy lobste.rs text area to the luxurious comforts of an Emacs buffer, is that right now rust-analyzer is a bit like that half-drawn horse meme, except that it’s only the head half of the horse.

One Big Idea of IntelliJ is that its PSI API (essentially AST with resolved types) is really an interface, and there are multiple providers. And in a typical usage, there’re at least three backends in play:

• For the files opened in the editor, actively modified by the user, the PSI is backed by the concrete syntax trees.
• For the rest of the project files, the PSI is backed by the so called Stub Tree, a compact on disk representation storing only the “externally visible” parts of the file (so, without function bodies). If the user navigates to a new file, its PSI transparently switches from stubs to syntax tree.
• For dependencies, the PSI is often backed by the compiled .class files, produced by javac. If you navigate there, the IDE just decompiles stuff for you! Super cool!

This is how I think such things should work. rust analyzer shouldn’t use salsa for all those 6666 dependencies you still haven’t looked at. It should just use rustc’s .rmeta files, switching to salsa, transparently, only when the user starts messing around their ~/.cargo/registry/src folder.

The prerequisite for that is defining the abstract API for accessing Rust code. That was always the plan, and we did start on that at some point:

https://hackmd.io/ytd82QNiT_Ku2XFr1EAtiQ

But I don’t think that work was ever completed.

This still seems to me to be the lowest-hanging watermelon here — split the world into arcy-pointy incremental tip of the iceberg, and mostly read-only, on disk, compact, dark, moist breeding ground for supply chain attacks.

Such glance analyzer architecture would be great, imo!

Better Batteries

One of the eternal schisms in programming is over the question of whether the standard library should be minimal or encompassing. This is the wrong question to ask. The right one is:

Python is always brought up as example of leaky batteries exploding in slow motion, but this has nothing to do with size. The problem with Python’s stdlib is its, ahem, uneven quality. Some standard library modules don’t follow language naming conventions! You know which unittest module I am talking about :-)

But even that is not a mistake. It’s actually Python core’s advantage — that it makes functionality available early, not thinking about the future too much. That’s how we ended up with ossified cAPI which makes CPython the language, but that is also how we ended up with Python powering data scientific revolution.

The Go standard library is similarly encompassing, but it is held in a high regard. Go team has institutional capacity to deliver well-designed API for the standard library, and then some: https://pkg.go.dev/golang.org/x

Rust is an interesting case. The 1.0 standard library APIs are brilliant. Collections and iterators are a work of art. But it also feels that, while the current team has the capacity to preserve existing APIs and fill in some gaps, the capacity to execute design decisions is limited. While golang.org/x captures excess capacity, rust-lang-nursery is a graveyard. Maybe I am over-indexing on my favorite hobby-horse, but it seems that the reason for Rust not having an API to get a stream of random bytes from the OS in 2026 is that, while it is an easy technical problem, it requires tricky organization architecture (including getting money in peoples’ pockets, of course) to actually get solved in the high-stakes environment of a world-wide coordination problem called a programming language.

Printing Lists

To print a comma-separated list, a concise idiom is to optionally print the comma first, before the element:

for (items, 0..) |item, item_index| {
    if (item_index > 0) std.debug.print(", ");
    std.debug.print("{}", .{item});
}

Zig’s Io.Threaded is Neat

std.Io.Threaded is one of the implementations of Zig’s new Io interface that enables concurrency. This is a boring “just use threads” impl. I personally find it neat though — it does this weird thing that I wanted to do for ages, that to my knowledge no one else is doing properly, and implements it better than I thought to be possible.

Io.Threaded uses blocking syscalls and fully supports cancelation.

use rayon::prelude::*;
fn sum_of_squares(input: &[i32]) -> i32 {
    input.par_iter()
         .map(|i| i * i)
         .sum()
}

while (true) {
    if (is_canceled()) return error.Canceld; /// Easy!
    ...
}

const read_size = try read(fd, buffer); // ???

Memory Safety’s Hardest Problem

Uplifting a lobsters comment for easier reference.

The central memory safety counter example, the hardest case to solve, doesn’t have anything to do with destructors or heap:

const std = @import("std");

const E = union(enum) {
    a: u128,
    b: []const u8,
};

pub fn main() void {
    const bad_addr: u128 = @intFromPtr(&main);

    var e: E = .{ .b = "hello" };
    const oh_no_pointer: *const []const u8 = switch (e) {
        .a => unreachable,
        .b => |*p| p,
    };
    e = .{ .a = (16 << 64) + bad_addr };
    const oh_no: []const u8 = oh_no_pointer.*;
    std.debug.print("{s}\n", .{oh_no});
}

$ zig run main.zig
��C�� �

This sort of example also breaks Ada:

https://www.enyo.de/fw/notes/ada-type-safety.html

We have a tagged union, which can hold either A or B. We initialize the union as A, take a pointer to its internals, overwrite the original with B, and then use the pointer. The pointer is still typed as A, but the bytes it points to now belong to B: a type confusion.

This being said, we care about memory unsafety primarily because it leads to exploitable software, and it’s unclear just how impactful the example above is in practice. It is a happy coincidence that by far the most exploitable memory error in practice, the infamous buffer overflow, is also trivial to fix with compiler-inserted bounds checks. The biggest miss of the industry when it comes to memory safety is not listening to Walter Bright:

https://digitalmars.com/articles/C-biggest-mistake.html

I bet that, had we got char a[..] syntax around C11, quite a few issues wouldn’t have happened!

See also What is Memory Safety?

CSS: Unavoidable Bad Parts

An ersatz CSS tutorial for people who need to style a web page, but aren’t web developers. I am a wrong person to write this kind of thing, as I have neither the time, nor experience. I’d much rather read a book about this. Alas, I had to learn all this stuff from trawling MDN, so perhaps it is valuable to document what I have so far.

CSS, HTML and Web APIs are truly vast, and it takes a career to become a professional. The good news is that modern web has a reasonably-sized, learnable subset which is enough for simple tasks like a programming blog or a simple GUI. I haven’t seen a resource that teaches just this subset, but it’s not too hard to figure this out. The bad news is that there’s also a nasty set of gotchas, which will mess up your page, which you won’t suspect to exist, and which will need days of debugging to figure out. Still, it’s not that bad. I am quite happy with the styling on this site, and it’s only about 200 of readable CSS.

Good: HTML5 semantic tag names
It’s worth looking through MDN Elements Reference. There aren’t that many elements, and things like main, article, nav, kbd make it much easier to structure your page. Less obvious:

• ul for any kind of list, like site’s sections in header > nav.
• details for table-of-contents (check the source of MDN).
• dl/dt for list of pairs.

Bad: Wrappers
If you “View Source” on any “real” website, you’ll notice that everything has layers and layers of wrapper elements, so you might be tricked into thinking that wrappers are how you solve layout problems. I can’t really agree or disagree here, as I never wrote “production” CSS, but, in my experience, it’s much easier to understand if you do the opposite — restrict yourself to using only markup-meaningful semantic tags, and then figure out CSS which works with the markup you have.

Bad: Layout
This one is not an exclusively Web problem, layout is a struggle in every GUI framework I know. Imagine a fixed sized raster image, and a paragraph of text describing it. There are many ways to arrange these two elements on the screen’s rectangle. Generally, for every given width and height, you can do a decent job, as long as the total area is enough. A typical GUI is a hierarchy of such boxes, with a lot of “layout freedom”. The problem though is that layout of each box affects the layouts of all other boxes, as you generally want all boxes to meet exactly, without gaps and overlaps. An important negative realization is that the layout algorithm doesn’t exist. There isn’t a fully general solution to positioning and sizing GUI boxes. Rather, different systems use different sets of heuristics to do the job, from simple RectCut, to fully general constraint solvers, with everything in between. It is hard to get the mental model of how layout works, in general. So, don’t think “how can I do my layout in a given system”, think instead “what possible layouts are allowed by the system”.

Bad: Browser defaults
Let’s start with a bare (but still semantic) HTML markup of a blog article, without any CSS. If you open it in a browser, it will show something. The content isn’t unstyled — the text is of a certain color, font and size. Headers are bigger than the main text, links are underlined, etc. These are the default styles of your browser. They are helpful! The problem is that these styles differ between the browsers. So, even when you add your own CSS, and the end result looks fine in your browser, I might see something different, because you might rely on a browser default, without knowing it. The last bit is the killer here — the problem is in something you didn’t write.

The general solution here is a CSS reset, or normalization — starting your CSS with an explicit set of rules, overriding defaults. Not because defaults are inherently bad, because they are inconsistent. I don’t know which set of rules you need to override in practice, it’s a good idea to compare several existing CSS resets.

This touches on the big question: should you style your web page? There are two competing views of the Web platform — some people treat it as a flexible, adaptive, primarily visual medium for expressing design, others would prefer if the Web focused on delivering the content, allowing each user to customize the presentation. My personal answer here is pragmatic — by default, an unstyled page is poorly usable and looks bad. I would have preferred the world where CSS-less pages were readable as is, but, in this world, I think it is helpful to style the content. At the same time, it’s a good idea to allow advanced users to bring their own CSS. Make sure that your HTML markup is reasonable, that you don’t overfit your HTML to CSS (vice-versa is fine), and that your page functions in reader mode.

Good: Classless CSS
You can’t reset styles to true neutral nothing: if you make the text invisible (white or transparent), it is still a style. So you might as well embrace it: after reset, style common HTML elements directly. For example, to set your favorite font for all code snippets:

code { font-family: "JetBrains Mono", monospace; }

If you use main, header, footer, nav tags you can set the overall page layout without writing any CSS selectors. This of course requires making assumptions, in CSS, about the structure of your HTML, but, like, this is your HTML and your CSS, you can do whatever, and, if you don’t like the result, you can always change it!

Bad: CSS selectors
In programming, we collectively came around to distrust inheritance and prefer composition. Default CSS is like supercharged inheritance, each design element on your web page is affected by multiple rules, and you can always “monkey patch” existing elements by appending to your CSS. There’s an unfortunate gap between CSS affordances, and what you actually want to do. The two reasonable approaches are:

1. Conclude that CSS selectors add abstraction capability along the wrong axis, and stick to classless CSS and inline styles, using something like Tailwind to make writing inlines prettier, and something like JSX (or any other templating engine supporting composition) to avoid repetition in HTML.

2. Use CSS nesting to avoid writing “far reaching” selectors and style component-per-component:

header { /* Site Header */
    margin-bottom: 2rem;
    & nav {
        /* Styles, specific to nav in the Header. */
    }
}

Bad: box-sizing
UIs are recursive rectangles, layout is the process of figuring out where each rectangles goes, and it is determined by the sizes of rectangles themselves. So, understanding what is the size is quite fundamental. Sadly, by default the definition of size in HTML is very unintuitive: element’s width and height do not include element’s border and padding, which leads to surprising results: everything looks perfect at first, but increasing padding somewhere shifts the entire layout unexpectedly. For this reason, * { box-sizing: border-box; } deserves to be the first line in your CSS reset. It makes elements encapsulated, such that adding borders is a local-only change.

Chaotic Good: margin collapsing
Suppose you want to have a 8px gap around an element. You would think that you need to set the padding property. But that would be wrong — if you have two such elements next to each other, the gap between them would be 16px. The paddings would add, creating a visual gap larger than intended. You want something more akin to social distancing, where if one person is more introverted, this person’s bigger radius of exclusion is what defines the distance. And that’s how the margin property works. Two neighboring margins are combined using max rather than sum. Margin collapsing is very useful, but it can surprise you. E.g. I think child margin can stick beyond parent’s? To be honest, I don’t have a good intuitive understanding of margins, but I know enough to at least identify when it is the problem.

Margins are also one of the indirect inspirations for this post. In

Moving away from Tailwind, and learning to structure my CSS

Julia Evans writes that you generally don’t want to set margin on an element, and should rather let the parent control the inter-element margin of the children, using the so-called owl selector:

section > *+* {
  margin-top: 1rem;
}

That is, add margin to all section’s children exempting the first one. I didn’t know that! And, given all the pain that margin gave me so far, I actually get why you want to do this, and why this is a good idea. But it bugs me that you can’t learn that without becoming “professional” web developer, or reverse-engineering someone else’s CSS framework.

Bad: Default (flow) layout
Layout in general is tricky, because there’s no universal “layout algorithm”, just a bunch of special cases. But what does HTML actually do? The default layout algorithm I think goes back to the origin of HTML as a language for documents, and overfits a use-case of producing papers — mostly text content with some illustrations, where the text can flow around the pictures. That’s actually what you want for the main body of text of your blog, but, as soon as you want to actually control the spatial arrangement of the elements on your page, you want something different, for example…

Good: flexbox
This is really what separates modern web-development from the olden days, where you’d need a CSS PhD or a full-blown opaque CSS framework to be able to say “this goes to the left, and this goes to the right”. This layout allows you to arrange a series of elements either vertically or horizontally, adapting to the available space. It is rather complex and I can’t use flexbox without referencing MDN all the time, but usually I am able to get things done in the end.

Bad: responsive design
Modern CSS allows querying screen size, and implementing conditional logic based on that — a design that “responds” to user-agent constraints. This probably what you should use for “real” CSS, but note that HTML is inherently responsive. Unlike PostScript (PDF), it will automatically reflow the paragraphs when you change window size. So, it’s a good idea to avoid writing explicit responsive rules, and just rely on layout to do the reasonable thing. For example, this blog looks OK on mobile, tablet and desktop without any explicit @media queries. Unconditionally setting max-width on the main column of text is all that it takes.

Lawful Evil: pixels
1px does what you want, but not what it says. It’s not a size of one physical pixel on your screen. Rather, it’s a measure of visual angle. That is, 1px should look perceptually the same on any screen, and it is converted to different number of physical pixels, depending on the screen size, its pixel density, and the typical viewing distance. So you can just size everything in pixels, without thinking about different displays’ pixel densities. It gets weirder. CSS allows “real” units like centimeters or inches, but they are also angles, because everything is defined in terms of pixels.

Doubleplusungood: font-size
Flexbox is a good way to layout UI-elements. Flow layout works ok for laying out paragraphs of text. But what happens on the level of individual lines and glyphs is, in my opinion, a train wreck and a noob trap. Let’s start with the basics: if you write font-size: 16px then 16px is the size of what? Sadly, the answer is “nothing in particular” — this is a size of a virtual box around the glyph, but the box isn’t tight, and the size of the glyph varies, depending on the font. Luckily, font-size-adjust property can fix it, and make font-size consistent across fonts. See these two posts for details:

• font-size-adjust Is Useful
• Font size is useless; let’s fix it

Though, at the moment font-size-adjust seems to be very niche, so, while personally I’d put font-size-adjust: ex-height 0.53; right next to box-sizing, few pages do that.

The next issue with font-size is a thorny question of defaults. The good news is that it’s one of the properties that is fairly consistent across browsers, with 16px being the overwhelming default. The bad news is that, depending on the font, 16px can be on the smaller size. Not completely illegible, but very close to the lower bound. What’s worse, some default fonts are particularly small. For example, on Apple, font-family: serif looks much smaller than sans-serif, and is almost uncomfortable to read at 16px.

Can you just set font-size: 18px or whatever works best for your chosen font? I think the answer is yes, but there are some caveats to keep in mind. Refer to Accessibility: px or rem? for details. The issue is that modern browsers support two ways of making text on a page bigger:

• Zoom, which has a dedicated UI element, shortcuts/gestures, per-page persistence/overrides and a global default.
• Changing default font-size, a global setting buried deeply in the configuration page.

Setting font-size in your CSS disables that second approach.

Taking everything together: don’t assume that text on your page will be readable by default, check different configurations. Set font-size-adjust to reduce the number of degrees of freedom and to pin down the meaning of font-size. If the result looks fine with your chosen (or your user’s default) font and default font-size of 16px, then you are done. Otherwise, set font-size to a bigger number. Afterwards, check that the page is readable in reader mode as well.

Bad: line-height
Despite the name, line-height doesn’t set the height of a line. It is a height of a run of glyphs, set in the same font. The two coincide when all the text is in the same font. But if you have, e.g., some words set in monospace font, you are in for a surprise. While font-size-adjust fixes the size of a glyph inside the box, it still leaves its relative position unspecified. So, when two runs of text in different fonts are aligned vertically to share the baseline, their line-height line-boxes get shifted relative to each other: one sticks below, one sticks above. The line height overall becomes larger that what you’d expect, as it is configured as a union. See

Deep dive CSS: font metrics, line-height and vertical-align

for a thorough explanation of this effect.

Bad: vertical rhythm
If you google long enough this cluster of problems, sooner or later you’ll come across the idea of vertical rhythm, that you should make sure that lines are in the same relative position across different paragraphs, even if you have headings, images, and what not. As if there’s invisible lined paper behind your web-page. As far as I can tell, this is pure voodoo and is not useful. If you do two-column layout, then you want lines on opposite sides to align, but it makes no sense to jump through hoops for a single-column layout (hat tip to @chrismorgan).

Bad: word-break
The genius of the flow layout is its dynamism. It takes a moment of reflection to appreciate the technical marvel of text breaking itself neatly into lines as the window is resized to be narrower. Getting that to work for the first time ever in the world of durably printed text must have felt incredible. But the magic has its limits — you can only break the line at the whitespace, or at the hyphenation points. And some long spans, like inline code or URLs, might be unbreakable. This leads to overflow annoyance on mobile devices, something you notice only after you publish your work. There’s no one trick to fix it, but some tips are available here: Against Horizontal Scroll for details.

And … that’s all I remember so far? I reiterate my request for someone to write a short 100-page book explaining just enough of HTML&CSS to make a simple blog without getting collapsed by the margins!

TIL: Symlinking NixOS Dotfiles

The standard answer to managing dotfiles on NixOS is home-manager. I’ve never used it, due to two aesthetic and one practical objection:

• I avoid dependencies, especially in nix, which rivals Python in the number of approaches to dependency management.
• home-manager installs packages for the current user only, which makes sense on non-NixOS systems. But on a single-user desktop system, I prefer having just one set of packages.
• Having a source of truth for dotfiles be in nix store requires rebuilding your system to change config, which gets in the way of Emacs-style direct tinkering.

The approach I like is storing dotfiles in the same repository as flake.nix / configuration.nix and symlinking them in place.

The problem here is that NixOS seemingly doesn’t have a “native” way to say that /a/b/c should be a symlink to /c/d/e. Or has it?

If you search options for symlink, you’ll learn about environment.etc which allows you to configure symlinks, but only for things in /etc, not your ~/.config.

For the latter, you can use gnu stow or some other dotfile link manager, but the complexity of the problem of just managing symlinks doesn’t warrant yet another dependency. It’s fine to do this manually.

But wouldn’t it be nice if this framework for declarative configuration of your system allowed you to declaratively configure symlinks? Turns out this is possible, in roundabout way. Inaptly-named systemd-tmpfiles allows creating symlinks from a declarative config, and you can use NixOS to configure systemd-tmpfiles itself (thanks, NobbZ!).

For example, if I want to symlink ~/dotfiles/git/config to .config/git/config:

{
  systemd.tmpfiles.rules = [
    "L+ /home/matklad/.config/git/config - - - - /home/matklad/dotfiles/git/config"
  ];
}

No opinion at this point how this compares to a bespoke script or something more purpose-built.

Always Be Blaming

A few tips on 4D-ing your code comprehension skills.

I wrote on the importance of reading code before: Look Out For Bugs My default approach to reading is “predictive”: I don’t actually read the code line by line. Rather, I try to understand the problem that it wants to solve, then imagine my own solution, and read the “diff” between what I have in my mind and what I see in the editor. Non-empty “diff” signifies either a bug in my understanding, or an opportunity to improve the code.

This is 2D reading, understanding a snapshot of code, frozen in time. This is usually enough to spot “this feels odd” anomalies, worthy of further investigation.

Ideal code is memoryless — it precisely solves the problem at hand. Most real code is Markov — the shape of the code at time T depends not only on the problem statement, but also on the shape of the code at time T - 1. The 3D step is to trace the evolution of code over time, Where Do We Come From? What Are We? Where Are We Going?.

The step after that is to understand the why. What were we thinking back then, when we wrote this code? It’s useful to have the “theory of mind” concept ready here. I personally learned the term way too late in my life, so let me give a short intro for today’s lucky 10 000. Theory of mind is the ability to imagine yourself in someone else’s skin. Not just in their shoes (“I certainly would have acted differently in that situation”), but with their mind (“I wouldn’t have acted that way, but I get why they did”). This is something people learn. The experimental setup here is to have a child in a room with toys, with a doll sitting near the opposite end of the room, and asking the child “what does the doll see?”. Younger children describe the room from their perspective, older begin to intuit that doll’s perspective is different.

So this is the goal of reading code — understanding what the original author was thinking, and why.

End of the mumbo-jumbo, some practical advice. First, read Every line of code is always documented, it is very good.

Second, make sure it is effortless for you to find out how a given snippet of code evolved. This is harder than it seems! Just git blame isn’t an answer — mind the gap between the problem that’s easy to solve, and the problem in need of solving.

git blame answers spatial question of “how each line appeared in this file”, because there’s a relatively straightforward UI for this — annotate each line with a commit hash. But this is not the question you are asking most of the time! You don’t care about the file! There’s a small snippet of code in the middle, and you want a temporal history of that.

As much as I don’t like working in the browser GitHub’s web interface for blaming is probably better than what you get locally by default. It starts with the y shortcut, which resolves a symbolic reference like

https://github.com/tigerbeetle/tigerbeetle/blob/main/src/vsr/replica.zig

into the one which has a commit hash in the URL:

https://github.com/tigerbeetle/tigerbeetle/blob/c54f613a2eb2a127a0ba212704e3fa988c42e5cb/src/vsr/replica.zig

This commit hash is critical, because it anchors the entire repository — if you open a different file from the web UI, it will be shown as of that commit. This enables you to not myopically focus on just the diff in question, but to absorb the entire context at that point in time.

So my usual web workflow is:

• ctrl+f to find the line I am interested in
• b to toggle blame
• Click “blame prior to change” a couple of times, repeating ctrl+f to go back to the snippet I am curious about.
• cmd-click on the commits that are potentially relevant, pinning their commit hashes in the URL in new tabs.
• Then, from the commit page, “Browse files” button to then go and t to other files. Or, cmd+l to focus browser’s address bar, and s/commit/tree/ (or back!) as needed, to switch between diff and snapshot views.

Again, my goal here is not to annotate a diff on a file but rather to get a “virtual checkout” as of the interesting commit.

This web approach is what I was using throughout most of my career, but I’ve finally found a way to replicate it locally. The idea is to make blaming “in-place”. Instead of git blame annotating lines of code, I directly switch to a historical commit. I have the following devil hydra of shortcuts:

, b l blames line. It notes the $line the cursor is at, runs git blame -L $line,$line to find $commit that introduced the line, and then runs git switch --detach $commit to check it out. I have a dedicated worktree for code archeology, so I don’t worry about trashing my work. There’s also a half-hearted attempt to maintain “logical” cursor position, but it doesn’t work very well. Is there some git command that tells me directly “what’s the equivalent of $file:$line:column in $sha-A for $sha-B?”

, b p blames parent. Which is just switching to the parent commit of the current HEAD, what “blame before this change” does on GitHub (it works slightly differently because it assumes that , b l was the previous command)

, b u undoes the last blaming operation, switching to the previous point. I really love that, on the web, I can cmd-click to create an alternative branch of exploration. In theory, this is replicatable locally, but I prefer to destructively mutate a single working tree on disk. A big reason for preferring in-place blame is that LSP, ./zig/zig build test, rg and the like just work. That’s more important for me than the garden of forking paths, and undo is an acceptable work-around.

Finally, , b w copies GitHub link to the current commit and line, which I can paste into the browser. An enormous problem with modern version control landscape is that absolutely critical information in the form of code review comments is not a part of the git repository, and is locked in someone else’s proprietary database. I failed to solve this problem in one weekend, and had to begrudgingly adapt. Opening the commit in a browser links you to the PR and its discussion as well.

Implementing this blame workflow required a bit of custom code. Feel free to use it, but beware that it’s somewhat crufty, especially around maintaining current cursor position. Making a production-ready version of this sounds like a fun project ;-)