joshlf.com

Netstack.FM Interview

2026-08-17T00:00:00+00:00

This is an edited transcript of my October 21, 2025 interview on episode 10 of the Netstack.FM podcast, "Zerocopy with Joshua Liebow-Feeser", hosted by Glen De Cauwsemaecker and produced by Plabayo BV. Published here with permission. The transcript has been lightly edited for clarity.

In this conversation, Glen and I discuss the origins and design of Netstack3 and zerocopy, along with zero-copy parsing, formal verification, and safe transmutation. We also discuss the broader philosophy behind my work: applying a mathematical standard of precision to systems engineering, distinguishing a fuzzy and intuitive understanding of a system from one strong enough to justify confidence, and using that deeper understanding to hide complexity behind simple, reliable abstractions.

Elizabeth (Plabayo)

This is netstack.fm, your weekly podcast about networking, Rust, and everything in between. You are listening to episode 10, recorded on the 14th of October, 2025, where Glen has a conversation with Joshua, a software engineer at Google, where he drives adoption of zerocopy and similar design patterns forward.

Glen (Plabayo)

Hello, welcome to netstack.fm. This week I will have a conversation with Joshua. He's one of the original developers behind Netstack3 and he's also the creator of zerocopy, which is a crate that makes zero-cost memory manipulation effortless. It was originally built for Fuchsia, but zerocopy is now used by over 400 crates on crates.io with nearly 300 million downloads. So welcome, Joshua.

Josh

Yeah, thanks so much for having me. I'm really looking forward to this.

Glen

Yes, I'm very happy with this conversation because we talked in episode 8 with your colleague Bruno a bit about Netstack3 and he mentioned a bit about zerocopy. He also mentioned a lot about Netstack3 but he was less familiar with the origins of it. So before we dive into that, could you tell me a bit more about your own origins, Joshua?

Josh

Sure. So, yeah, I got into the software world in a way that a lot of people do, I think, which is I was originally interested in middle school, so I was maybe 12, 13, in just making my computer do things that it did not want to do. And I got into it through necessity—necessity is the mother of invention. I would have to Google, "How do I do this thing?" And then, you know, you'd find a blog post that says, you type this into the command line and I wouldn't really understand what I was doing, but, you know, and then it would break and whatever.

And so I sort of got into computers that way, just sort of wanting to make my computers do, you know, like I got into, into, let's say, the IT side of things before I got into software, like into programming. And then I took a software engineering class, like a programming class, in my senior year of high school and just fell in love with it. I mean, for all of the reasons that all of us who are listening to this podcast, I'm sure can empathize with.

So I ended up majoring in computer science in college and then did a follow-on master's. Throughout that experience, I was interested in the mathematical side of computer science. I was never amazing at math, but I was pretty decent at discrete math, proofs, models of computation, Turing machines, and that sort of thing. But I also really, really enjoyed the system side. And obviously that's sort of where I ended up. And as we'll talk about in this episode, there's sort of a through line there of figuring out how you can make systems feel like math. I think that's sort of a theme in my career is really liking the nitty-gritty details but being bothered by how ugly they are and sort of feeling like, wouldn't it be cool if the networking code that you write could feel as clean and pretty as your CS 101 binary tree implementation or something like that.

So that was my history. I focused earlier in my career on security for the most part. So in my first few jobs, I was at Cloudflare working on their security team for a bit. And then I moved to Google working on the Fuchsia operating system. And I was on the security team there for a while. And actually, Netstack3, which as you said, we'll talk about in a second, was just a side project for a while while I was on the security team. And then I worked on Netstack3 full-time for about four years.

And then now I'm back on the security team and focusing more on the programming language security side of things. You know, there's a whole world of, you know, being interested in cryptography and, you know, more hardcore sorts of stuff. I mean, I had a, you know, a stint at Cloudflare where I was even like mostly working with the lawyers on like, you know, corporate policy, you know, how many days does a message get to exist in our chat system before it's automatically removed as part of our data retention policy.

So I had a previous life interested more in the sort of more real world, let's say, side of security, but those days are maybe behind me. We'll see. But that's not what I work on right now.

Glen

Yeah, anyway, there's always a lot of connection with networking and systems and the real world. So, I mean, it's very cool. Not everybody who develops systems studied computer science. Do you think knowing particular areas of mathematics—discrete mathematics, for example—would help them?

Josh

Yeah, that's a really good question. I have seen this both ways. I tend to be biased toward the theoretical side of things. It's not that I ever actually think that your average systems engineer needs to do theory on a day-to-day basis, right? Even in my extremely weird niche job, which we'll talk about later, I rarely do proofs. I do them occasionally, but not often. The vast majority of software engineers and even of systems engineers are never going to be doing like abstract math.

The point is not that it's actually a job skill in the way that like learning calculus is a job skill for an engineer or something like that, right? For like a mechanical engineer. Rather it's about training yourself to think precisely. You know, thinking precisely is not something humans do naturally. It's not something that we're born to be very good at. And so you really have to train that intuition. And I happen to think that formal math training is one of the fastest ways of practicing precise thinking.

So the recommendation that I usually give to people who are early in their CS careers, they're doing a boot camp or they're in college or something like that, is: at least take a discrete math class, learn to do proofs, and take it really seriously. It's not that you're going to be using induction or set theory or something like that in your actual job. You probably won't. But the ability to think precisely is one of the characteristics that separates the great systems engineers from the good systems engineers. I basically don't know anyone who is in charge of large-scale systems or really, really challenging, really advanced systems who doesn't have that ability, whether they learned it formally or not. I know a lot of people who are extremely good at software engineering who do not have undergraduate degrees, but all of them have the ability to think precisely and clearly.

And all of them had to learn that at some point because it's not something that comes naturally. So however you get that, practice thinking clearly and precisely.

Glen

Okay, that's a very precise answer. Thank you very much. So Rust helps us a lot because it has a very rich type system. It also has a borrow checker and all kinds of nice tools, but it doesn't help with all kinds of logic bugs, like bugs in your system logic. Of course it couldn't. Then, next to that, sometimes you need to do unsafe actions. Later we will talk about zerocopy, which claims it does the unsafe for you so that we don't have to do it.

And we will also talk a bit about how you can apply this methodology to your own projects or your own libraries. Now as part of proving that you are safely doing unsafe, you make use of formal verification tools like Kani. Does using Kani require mathematical knowledge, or is the mindset of precise thinking enough? How much mathematics is actually involved? I know a little about Coq—C-O-Q—but not much about Kani. How closely related are they, and how does mathematics help?

Josh

Yeah, the answer in terms of something like Kani is that it probably helps to have a little bit of sense of the math, but it is not necessary. Now, the work the people building Kani are doing is super hardcore and theory-heavy. I mean, they're basically building on top of a theorem prover. I don't remember which theorem prover they're building on top of, but they're building on top of like a SAT solver or an SMT solver or something like that, so apologies for the word soup there.

But basically they're building on top of tools that are very math heavy. And so if you're developing the internals of Kani, you probably need to have a pretty solid grasp of the theory there. To build on top of it, to use Kani, I have not found it to be super necessary. But again, you don't need to know the theory, you just need to be able to think precisely. The Kani developers will yell at me for oversimplifying, but a rough example is that you give it a program resembling a test.

So if you think about, I have a function and I want to confirm that it is correct, I'm going to write a test and I'll provide a few different example inputs and I'll make sure that I get the output that I wanted. Kani basically allows you to do that for every possible input. Instead of testing a few selected inputs—or fuzzing with randomly generated inputs—Kani constructs a mathematical proof over every possible input. It asks: across the abstract space of every input this function could receive, can it prove that the output has the expected properties?

What that really means is that it has a list of properties, like whether your code will ever panic or exhibit undefined behavior. So it has a few specific things that it means when it says "correct". But basically that's the idea. It's thinking very precisely. So again, you don't need to actually necessarily understand the theory, but it's useful to be able to understand and sort of work with in your head a statement like, for all possible inputs, the program behaves correctly, right? It's a very sort of discrete math proof sort of statement.

So tools like that are, they're super helpful. You don't necessarily need to understand the rigor underneath them, but it does help to be able to think precisely in order to know what it is that they are giving you, like the power that they're giving you.

Glen

Yeah, I understand. Thank you very much. Now, we talked a bit about your background and I would like to shift our attention a bit to the start of Netstack3. You discussed or you said before that it got started as a side project. Is this one of those things that you're allowed to work on some personal interests within Google? Is that how it got started?

Josh

Yeah, so basically the history here is I joined the Fuchsia team in April of 2018. At that point, Fuchsia was a very research-oriented project. So at that point in its history, there were a lot of people who were thinking very deep philosophical thoughts about how to design an operating system. The existing networking stack at the time was this thing called Netstack2. I won't go too much into the history here because Bruno is in your previous episode, I think it's episode eight. Bruno already went into a lot of the background here for Netstack2, but basically it was a networking stack borrowed from another team at Google—"forked" is not quite the right term. It's not forked because they were still, you know, they were contributing upstream patches and all that.

It was written in Go. And there was a desire to move to a new networking stack that was (A) written in a different language. So there was a lot of Rust in the Fuchsia operating system at the time. And so the thinking was, if we do this in Rust, then it'll be better from a performance standpoint. It will be more predictable, right? Garbage collected languages tend to be unpredictable in terms of their performance. They tend to introduce latency spikes at different points.

It'll be better on memory usage because you can more tightly control when you allocate on the heap as opposed to the stack and how eagerly resources are freed. So all of those things were in the air. People were thinking about this, but no one was taking the idea seriously or no one was actively working on it. Since I have the platform of a podcast, I'll use it as a bit of a soapbox. This was a classic example of—I'm going to say that the phrase is, like, ask for forgiveness, not permission. That's a bit glib. It wasn't like I went off and did something that people didn't want me to do.

But I was this junior engineer, right? I was only about two years into my career at that point. It was sort of absurd for me to claim I'm going to go build this networking stack. But I did it. I just went to one of the people on the networking team and I said, hey, I want to do a networking stack from scratch in Rust. And this guy, Tim, Tim is still on the team. Tim's a great guy.

Tim, to his credit, was like, sure, kid, go ahead. Like, you know, like, I support you. I think this is going to be awesome. Let's see what happens. And just to give a sense of the hubris, when Tim first asked me, he said, OK, like, how long do you think it's going to take us to get to like TCP? My previous networking experience at this point, I had worked like, you know, on networking software in general, but I had not worked in the networking stack itself except for an academic context.

So I had implemented TCP/IP. I had implemented a networking stack as part of a course in college, and then I actually TA'd that course, so I implemented it again. So I had done it before, but in an academic context. And those of you who are professional software engineers will know those are very different things. And so Tim said, when do you think you're going to get TCP? And I said, ah, three months from now maybe. I think it ended up taking us like four years to even start working on TCP.

So it was definitely the sort of thing where a lot of it was just me being like just sort of naive, right? Thinking, "This will be easy." Well, you know, I've done a net stack twice before, right? Like how hard could it be? But again, to Tim's credit, he was willing to sort of, you know, take a chance and, you know, as we now know, Netstack3 ended up being very successful. And so, you know, it was definitely the right call, I think, to invest in it. So, yeah, to get back to your specific question about the history, it was a side project for me for about a year.

So I was working on it, you know, at first it was literally a 20% project, which is like a formal Google thing where, as you said, you can work on a project of your own interest. And then it switched to being sort of more like a 50% project where my boss on the security team was letting me work a little bit more on it. We ended up having external contributors during that year. So that was how Bruno came on as one of the people who was working on the project within its first, I don't know, maybe six months, nine months, something like that. We had a few other people who were interested in it. I was very lucky that a net stack is a very sort of like... you know, sexy project to work on, right?

And so we had a lot of interest from other people at the company who were really excited to donate their time to it. But that was sort of the history. It was just sort of a scrappy, like, you know, us-in-our-free-time sort of thing for about the first year. And then within a year, it became very clear that it was not going to be possible for me to give it the amount of attention that it deserved as just sort of this like part-time thing.

So I actually switched in early 2019. I switched to working on the networking team. So I actually moved bosses and all that jazz and then worked on it for, I guess it would have been another three years. So I think I worked on it in total for about four years. And yeah, so I was working on it for that time. And then when I moved back to the security team, which is where I am now, and back to working on zerocopy, which we'll talk about later, Bruno took over leadership from me at that point. And Bruno has been the tech lead of that project since then.

And so, you know, all of the cool shit that they've done over the past few years is, you know, credit to him and the current team. And they've done some very, very, very impressive things, which I'm sure we'll talk about. So he deserves a huge amount of credit for shepherding that project to where it is now. And we've just had a really, really good, really talented set of engineers who have worked on it. But anyway, yeah, so that was sort of the history. As Bruno mentioned, it's now at the point where we're mostly transitioned, Fuchsia is mostly transitioned from Netstack2 to Netstack3. I don't know that, again, I'm not on the team anymore, so I don't know the exact details.

I don't keep like, you know, super, super good tabs on it. But I think they're a good chunk of the way through that transition, is my rough understanding.

Glen

Yeah, that's also what I got more or less. Now, in your own words, would you be able to tell me like why they believed in it so much that they allowed this project to be worked on more and more because, okay, first it was just a little side project and Tim believed in you so far, I get it. But at some point they do need to see something that they think like, okay, okay, this is definitely even though it's like a kid doing it, like clearly this is something like—would you be able to know why that was that they certainly believed in it so much?

Josh

So that's a really good question. The honest answer is I don't know. I can speak to the virtues of the project, and I'm happy to talk about that. What was their actual calculus? Impossible to know what's in somebody else's head. You should have Tim on the podcast. He would be a good guest.

I think, I don't know, do you feel like this is maybe a good opportunity to talk about some of the project? We were talking beforehand about, you know, some of the interesting things we did. I think that's maybe a way of answering that question without actually speculating about what was in Tim's head at the time. Maybe that is a thing we should transition to.

Glen

Yeah, yeah, that's fair enough. We definitely don't want to start speculating.

Josh

Okay. Yeah, exactly. Again, you should have Tim on. He's a cool guy. So, yeah, basically, I mentioned before that a theme in my career has been really enjoying systems software, but being bothered by how messy it is. And I think, again, this dovetails really nicely with the thing we were talking about before about thinking precisely. One of the things that I have always really valued, and this sort of comes from the security world, I think, is that in the security world, especially in math and when you're doing cryptography, for example, right, the very sort of like mathy side of security, you are taught to never, never be confident that something is the case unless you can completely prove formally, rigorously that it is actually the case.

Right? So, you know, the history of cryptography is littered with, I think the Bruce Schneier quote is, "any person is smart enough to construct a system that is clever enough that they are not smart enough to figure out how to break." So it's very tempting to feel like you have designed something that is perfectly secure that is not actually secure. And it contains, you know, a vulnerability that you didn't think of.

And so you are constantly trained to be adversarial towards your own designs and to really understand what it feels like, the sort of emotional, psychological feeling of being certain and to distinguish “this feels basically true” from “no, I am 100% confident that this is true.” And so that cryptographic experience, like learning cryptography and taking cryptography classes and working through exercises and things like that, really gave me this sort of appreciation for the difference between "I understand how this works" and "I really understand how this works." I think that was one of the really useful parts of my education for the work that I do now was being able to distinguish between "I understand how this works" and "I really understand how this works." And so, as I said, the theme in my career has been trying to marry these two things, right?

Enjoying systems programming, but wanting to get back that cryptographic type of feeling of, "I really understand how this system works." And systems programming in general has historically encouraged you to live more in the former domain. It's just often not possible to so completely understand the details of the system that you have that level of certainty about how it behaves. And so maybe there are some super hardcore kernel hackers, blah, blah, people who really, really understand the system.

But in general, there's a lot of sort of like, look, you understand it as well as you can, and then you make progress. It's sort of a pragmatic thing. And so when I started working on Netstack3, I wanted to have that feeling, right? I wanted to be able to say that I actually understood how the system worked. And the thing that I love about Rust is that it basically turns what normally is a systems programming type of problem, where you just kind of have to like vaguely understand what's happening and it's going to give you access to lots of low level system details and just sort of, you know, in a C or C++ style, trust that you're holding it correctly. It basically in the type system encourages you to think more precisely and say, no, I need you to tell me exactly the behavior of this program, right? If the lifetimes don't match up, I'm not going to compile. If the ownership isn't perfect, I'm not going to compile, right? It's forcing you to think very precisely about the behavior of your program and encode that behavior in a way that is legible to the compiler.

And so that is a lot of philosophy. But there's a question then of how you actually turn that into practice. And so in Netstack3, what that looked like was basically saying: every time there is some form of correctness, some sort of property that we might want the netstack to have, we're going to try to figure out how to actually represent that as a first-class abstraction with its own types and its own representation. And then we sort of pushed that very far. One of the things that's really nice about working in the networking world, and I think I've been in some sense spoiled by this, is it is completely unambiguous what a correct program looks like.

If you were building a video game, there is no sense in which you have built the correct or incorrect video game, right? Maybe you've built a better video game or a worse video game or something like that. There's no definition of correctness. In the networking world, the definition of correctness is: "Did you implement the RFC correctly?" And so artists talk all the time about how requirements and restrictions can actually be very freeing because they sort of free you from the paralysis of choice. The same is true in the software world. Having a very precise specification of what you're supposed to be building is very freeing because it lets you innovate not on what you're building, but how you're building it.

And that was sort of one of the things that we experienced in Netstack3 was that having such a precise constraint about what we were supposed to build, let us focus all of our creative energies on the software engineering aspect of things instead. And that was really sort of how I conceived of Netstack3. I thought of it as this, like the main challenge was the software engineering, right? How do you, if you've taken a networking class, you may have seen the five layer model or the seven layer OSI model or whatever, and you have this very clean abstraction boundary, Ethernet sits there and doesn't know anything about how IP works and IP sits above ethernet and doesn't know anything about how ethernet works. And TCP sits above IP and doesn't know how IP works. It doesn't even know that ethernet exists.

That is a complete lie. That is a beautiful, nice, simple picture to provide a mental model that we can teach to students. But if you've worked in the real networking world, you know that we violate those abstraction boundaries all the time. So there are instances in which TCP actually needs to reach down not just into IP but actually through IP into the boundary layer between IP and Ethernet in order to muck with the neighbor table entries in IPv6, if that's a term that the listeners are familiar with. If you're not familiar with it, it doesn't matter. The high-level point is just that the clean modular picture that you were given of a network stack is a complete lie. These things are complected together to high heaven.

And so your natural inclination, if you're going to read all of these RFCs and then sit down and ask, "How do I actually implement this?" You're basically given what is sort of like the RFC equivalent of spaghetti code. And so your natural inclination is to write spaghetti code. The most natural, obvious way of implementing a networking stack is just to have every piece of the stack call directly into every other piece of the stack and all be complected together. And there's no way for you to pull apart an individual component in order to reason about its behavior in isolation. And so, as I said, the benefit of working on a net stack is that the requirements are specified for you so you can aim your creativity at the software implementation.

And what we did in Netstack3 was we aimed our creativity at the question, how do you recover good software engineering practices and principles out of a world that is just trying so hard to make you implement an ugly spaghetti code mess? How do you basically push back against that and say, no, we're going to do it right. We're going to do it in a way that is testable and modular and easy to reason about and so on and so forth.

And so that was the main challenge. And Rust's type system was so, so, so helpful in achieving that goal. Because basically there are so many instances in which one part of the stack needs to reason about what another part of the stack is doing. And if you were writing in a language that either didn't give you the rich type system of Rust, or even if you're writing in Rust, but just not making use of the rich type system, you would be really tempted to just go ahead and do that, right? TCP is going to call directly into NDP. That's just going to happen. And if you want to reason about the behavior of that interaction, you just kind of have to understand how the whole system works.

What Rust lets you do instead is say, okay, TCP is going to call into NDP. That is unavoidable. But we can very carefully shape the API in such a way that what it means for that interaction to be correct is completely expressed in the type system. And so if we hold that interaction incorrectly, it will not compile. So yes, there is complexity. These things are calling into one another. That is unavoidable.

But all of that complexity is made explicit in the type system. And what that lets me do is anytime I want to change things or experiment with different ways of expressing that complexity, I am sure that if I get it wrong, my program won't compile. And that is so much more pleasant of a developer experience. And it makes building complicated APIs and complicated interactions actually feasible because I can just iterate quickly. I can trust that if I made a mistake, I'll catch it immediately as opposed to the experience that I think a lot of us have had in large code bases where you say, "Boy, I could do that, but I would have no way of convincing myself that it was correct."

And so I'm just not going to do it. There is an engineering virtue in simplicity. There is an engineering virtue in holding yourself back and saying, I am going to sort of exercise humility and not do the crazy optimization because I can't trust that it's correct. What Rust says is, "No, go ahead, try it. As long as you encoded it in the type system, I'll tell you whether you got it correct or not." And so, in some sense, Rust is like a language of hubris, right? It's a language that says, "go ahead, try your crazy idea. I'll stop you if you got it wrong."

And so that was really what we did in Netstack3 was to say, let's actually go ahead and try all the crazy optimizations, but we'll be very careful to encode correctness in the type system so that we know when we got it wrong. That attitude is the origin of many of our abstractions, including zerocopy, which we'll discuss later of saying, let's push this as hard as we can. But at every single step, make sure we're doing good software engineering and never getting out over our skis.

Glen

Hmm, that's very cool. And I totally agree. I mean, my original background is C++ and compared to that developing Rust is a world of difference. Now we are also the developers of Rama. It's like an open source network framework and it's also there where we also really think in these layers and where we like, and it's nice because like you said, when you implement these RFCs, which is a very pleasant experience because you can really, as you say yourself, think about how you implement it. Then it's very nice. And the way we implement this is that we can very abstractly, okay, let's say we implement the SOCKS5 RFCs or we implement the HTTP RFCs.

Well, there are many HTTP RFCs, but like we can really think about it and implement them in a way that's very abstract. Like, let's say SOCKS5 needs to work with some kind of byte stream. It doesn't really matter which byte stream. You don't need to worry about whether it's on TCP or not. It also affects the way you stack it. Are you familiar with the Tower system and the Tower Service trait?

Josh

I have heard of it. I know a very small amount about it, so basically no.

Glen

Yeah, so to introduce you a bit to it, it's basically a very simple trait, which was inspired by an original framework from Twitter back in the day, which was called Finagle if I'm correct. And it revolves around one single service trait, which takes an input and needs to produce an output, which can either be like, okay, or like an error. And that's basically it. And that's how you would like in a framework like Rama, like stack your layers because you would basically start with like something which may be like a TCP server and it needs something that can serve TCP streams.

So that could be like some kind of TLS acceptor or it could not be, it could be like a SOCKS5 and you kind of like just stack these things on top of each other because they just wrap each other or they stack on top of each other like a turtle, however you want to think about it. And it's very nice and that's one example where I like it a lot and where we already make very good use of the type system, whereas I feel that it's much harder sometimes, it's like where you have to work with dynamic states. And I have a feeling that in Netstack3 you have a lot of clever tricks to work around this.

So examples of this are things like, I don't know, let's say the TCP acceptor accepted something and it injects socket information into kind of extensions of your input because we, like I said, the service trait accepts inputs, which could be a request, but could also be like a TCP stream with some extensions, or basically just you can inject anything into it. And so in order to access those, you always have to kind of check, "Is it there or not?" And sometimes you might assume, "This is always there," because let's say I can always assume that it's socket information, but still you have to do an unwrap there or you could just say it's fallible.

And then another example of things where you kind of have to—yeah, I don't know, like those are things like all those dynamic states, I find that a lot harder to make very good use of a type system because you don't have static verification of like, yeah, this is always there in the flow. And at the same time, sometimes, for example, in our case, we have these stacks and then you can have like a giant mountain of generics, which stack and stack up, like it wraps, it wraps. And so at some point you might just choose to box or something because yeah, it's just easier.

But then again, also you erase your type. Yeah, like, because I know from like a talk with Bruno that you had some clever tricks, according to Bruno, where, for example, you might change behavior based on whether it's IPv4 or IPv6. And for example, we also deal with IPv4 and IPv6, but in our case, we just use match statements, right? Like we check, okay, it's an IP address. We know that; that's what the type system says. The type system also allows us to match in an exhaustive way. We do have that and we could even use it there, but I wonder: do you ever have to work in that stack with dynamic state or do you always get away with like somehow, I don't know, not having to inject such states?

Josh

Yeah, so I think in general, the philosophy that we tried to push towards was the only things that are going to be dynamic are the things that are actually dynamic. And what I mean by that is like, you know, the protocol defines them to be dynamic, right? So for example, a UDP packet is allowed to have a missing source port if memory serves. I'm pretty sure that's true. There's one of the fields that's allowed to be optional. I think it's the source port. And people listening can go confirm that I'm right about that.

But there's one of the fields that's allowed to be optional. And so that's an instance of inherent dynamism. There's just no choice. Your netstack has to be able to handle the case that there's a missing source port. And so in those cases, what we do is we try to encode that in the type system. So we'll say, you know, I think if memory serves the UDP packet parser that we wrote, if you want to extract the source port, the source port is represented as a NonZeroU16, and the getter for that field returns an Option<NonZeroU16>. The caller is forced to deal with that – what do I do if there's no source port here?

By contrast, if there are things that are guaranteed to be present, and it's just a limitation of the type system, then we try as hard as we can to figure out how to introduce new APIs that actually guarantee the presence of that thing. So I'm trying to think of a good example of this, but basically, there are instances where you know that the RFC requires a particular thing.

And so in, let's say, like parsing a packet, you might say, "If this field is missing, then that's a parse error." So we're never going to parse and produce a parsed representation of this packet where the field is missing. And so that becomes an internal invariant. And then your APIs to access the fields of the packet can be infallible.

Glen

The reason why I brought it up is because you were talking about when you're parsing and that's within a single layer and I feel there it's a lot easier and also in the case of something missing, once you've developed long enough in Rust you would quite quickly think about making an option for that, so far so clear. What for me is much harder is like, and I'm sure you have that same problem set in netstack, is where you cross boundaries. Because between, let's say, a TCP socket, which you accepted, and your application layers, let's say, HTTP, there are a lot of different layers, and sometimes there could be more, sometimes could be less.

For example, in like a very extreme case, you could end up with eight layers in between. Right? Because you're dealing with SOCKS5, you're dealing with TLS, you're dealing with all kinds of stuff. Maybe also like HAProxy. I mean, you can, you know, you can think of a lot of them. And so there lies the problem, because if I would just deal with, "Okay, my service is accepting a TCP stream." So that's one layer.

Then I can guarantee very easily like, yeah, I can, I can get from that my client address or my server address or whatever, or my peer or whatever you call it. But once I get to, "Okay, now I have a service accepting, let's say, an HTTP request." I'm already very, very far in the app structure. And at that point, it's a lot harder to still, even though like conceptually, you know for certain that this comes from like a TCP packet. I mean, it somehow has a connection to that, right? Like, okay, somewhere is some task parsing this and whatever, and dealing with an actual TCP socket.

But I find it much harder at that point to guarantee that you can always get socket information from this without placing a huge burden or restrictions on the system, unless I would put that kind of information directly into an HTTP request, but that doesn't seem to really fit there either.

Josh

So I think that a big answer that we have in the Netstack3 code base is just that we use types a lot. Like a lot a lot. And so I do think this is one of the limitations. So a point of philosophy here is the concept of using types. Compare and contrast that with Python or assembly or something like that, where you don't have any typing. The concept of introducing a type system tends to be really good at reasoning about properties that are local. As in, you have, if you have a function that takes an integer and returns a bool, and if I have an integer, then I can call that function and I get a bool back. It's a lot less good, in general, at talking about global program properties, right? It's a lot less good at expressing the idea that, you know, the—what's a good example of this? You know, an instance we ran into in Netstack3 is that you have interfaces and then you have sockets that can be bound to those interfaces. Those live in very different parts of the code base, but they're actually related to one another.

So in theory, if the interface goes up or down, then, so let's say you have an interface, you have a TCP socket bound to that interface. If the interface goes down, the TCP socket, the correct behavior might be to close the TCP socket. That's a sort of configuration thing. But let's say, just for the sake of argument, that you want to close the TCP socket because the interface went down. How do you express that in the type system? How do you express the idea that this is a TCP socket and that it has an interface, and that the interface that it is on is definitely up because the TCP socket exists? That's actually really hard.

And I will say, this is one of the things that Netstack3 didn't have a good answer for the entire time that I was there. After I left, this was one of the things that they tackled. I know that Bruno, I think, mentioned in his episode that they ended up with a fancy Arc-based system in order to track resources like this where you need to have multiple references throughout the stack to, for example, a particular interface. I don't know the exact answer of how they deal with this, but it's actually genuinely hard. And I think the Rust type system lets you encode information that is non-local more powerfully than other type systems do, but it is not perfect. Right, the Rust type system is Turing-complete. In theory, you can do anything you want in the Rust type system. That doesn't mean that it's going to be ergonomic and it doesn't mean that it's a good idea.

And so in theory, you could imagine structuring a system where somehow you encode the information in the type system of like the liveness of every interface. That's probably going to make your type signatures on every single part of the stack like awful. And so partially the answer to your question is we didn't have a great answer for this. It was like an open area of research. Again, I think Rust makes it easier to encode non-local information or at least to encode more complex information. And I can talk in a second about an example of that.

But yeah, I think it's a really hard problem. And I don't think we had a great answer for it. I think that being able to encode the local information alone is such a win compared to C/C++ that it's still a big difference, but that does not mean that the effort is done. We still have a long ways to go in figuring out how to develop large system software in Rust in a way that is completely infallible, in a way that if it compiles then it works. If it compiles then it works is true in Rust way more than it is in other languages, but it's not 100% true.

Glen

Yeah, yeah, for sure. And anyway, in the end, you're forced to deal with the fact that it's there or not there and you can fail it. And in the end, I learned to live with it. Like in the beginning, I tried to do everything with static states and it was, it became after three years of development, such a disaster that recently we switched away from that. And at some locations it makes sense and you can do it. And other layers, most of the layers, you actually just learn to deal with the fact that some things are just dynamic. And given the fallible nature of networking, that's fine as long as you do use, like you said, locally, the Rust type system very well to deal with those things and not just panic or assume; you do deal with the possibility, even if you know most likely it's not because you want to make sure that depending on how you stack it, it works.

So, I mean, I learned to live with it, but yeah, sometimes I do wish like a lot more could be done, but yeah, it's a trade-off and okay. So we talked a lot about systems already. We talked a bit about designs. We talked about how Rust helped you. We talked about your background. Now, what is not clear to me and which also allows us to segue into zerocopy is at what point zerocopy became a thing, how it was conceptualized, and at what point you started to work on it.

Josh

Yeah, so zerocopy was like a classic necessity as the mother of invention project. We actually started it probably within a month of starting Netstack3. So Netstack3 dates to maybe May of 2018, if memory serves, and zerocopy, I think I wrote the first lines of code in like June of 2018. So super, super early on. Basically, the problem that we were trying to solve was that we wanted to do zero-copy parsing. So I should clarify here, zerocopy as a phrase is the name of this crate. Zero-copy, which unfortunately is pronounced identically, refers to a style of parsing.

So if you receive a message from a network or you've read in a file format off of the disk or something like that, and you want to parse it, for performance reasons, you would like to not have to allocate more memory elsewhere. You would like to be able to reuse the existing buffer that you have and just sort of operate on it in place. That's what zero-copy parsing refers to: the idea that you're not making copies from the buffer into another location.

So we wanted to do that style of parsing, and in C/C++ this would be fairly straightforward, right? You would define a struct whose memory layout happens to be the same as the memory layout of the packet format that you're trying to parse. And then you would take your pointer, right? You would have like a char * or something like that. And you would do a cast into your struct type, right? So let's say you're trying to parse a UDP packet. You might say, "Okay, I'm going to cast my char * pointer into something like a udp_header *," right?

And now I have a pointer that is typed, right? It's a struct pointer. And I can just go do field access, right? So I can take my pointer and I can do, you know, packet->src_port or something like that. So that was roughly what we wanted to do in Netstack3. Of course, doing that in Rust is unsafe. And, you know, if you've worked with this stuff, you'll know.

And if you haven't worked with it, then you can just take my word for it. It's extremely, extremely subtle. There are many, many, many edge cases that you have to consider in order to determine whether this is sound at all. And I'm not talking about edge cases at runtime that your code has to handle. I mean, edge cases in order to figure out whether it's even correct to make this program at all. So edge cases in the type system, basically.

Glen

Yeah, no, I mean, I'm very familiar with it because, mean, originally I come from the game industry and we did a lot of that as well. And it's also probably why in the RFCs, a lot of them, I mean, I feel the layout is often made with the exact structure in mind and some RFCs even explicitly mention the structure. And it does work, except of course, where you deal with flags. mean, if you... Maybe my memory is a bit vague here, but as far as I know, once you deal with flags you do have to at least do some kind of bitwise operation to turn it into some kind of boolean, because there is no way, even in C as far as I know, that you can just turn it directly into a struct, or am I wrong?

Josh

Yeah, in C I'm not exactly sure. I don't know off the top of my head. Something tells me that there are subfields, yada yada, but I'm speaking out of turn there. In Rust, the answer is no. We don't have any types in Rust that are less than one byte in size, except for types that are zero bytes in size, but that's a separate thing. There are crates out there, like the bitflags crate, or bitfields or something like that that will handle this for you, but it's a little bit higher level than we needed because we need to be able to precisely control the memory layout. I may be speaking out of turn there, maybe that is something that they support, but for whatever reason, it wasn't something we ended up reaching for.

Luckily, it's not that big of a deal to do the bit manipulation stuff. It's a bit of a pain to write some manual code. Luckily, it's not a memory safety thing. The thing that really mattered for us was the memory safety aspect. The big things that you have to deal with here are you have to make sure you've done bounds checks, right?

So if I give you eight bytes and you try to cast a pointer to eight bytes, or a slice reference, right? A u8 slice reference that's eight bytes long, and then you convert that into a reference to some struct type that is longer than eight bytes, then that's undefined behavior, right? Because it'll allow you to read off the end into some other garbage memory. The other thing you have to deal with is alignment, right? A lot of types in Rust have some form of alignment requirement where they have to only exist at even byte addresses or byte addresses that are a multiple of four or multiple of eight or something like that. In general, if you have a slice of u8, you don't know where in memory that lives, right? It might live at any arbitrary byte offset.

And so even if you've already checked the size, you also have to check the alignment. We realized that repeating all of this for every packet parser would be a persistent source of bugs. And it was also just going to be really annoying to deal with. It was like, even if we got it right, it would take a lot of effort every single time that we wanted to get it right. And so what the zerocopy library gives you is a type-level representation of properties that your type satisfies.

So it's a bunch of traits, basically, and if your type implements one of those traits, that tells us something about the memory layout of your type. One of the base traits is FromBytes, which basically says any bit pattern is valid for this particular type. So any pair of two u8s is going to be a valid instance of a u16, for example. Any bit pattern is okay for u16. This is not true for all types, right? So a boolean, right, a bool in Rust can only be 0 or 1. If you try to interpret the byte 2 as a bool in Rust, that's undefined behavior in your program that's going to be arbitrarily wrong.

So basically that's the idea, is we have these traits in zerocopy that encode the idea of... whether your types have certain properties. And then we have a bunch of functions that will do these conversions for you and use those trait bounds to force you, the user, to prove that this is actually going to be an okay conversion to do. So for example, if you want to convert a reference to a slice of u8s into a reference to your type, you're going to have to prove that your type is FromBytes, right? That any possible bit pattern is acceptable for your type. And you're going to have to prove that it is unaligned or that it has no alignment requirement.

If you do those things, then we will provide to you an API that will at runtime check the size. We'll make sure that you've actually given us the correct number of bytes. And then if the size check passes, then we'll do that reference conversion internally and we'll give you back a reference to your type.

So that's the basic premise. It's this very low-level little like Swiss Army knife type of thing, right? It's very low-level utilities, but the point is that the API is 100% safe and you can build up much larger abstractions on top of that. All of the packet parsing in Netstack3 is built on top of zerocopy, right? So it's all you give me a buffer, I'm going to convert in place the type of the buffer into a type that represents the memory layout of the particular packet format that I'm trying to parse. And then I'm just going to go on my way using normal Rust operations.

So that's sort of what zerocopy does. It has grown a ton over time. Like it was very small in the beginning. Now it's a much, much larger crate. It's the full-time job of two different engineers. So it's grown a lot since then, but it originated with that basic need for zero-copy parsing.

Glen

Yeah, I mean, it's awesome because I feel I was like I was reading through the source code of zerocopy earlier this week as I knew we would or actually last week because it's just Tuesday. Yeah, sometimes the week boundaries are unclear to me. I have three small kids, so they flow so quickly. So the last couple of days I was reading a bit through it. And yeah, it seems like then you have the zerocopy-derive crate and there, I mean...

Josh

Yeah, sure.

Glen

It seems that you are doing all the hard work that otherwise you would have to do manually because you're putting the correct annotations, you're putting the correct operations and checks, and you also make use of the type system to verify statically what you can verify statically. I mean... Is that like a fair summary? That it is just instead of doing it like all manually, like now you can just use like your procedural macros and the type system that you get the system of types that you build around it so that you don't have to do all that work yourself each time.

Josh

Yeah, that's exactly right. Basically, there are two steps here. The second step, I'll say the second one first. The second step is if you have proven that your type satisfies these properties and that that proof is carried by the type system in the form of one of these trait implementations, then we'll provide for you all of these runtime functions that will do various operations that would normally be dangerous. Then that raises the question, how do you actually prove that your type satisfies these properties? And that's where the derives come in. Most of these properties are recursive.

So basically a struct can be FromBytes if all of its fields are FromBytes. And so what that means is that it bottoms out at the core primitive types. So we basically just hand-wrote in the library implementations of these traits for all of the built-in types. So, you know, u8, u16, u32, i64, et cetera. We have implemented the appropriate traits for all of those built-in primitives. And then the custom derive will allow you to automatically derive an implementation of these traits for your type. And that derive will either fail during the processing of the derive itself, or it will emit code that will fail to compile if your type doesn't actually satisfy these requirements.

So for example, if you have a struct that has a boolean in it and you run the derive, then the emitted code from the derive will say, "Sure, your type can be from bytes if bool is FromBytes. But bool is not FromBytes," so it fails. And then in addition to that, there are extra constraints that we have where in the derive itself, we can check for other conditions and sometimes we fail during that pass as well.

But yeah, your high-level description is exactly right. The derive lets us determine whether your type satisfies certain properties. And if it does, then we turn those properties into trait implementations and then zerocopy itself, the runtime code consumes those traits as a bound in order to prove that we can do certain operations soundly at runtime.

Glen

Yeah, very cool. In a way, I feel, let's for example, within the framework Rama for now I'm not using zerocopy myself. I feel I could definitely make use of it because for now I'm doing a lot of that work myself. At the same time, I do wonder because like, okay, it's nice to do zero-copy and that's possible because you're borrowing from the memory of the slice, or you are just basically viewing it through another lens. Well, I suppose that sometimes it also just means that you are copying, you are copying, right? It's just on the stack. No, I mean, in the end, it's not like you are literally just, I suppose you just mean you don't make any allocation on the heap, but you are copying it to another, like on your stack.

No, am I wrong there? Like if you, for example, would interpret a byte slice as some kind of, I don't know, TCP header or any kind of other protocol. In the end, you are copying it, but like on the stack, right?

Josh

Yeah, so basically the thing that is getting copied here is the pointer, right, or is the reference. So if you have a TCP packet that you read in as a slice of u8 in a buffer, that is represented as a u8 slice, right? When you do a zero-copy conversion, you're not actually moving the bytes that are the referent of that reference. You are leaving those where they are, but the reference itself is going to be a stack variable, right?

So that the reference, as in the ampersand thing, is going to live on your stack. What zerocopy is letting you do is just change your mind about the type of that reference. So it's going from a slice-of-u8 reference to a UDP header reference or a TCP segment header reference or something like that. The bytes themselves that are live somewhere else, maybe they live in somebody else's stack frame, maybe they live on the heap, those stay where they are and they don't get copied around.

But the reference to those bytes, that is going to get moved and copied and passed around and all that jazz. And it's just a matter of changing the type system's mind about what type lives on the heap or in somebody else's stack frame or wherever it is that you're pointing to.

Glen

Does that mean that you have an opaque struct and to get those properties you need to call methods and it's only then that you copy to the stack, or how does it work? I just want to figure it out. Yeah, yeah, yeah.

Josh

Right, exactly. Yeah, so let's say, I mean, it doesn't have to be opaque necessarily, but the idea would be, let's say that I want to operate on the destination port of a UDP packet. What I'm gonna do is I'm gonna take my slice of bytes, I'm gonna convert it into a reference to a UDP packet header. Let's say that's the name of the type. That type is going to have fields, right? So it'll be a struct that has a source port, destination port, length, and checksum—the four UDP header fields. When you actually access one of those fields, let's say you do packet.dest_port and you pull that out, the compiled code that Rust generates is going to be copying those two bytes from somewhere in memory into your stack, maybe into a register or maybe your stack, whatever, and then you can operate on it like normal.

But basically the idea is it should generate the same machine code as if somebody had passed to your function a reference to a UDP packet header. That's sort of the idea, right? Let's imagine what would have happened if we had been operating on a reference to a UDP packet header this entire time. Ideally, that's gonna be exactly the same machine code that gets generated as if you're using zerocopy. There is a slight asterisk there, which is that we're always going to have to do a length check at runtime, so it's not going to be identical, identical, but it'll be pretty close.

Glen

That also probably means that let's say you have a function and you access that address 10 times, I suppose you want to just call that method once and then just always use the final version, right? Because you wouldn't want to do that work 10 times, I suppose, in a single function. Yeah, yeah, yeah.

Josh

Right, like you wouldn't want to do the length check 10 times. And so yeah, usually you will just do this conversion once and then you'll operate on the typed reference once you have it. You'll just operate on that from then on.

Glen

Okay, cool. And then a lot of these protocols, also... I mean, it's very nice if everything is like in a fixed location for, like, the metadata, ports, and flags. And so that's usually the case, but then usually you also have like optional parts or, or length, like, like let's say string values or something like that. It could be like a domain name. And then usually they encode like the length in the first two bytes. Then maybe, I mean, they go even fancy to maybe they even do something like where, if it was a small value, it's like one byte; if it's a big value, it's three bytes of length and then you know the other length amount of bytes is your string value.

Does zerocopy deal with that kind of stuff too?

Josh

Yeah, so this has been a huge headache for us, is the answer. Basically, zerocopy today is capable of supporting any byte pattern that can be represented as a native Rust type. So if you can build a struct or an enum or a union whose memory layout is the same as the memory layout that you're trying to parse, then you're golden and zerocopy can help. But there are certain things, there are certain memory layouts that just don't correspond to any native Rust type.

So the format that you were just describing, which is called TLV, or type-length-value, is basically a variable-length number of objects. And the way that you parse them is you have to sort of chomp them one at a time, where you look at the header and that tells you how many bytes to eat. And then you eat that number and then you parse to the end, and then you look at the next thing's header and that tells you how many bytes to eat and you iterate that process over and over again.

There is no native Rust type whose layout is equivalent to that thing. And so zerocopy can't really help you. It could maybe help you parse an individual one of those things, right? So it could maybe help you parse like a single instance of the TLV format, a single record in the TLV sequence of records, but it can't help you parse the whole thing. And so what we did in Netstack3 is we have a crate called packet.

So generally speaking, there are three sort of utility crates here that we talk about a lot. There is net-types, which contains a bunch of our low-level networking primitives. This is like IP addresses and MAC addresses and stuff like that. There is packet, which is low-level utilities for packet parsing. So it builds on top of zerocopy and gives you a framework for doing packet parsing, but it doesn't actually implement any specific protocol. And then we have packet-formats, which is all the packet formats. So TCP, UDP, IPv4, IPv6, MLD, IGMP. The list goes on. It's an absolute alphabet soup of different packet formats.

So in the packet crate itself, which is just this framework, we have a set of helpers for parsing repeated records. It turns out that in the networking world, there are many different formats for repeated records. TLV—type-length-value—is just one of the instances of that. So TLV-style parsing tends to show up in IPv4. It shows up in TCP. Both of those have like optional variable-length options fields. IPv6 uses this like crazy thing called extension headers, which can be nested. So you can have multiple different extension headers that follow after the main IPv6 header. And then inside each extension header, you can have variable-length options. It's crazy. The amount of code that we had to write to parse IPv6 packet headers is insane. It's like thousands of lines of code because of how complex the format is.

So zerocopy can only help so much with these things. As I said, it'll help you do like a single struct, a single union, something like that. But you can't just magically generate code that does all of that—the entirety of IPv4 options or IPv6 extension headers. We would love to move in that direction at some point, but it's an open research problem of how you actually represent that.

What we do in the packet crate is we have utilities that will at least help you structure your code when you are parsing iterated records. So generally speaking, what this looks like is you give us a slice of bytes, a slice of u8, and then you also have to give us some amount of code that tells us how to parse an individual record. And then out of that, we will give you back an iterator over all of the records. So for really bespoke things like IPv6 extension headers, it's fairly low level. We'll just say, like, listen, if you can parse an individual IPv6 extension header, then we'll let you parse multiple of them one after another. We'll sort of like... structure—we'll scaffold all of the code around that. For some more common types like TLV, we actually have a native TLV parser in the packet crate, but that is not in zerocopy.

For those of you who are interested in using these things, zerocopy is on crates.io. You can use it right now. net-types, packet, and packet-formats are not published on crates.io. It would be super cool to do that at some point, but we're not sure when we will or what it would entail to do that. You are more than welcome to just copy-paste the code out of the Fuchsia repository. It's all there. It's open source.

You can read the license and determine whether the license is acceptable for you. But it's not like on crates.io. So it's not super nice and easy to use. But yeah, so all of the record stuff that I was talking about is in packet. So it's in the Fuchsia code base. You can go copy-paste it, but there's no crates.io package you can just take a dependency on.

Glen

Yeah, cool, no, I'm definitely going to at least study it. It's always interesting to read other people's code. It's one of my things I like to do next to reading RFCs. I used to read the newspaper more, but nowadays I'm reading mostly RFCs.

Josh

RFCs are a lot less depressing.

Glen

Yeah, yeah, you learn a lot from it and nice to read. Anyway, so that's one thing. Okay, cool.

So sadly not in zerocopy, but at least you solve it in a different way.

And it's an open research problem, so the future is bright, who knows. Okay, another issue I have, or not really issue, like problem space I maybe create for myself is this, which deals with the function coloring problem. And, for example, the Rama framework is heavily async. And sometimes it's not clear, okay, when you want to make things async and when not. And so for clarity, a lot of stuff we do is async just because, depending on how you implement a trait, you might anyway want to have some kind of async behavior because I don't know, maybe you need to validate something, but you need to check something in I/O or some kind of database or whatever. Having it async first helps a lot there, but it does mean that you cannot really as easily make use of something like zerocopy.

I would think, because like I couldn't just hold a reference to a slice and then move it over like you're requiring Send + Sync trait bounds or whatever; I couldn't just move it over because it wouldn't work. I mean, but I think within Netstack3, I don't think you have those issues. Yeah, actually, do you have those issues there? I suppose maybe you're Arc-ing the bytes there or something, and that's how you get away with it. I don't know.

Josh

Yeah, so I'll say a few things. First of all, I'm not intimately familiar with these issues. I have not done much async programming myself. So I may be speaking out of turn. My general feeling about powerful frameworks for systems code is I try to avoid them as much as I can unless I really need them. My view on async is that the Rust async solution—the whole async system—is an engineering marvel that you should try to avoid if you can.

I work with Tyler Mandry, who is one of the core designers of the async system. He has done amazing work. The Rust folks have done amazing work on designing async. It is a beauty. But it's really complicated. It is necessarily solving an extremely complicated problem. And it makes it difficult for you to reason about the behavior of your code. And so you should use it if you need it, right? If you're writing some highly parallel web server and it makes natural sense to use async, like go ahead and use it obviously.

We tried to avoid it in Netstack3 as much as possible because of how tight a control we need over the execution model of the code, right? I need to know not just: when the code runs, what does it do? I also need to know: when does the code run? And what does the control flow look like? And async makes it difficult to reason about what the control flow is going to be.

So Bruno talked about this in the previous episode, which listeners should definitely go listen to. It'll provide a lot of extra context for what we're talking about here. But I'll repeat briefly what he said, which is that 90-ish percent of the code in Netstack3, by lines of code, is in a crate called the core—or a part of the code base called the core. It was a single crate at some point. Maybe they split it up since then.

But the idea is that philosophically, there's this region of the code base that is completely platform agnostic. So if you go and you read that source code, you won't see anything specific to Fuchsia. And in that code, that's where all of the actual protocol implementations live. That's TCP, IPv4, IPv6, yada yada yada yada. The whole protocol soup lives in there. Inside of that part of the code base, there is no async. It's all just straight-line functions.

So you call in and the code executes to completion and then it returns. In the course of executing, it might call back out into the external system. We basically have a dependency injection mechanism so that code that lives outside, in what we call the bindings—which is the thing that binds the core to the specific system—can interact with the core. So the bindings are where the Fuchsia-specific code lives. We have a mechanism for the bindings to call into the core and then a dependency injection mechanism that allows the core to call back out into the bindings.

This is what lets you do something like send a UDP packet. You can call from the bindings into the core and say, "I would like to send this packet." And then once the core bottoms out at like ethernet and needs to send a message to, you know, an Ethernet driver, it then calls back into the bindings and says, "Hey, I need to talk to the driver. Here's the Ethernet frame that I want to send." So we can call into the core and the core can call back out into the bindings.

But that whole system is entirely synchronous. There's no async involved. It's just straight-line code. In that UDP example, the bindings would call into the core and it would just go straight through all the way through. At some point, the core would call back out into the bindings and say, hey, I want to send this ethernet frame to the driver. That call would happen, right? Either inline, the bindings would actually do the communication with the driver, or maybe it would just queue that work for later.

And then it would return and the function call would return all the way through the core and back out into the bindings where it was originally initiated. The bindings do have async. For various reasons, it's important to have async there because there's a lot more asynchronous stuff going on. The communication mechanism that Fuchsia uses to communicate between processes is very naturally asynchronous. And so it makes a lot more sense for there to be async code there.

So there's async code initiating synchronous calls into the core; that's basically the way to think about it. But inside of the core, there is no async. And again, just to reiterate, the reason is that we wanted to be able to have a very clear mental model of how control is flowing through the core so that we can reason about performance and like resource liveness, right? When are certain resources live or dead? What is the possibility of, you know, deadlocks and stuff like that? And generally speaking, I have found that it is a useful, you know, sort of rule of thumb to avoid async until you actually need it.

Once you need it, Rust async is super cool. And I don't want this to come off as like me denigrating the folks who did async. Async is absolute magic. It's so, so, so cool, but it is a very sharp tool and you should only use it once you know that you actually need it.

Glen

Okay, that's fair enough. But then I do wonder, as you said, the driver or whatever, like the bindings call the core and the core can call back into the bindings. Like how do they then make sure that, for example, that stuff doesn't block everything else? Because then, I think—is it because they give it an entire thread to do what it wants, or how does that work?

Josh

Okay, so the most accurate answer is: I don't know. Most of this stuff was written after I left. My rough understanding from having talked to Bruno since he took over the project is that basically they have a queuing mechanism where they will queue packets or frames to be sent to the driver at some point in the future. Now, naturally a thing you might imagine, and I don't know whether they do this, this is me just speculating.

But naturally, a thing you can imagine is you could do a fast path. You can say, if there is room in the buffer that I share with the driver at the point at which the core calls into the bindings and says, please send this frame, maybe they opportunistically, as a fast path, go ahead and write into the buffer. I don't actually know whether they do that. You can imagine them doing that. Certainly, they're not going to block. You're definitely not going to block on that operation. You're going to queue it for later and then immediately return up through the core.

Just to give a sense of numbers here, Netstack3 aims to be a general-purpose networking stack. You want it to be able to use on any system. And Fuchsia is a general-purpose operating system. You want to be able to use it on any system. And so you want to be able to have this stuff be as performant as possible so that someone in the future, let's say there's some like—I don't know, this is complete speculation. I'm making this up. This is not like real, you know, this is not a real-world example. But like, let's say that a decade from now, someone says, "You know what? I want to build a router. I want to use Fuchsia to build it." Well, if you're going to do that, if you're going to use Fuchsia to build a router, then you would need to be able to turn around a packet in, I think the numbers are something like a microsecond, right? Like a modern, you know, take your random, you know, Netgear router that you have in your home networking setup or whatever. You have about a microsecond, right? A millionth of a second to receive a packet, parse it, decide what you're gonna do with it, decide where you're gonna forward it, turn around, serialize the new version of that packet, forward it out.

You have about a millionth of a second budget to do that entire thing, right? That entire operation. So the point is that if Netstack3 has the goal of just being like a general-purpose networking stack that people can use for general-purpose networking stuff, then you need to be able to like hit those performance goals, right? And so if you have a millionth of a second to turn around a packet in the most extreme case, then you definitely cannot be blocking in the core.

And so the idea is that, yeah, you want all of the extra scaffolding around that sort of stuff to happen in the bindings, where you can have more opinions about what sorts of performance trade-offs are acceptable, but you definitely do not want to be blocking the core. And you might say, "Well, maybe if this thing blocks in the core, then that's fine." Like it blocks trying to send to the driver, but that's fine because a different thread could come in and do work at the same time. True, but what if the core has like, you know, locked a mutex in the interim, right?

And so in fact, you're blocking like other resources at the same time. So generally speaking, the core tries to be as much of a “get in, do your work, and get out” system as possible. This, by the way, for those of you who are interested in like networking optimization stuff, is generally speaking the direction that high-performance networking has gone over the past decade. If you read design docs from the Linux kernel, for example, there's a really great text file. There's a really great design doc that someone wrote that just describes all of the high performance stuff that the Linux net stack does.

And a lot of it basically entails, how can we be as dumb as possible? That's basically the idea, right? Like stop trying to synchronize between different cores and different threads. Stop trying to be fancy about stuff. Just do the dumbest possible thing—no work at all. Get in, do your work, get out as quickly as possible. There are various things that this looks like where you can like pin different sockets to different cores and you can like do, you know, hash-based bucketing so you know a priori which core you're going to send the packet to.

There's all sorts of fancy stuff. But all of it boils down to get in, do your work, get out as quickly as possible. And so for that reason, async is a bit antithetical to that because it just robs you of the ability to make your own decisions about what that execution flow is going to look like. Again, this is a very niche case. The netstack use case is very niche, especially from a performance standpoint. So there are lots of other cases where async is a very appropriate tool for the job. As I said before, if you're doing a web server and you're just going to have a bunch of people coming in and your web requests are very I/O-bound, someone makes a request and then you need to issue a read to the database and that's going to block for a while and yada, yada, yada, async is the obvious choice there.

But if you're doing low-level system stuff, it is probably a good idea to think very deeply about the control flow inside of your application, in which case async is something you should think carefully about. I'm not saying don't use it, but just make sure you understand how it's going to affect your control flow.

Glen

Yeah, for sure. I mean, first of all, it's very exciting to talk about this. Like it always makes me feel very warm. But yeah, it's a...

Josh

Yeah, Rust is a good... Rust is a warm and fuzzy language. That's why I like it.

Glen

Yeah, but even the networking world, I mean, it's kind of like for me until like eight years ago or seven years ago, networking was always an abstract concept. Like I was using it when I had to and I knew how to open sockets and I knew how to manipulate them. I knew how to write to them. I knew a lot about like all even writing our own protocols for the games because that's what they did. I don't know why, but everybody was inventing their protocols.

So I have a lot of experience with that, but it's only once I really got into, I don't know, security and data extraction and the web world and all those kinds of stuff and then proxies and yeah, again, security in another kind of capacity that I really got into the networking world. Yeah, but still, talking to someone with your experience working in those kinds of systems, I feel like there's still a whole world to explore where I definitely haven't scratched the surface yet.

Josh

It never ends. I mean, despite the fact that, you know, I was able to spend nearly half a decade working on this stuff. In my time there, I never got to touch a lot of the higher performance stuff. I never actually—you know, the current synchronization model is basically just mutex-based for Netstack3. I was always really excited about some hypothetical future in which we could go like super hardcore and do the core pinning stuff and like, map sockets to different threads and whatever.

We actually wrote up a design doc for that once and basically just, there were so many correctness edge cases that it just, we ended up deciding that it was too complicated to justify it in the short term. But man, you can go so deep. At least while I was there, we never did any of the hardware offload stuff, having the hardware do the checksumming for you or like coalesce TCP segments for you.

Glen

Mm-hmm.

Josh

I mean, man, yeah, it's easy to understand how you could make an entire career in this industry and never actually, never get to the bottom. It's very cool.

Glen

I mean, yeah, I mean, that's what I live for. I love to learn. I soak it like a sponge. Now, we already mentioned it a bit before and it's also written in your README, which by the way, I mean, in one respect, I love the README of zerocopy. Like it's very beautiful. At the same time—and maybe it's just me, but I feel as an outsider, if I hadn't read the actual code, I would still find it very difficult to know how to apply it because it doesn't really contain any examples—or examples I could easily find like for example one might expect, yeah, this is like a struct and that's coming from this kind of slice and that's how it's... And then you see all these concepts coming together.

Well, now I feel it's like a laundry list of concepts, and then it's a lot of work on the reader I find to then put it together. Maybe that's part of it because you need to read and understand that. By doing that work, kind of like it's maybe it's like a mathematical proof where you have to go through the proof yourself and only then you understand it. You couldn't just read it. Is it like that or is it just because no time?

Josh

You were being very kind. No, I mean, the answer there is twofold, right? I mean, a large part of it is just like, zerocopy is two engineers and we don't have time is the real answer. I would love to invest more time in documentation. And certainly, hey, let this be a call. Those of you who are still with us an hour and a half into this podcast or however much this gets edited down to, I'm seeing an hour and a half on my screen here. Those of you who are with us this far in the podcast, we would love open-source contributions.

At zerocopy, we love, love, love it when people come and contribute. We're always looking for open-source contributors, always looking for people to use it in new contexts and give us feedback, all that jazz. Yeah, the documentation thing is largely just a lack of time. There is somewhat of a challenge here. It's not entirely just a lack of time. There's a little bit of this, which is just that it is a new way of thinking about how to do systems programming that a lot of people aren't really used to.

And so we oftentimes find that people come in and they are... It's not just, "Do I do X?" But fundamentally, I don't even understand what it would mean to solve my problem. And so zerocopy really does have this sort of like... "We will give you the tools," but we can't just document what tools we're giving you, because oftentimes people don't know what tools they need. Zerocopy really is a very powerful Swiss Army knife. It's very powerful.

It'll let you do really cool stuff. But it's not a high-level framework. It's not a high-level network packet parsing framework or something like that. We have aspirations to build such a thing on top of zerocopy at some point in the future. But right now, our time has been entirely taken up just building the Swiss Army knife, basically—not even using it to build higher-level stuff.

Glen

Well, I mean, definitely reach out to me if you get to that. Maybe it could be like a partnership, I wouldn't mind.

Josh

Yeah, I mean, yeah, certainly if there are people out there who are interested in helping to build a high-level framework, that is certainly something we are interested in. As I said, we are I/O-bound, so to speak. We are not limited by a lack of ideas. We are limited by a lack of time. So yeah, If anyone's interested—if you have some crazy idea for a feature you could add or high-level framework we could build, please, please, please, please come talk to us. I would love to talk to you.

There's a Discord link on our docs.rs homepage, so that's a good way of getting in touch. Yeah, generally speaking, I think people are just not quite familiar with what tool they want to be reaching for. And so one of the big challenges is deciding how much we document what it is that we provide and how much do we try to be prescriptive about if this is your problem, here's what you should reach for. If this is your problem, here's what you should reach for for different problems. And part of that is it's actually kind of difficult to enumerate the entire space of possible things that people could use zerocopy for because it's extremely diverse. At a high level, it's basically just anytime you care about like the exact shape of what's happening in your memory, zerocopy might help you.

And so that's a pretty wide, you know, that's a pretty wide problem definition. I remember once I was giving a talk internally about zerocopy and someone came up to me and said, "Hey, I want to add an example to your list of example use cases." This person was doing video game hacking. So they basically had a video game and they wanted to hack it in order to get extra statistics about their own gameplay. And so they were reaching into the memory of this video game and pulling stuff out. And they were using zerocopy to make sure that they were parsing the memory of the video game correctly. It's not something that had ever occurred to me that was a possible use case, but it sort of makes sense.

So that's a big problem for us: it's not super feasible for us to just list all the possible use cases, because it would sort of be like saying, you know, "What are all the use cases for a for loop?" It's like, "I don't know, man. It's a for loop; you use it for a bunch of stuff." So that's sort of the challenge, but also, as I said at the beginning, that's not an excuse. We should have more documentation, but it's, you know, there's never enough.

Glen

Yeah. And then there is like Project Safe Transmute. And you say it's kind of like about moving zerocopy into the Rust compiler. Would it ever be the goal for this not to be distributed as a crate—zerocopy—and instead just be in the compiler?

Josh

Yeah, so... Yeah, so I suspect that we are... Well, let me give some background. The short answer to your question is there's probably not a future in which zerocopy and crates like it—bytemuck is another big one here—go away entirely, but there is hopefully a future in which they get much smaller.

So basically zerocopy and crates like it, for the most part, solve the problem of having a pile of bytes and wanting to convert it into a particular type, or having a particular type and wanting to look at its bytes. So it's a bidirectional transformation between bytes and some other type.

It turns out that you might want to do more powerful things than that. So you might actually want to go from a particular type, like some particular type T, and convert it into some other type U. And that is something that zerocopy is not good at reasoning about, because it's no longer a piecewise operation. So zerocopy today will let you go from T to U, but only if you can intermediate by going to a pile of bytes.

So if you can go from T to a pile of bytes, and then from a pile of bytes to U, then zerocopy will let you compose those two operations and just do them together as just T to U. But there are a lot of pairs of types where that's not valid. So for example, if you have like an enum that has three variants, and you want to go from bool to that enum, that should be allowed, right? A bool can be either 0 or 1, and then maybe I have some enum that is also one byte long that can be 0 or 1 or 2. In principle, I should be able to go from a bool to this enum type. The problem is, zerocopy, the only way that it knows how to reason about that operation is to go from bool to a pile of bytes.

Once you have a pile of bytes, now you just have a u8. And from zerocopy's perspective, if you try to go from a u8 to this enum, you're basically telling it, hey, can I take any u8 at all and convert it into this enum? And the answer is no. What if it's the value four or five or 27 or 38 or whatever? So, just saying I can go from T to a pile of bytes and a pile of bytes to U limits the number of types that you can actually support, the number of conversions that you can support.

So that's the reason that zerocopy and crates like it are limited. So the Safe Transmute project is a project that attempts to support that more powerful class of transmutations in the compiler itself. Basically the way that we do this is we have a representation inside of the compiler of the layout of a type. So there's a way that the compiler can reason about what is the set of every possible bit pattern that I might actually see for this particular type. And then it has a way of asking basically a subset question, it's doing a subset query. If I have a T and I'm trying to transmute it into a U, is the set of possible bit patterns in the T a subset of the set of possible bit patterns in U, right? Is it guaranteed that no matter what T I have, it's going to also be a valid instance of U if I just convert the bytes from a T to a U?

We're building a way in the compiler to do that transformation, right? To ask queries of that form. It turns out that is actually like a very interesting CS theory problem. We've had to develop some novel algorithms and there's been some really interesting engineering there. So it's a long-term effort. But the goal is that eventually the compiler will be able to answer questions of this form for you. Once it has done that, then you can rip out all of the analysis part of zerocopy, right?

So the custom derives can basically just go away and be replaced by this built-in compiler analysis. Now the high-level stuff in zerocopy still stays there, right? So basically what the compiler does is give you this trait implementation that you could use to say, "is this a valid transmutation?" But then you actually still have to write the code that does the transmutation. So all of the stuff in zerocopy that says, if this is a valid transmutation, then here's a bunch of extra stuff you can build on top. That stuff is gonna stick around probably, right? That stuff could in principle be moved into the compiler or at least into the standard library, but there's a large set of features there and we're probably not gonna move all of it. Certainly not anytime soon.

So more likely the short-term world is going to be, we add this analysis to the compiler and then we remove or strip down zerocopy-derive, but the zerocopy crate itself stays largely the same and maybe gets to grow new fancy features that are built on top of this.

But that's sort of the future that we expect.

Glen

Okay, yeah, that makes sense and still very exciting. So is this like for now a very theoretical working group or is this something like, no, this is getting very concrete and soon we will land these kind of things.

Josh

Yeah, so it's concrete in the sense that we have an existing experimental implementation. So you can go to nightly Rust today and find, in the mem module, a trait called, I think it's currently called TransmuteFrom. We have renamed it at various points. I can go there right now while we're talking and answer that question. But basically there's an existing nightly feature that provides this query. So it's in the compiler, but the short answer is that it's unstable. There's a good amount of...

Okay, here we go. I looked it up, yes, its current name is mem::TransmuteFrom. So it's unstable, but there is a fairly full-featured implementation in the compiler. But there are a lot of edge cases. So currently we don't handle like non-zero types, for example. That should be a relatively easy fix. Again, anyone who is interested in open-source contributions, we'd love to have you. There's a lot of like pretty easy low-hanging fruit for people who are already familiar with contributing to the compiler to add extra features.

Then there are optimizations that we would like to do. So currently, if you represent an array spanning a gigabyte of memory or something like that, you would hope that the representation would basically just be like, you know, the representation of a u8 followed by a number that says there are a billion of these.

That is not what we do. Currently, if you give us an array of a billion u8s, we are going to instantiate in the compiler a billion representations of u8, which is very stupid. So there's like low-hanging optimization fruit like that that we just need to like, you know, do the obvious optimizations. Eventually, the goal, of course, is to actually put this forward and try to stabilize it and all that jazz. We're not quite there yet, but we're hoping to get there. It is something you can experiment with today. If you are using Nightly Rust, you can try to play around with this with your types.

As I said, you may run into edge cases and stuff, but that's exactly what we want to see. We'd love people to experiment with it and give us bug reports. So yeah, the long-term goal is to put this forward formally for stabilization. But there's a good amount of engineering work between here and stabilization—

Glen

Okay, very cool. And then I feel we covered a lot of ground. I also feel we could talk a lot more, but I do think we should wrap up soon. Is there something that we didn't discuss yet or maybe we touched on it, but you think you might want to go a bit deeper into it before we leave?

Josh

Sure. Yeah, so I think, like, I would say that I had a few notes written down of just like stuff I wanted to touch on at some point. These are gonna be somewhat disconnected from one another, but it's just sort of the grab bag at the end. One thing is if you are interested in systems programming and you want to go deeper on stuff, I happen to think that writing a network stack is one of the most interesting and challenging systems-programming hobby projects that you can do. You would be surprised by how tractable and approachable it is to implement IPv4 and TCP for yourself. Buy a networking textbook that has an explanation of those things. Read the RFCs.

They are not the world's most readable documents, but they're not extremely arcane. Try to implement it for yourself. It is... It's a wonderful challenge in how to build large-scale software. I think one of the most educational experiences I had during college was being tasked with implementing this, even in an academic context. So if you're interested in systems projects, I highly, highly recommend it. It's a lot of fun.

The other thing I get asked a lot is whether there is a sort of high-level philosophy or attitude behind the work that I've done, like in Netstack3 and zerocopy and Safe Transmute and stuff like that. Like, how can I take a similar sort of engineering approach to my own work? The answer that I usually give—and this is definitely something that I recommend to a lot of people—is that Rust in particular, but this is applicable in theory to lots of other languages, gives you the ability to be extremely ambitious about how nice your software is.

And so what I would say is whatever you're doing—whether abstraction design or writing a library—let yourself be motivated by the functions and the features you need to provide, obviously, but also let yourself dream big about how nice it could be to use your API, right? Let yourself sort of engage in daydreaming about, man, wouldn't it be nice if I had a system that let me do this? If it were just this easy, right? If it just worked out of the box or it were just this straightforward to do this particular operation. Let yourself have that dream. And then the challenge to you as an abstraction designer is to connect the two things, right? To say, is there some way for me to provide the functionality that I need to provide, but in an API that is actually this nice?

Rob Pike has talked a lot about this in talks he's given at various conferences. One of my favorite talks that he's given is called “Simplicity Is Complicated.” He has one of my favorite quotes: “Simplicity is the art of hiding complexity.” Basically the idea is your job as an abstraction designer is to understand the thing that you are implementing so well that it's possible for you to provide a simple API. The simpler the API, the more deeply you have to understand the problem in order to be able to produce that API.

And so that's really my advice: let yourself be ambitious about how simple your APIs can be and about how easy and pleasant to use they can be and understand that in order to satisfy that ambition, you are going to have to just sit there and think deep thoughts about the problem for a long time and like mull it over in your head and really, really get to the point of deeply understanding how you could possibly shape an API that is as nice as you want it to be, basically. So that's sort of the— like, you know, I encourage people to be as ambitious as possible on that front. And if you get there, then that's how you end up with something like zerocopy that you're sort of scratching your own itch.

But then eventually you end up solving a problem that other people also have and, to use a somewhat overused phrase from the product design world, you delight your users or whatever, right? I think that... That phrase gets a lot of poo-pooing from more hardcore systems-engineer types. But I do think there's something there. The idea that you use a product that you really like and you just have this sort of emotional feeling of relief, right? You're just like, it's so nice. I don't have to think about this anymore.

That's the feeling you want your users to have. You want them to just sort of have that like, this is the thing I've been wanting to use my whole life, but now it's here and now I can just relax. Let yourself be ambitious about that. I think that's... That's all I got.

Glen

Yeah, only that. It's wonderful advice. Yeah, and I feel especially in a language like Rust, this is definitely a place where you can be ambitious because for example, if we think about the framework Rama, which I was mostly writing alone but now has more and more contributors, I would never have built that if not for Rust, especially not on my own because...

So one thing I often say to people is like, Rust, they often focus on like performance, et cetera. But what I really like about Rust, thanks to the type system and many other things, is how I can refactor things and change things. And I'm not scared to change it because it will, like, it will really just work out. I've done several large refactors and it just worked out like no scary surprises. If I think back to my long history in C++. Despite liking the language and liking to work in it, I would never have pulled that off.

Like I would have introduced so many bugs, so many segfaults, so many kinds of problems. Like it just wouldn't have worked. And so yeah, I like your advice a lot because it's something you can actually do in a language like Rust, for sure.

Josh

Yeah, no, exactly. I mean, this is something that I have seen. It's a very powerful philosophy because I say that specifically in the sense that I have seen other people take that philosophy and run with it. So we had this guy, Alex, who worked on Netstack3 for a while, and he did one of the coolest projects I've ever seen where he basically figured out a way to encode lock ordering in the type system. So basically the idea is like you have a bunch of mutexes and what is the order in which you acquire those mutexes, right? He figured out a way to put that in the type system and prevent the Netstack3 code from ever deadlocking.

Basically, if you attempt to acquire locks in an order that could result in a deadlock, it will cause a compilation failure. So cool. That was just an example of basically someone on the team who had seen this philosophy of trying to be really aggressive about how nice your APIs can be. And he was like, yeah, I bet we can do this. And again, it was that same, what's the goal? The goal is I want this to be a magical experience. I want to be delighted by how easy it is to use.

And he just sat there and he banged his head against it for a while. And then he came up with this really clever trick that now means that Netstack3 is never going to have a deadlock. And if you want, by the way, for the sort of like more business-minded people in the room or in the audience, if you are interested yourself or you're interested in arguing to leadership at your company that this is something that's worth doing, Netstack3 has this crazy statistic that I think really encapsulates just how this is not just a thing engineers do because we like correctness for its own sake and it makes us happy or whatever. It's not just like a nerd snipe thing. It actually really translates into business value.

In the first year and a half of Netstack3 being dogfooded—that is, used internally by the team as a real networking stack by people on real devices, sitting in their homes and using it to do development. Normally, the goal of rolling out something for dogfooding is that you expect to find a bunch of bugs that you had not found during testing. You'd expect that to be in the order of hundreds or something. Over the course of a year and a half, you're going to see this long stream of bugs, you're going to burn them down, yada yada yada. In the year and a half that Netstack3 was in dogfooding, and again, this is after I left, I'll sort of refer to them in the third person here—the team encountered four bugs in the field. Four. Which is just a stupidly low number, an absurdly low number.

And if you think about the cost from a business standpoint, especially for a networking stack, right, I mean, you know, if you have a device in the field, and its networking stack has a catastrophic bug, that device is effectively bricked because it can't connect to the internet to take a software update. That's how important a net stack is. So getting it right the first time is crucial. And you think about the amount of developer time that is saved by not having to do debugging, you think about the normal lifecycle of a product. If you are a software engineer, especially if you've worked on a C or C++ code base, how much of your time do you spend fixing old bugs as opposed to writing new code? Imagine what it would be like if most of your time was just spent writing new code.

That's the value proposition from a business standpoint of moving slowly and carefully in a language like Rust, you know, to use the phrase that the Marines have, slow is smooth and smooth is fast, right? If you measure on the order of lines of code written, maybe you're moving more slowly, but that's not actually the right measurement, right? The right measurement is like features shipped stably to production. And my thesis, which is not based on actual numbers because people haven't studied this yet, but my experience intuitively is that if you measure over a longer time horizon, Rust is not just a safer, more precise, more careful language, it's also a much faster language because you don't waste time in development fixing problems that you introduced earlier in development.

So if you're trying to sell this to leadership at your company, give them that quote, four bugs in a year and a half of dogfooding, with a small team that had not been working on it that long. Rust is a really special thing and it translates into dollars and cents. It's not just nerd sniping for nerd sniping's sake.

Glen

Yeah, I couldn't agree more and I will definitely use that quote. Thanks for that. So, with that being said, I want to thank you very much for your time and I wish you a very good life ahead of you.

Josh

Yeah, of course. Thanks, Glen. This has been an absolute pleasure. Anyone else who's thinking of doing this, take Glen up on his invitation. It's been a great time. Appreciate it.

Elizabeth

Netstack.fm is brought to you by Plabayo, building secure, open, and resilient infrastructure with Rust, protocols, and purpose. This show is also made possible by Rama, the open source networking framework. Plabayo offers service contracts and welcomes sponsorships to keep building and supporting its ecosystem. The theme music of this podcast was composed by DJ Mailbox. If you enjoyed this episode, don't forget to subscribe on your favorite podcast platform and leave a five-star review. It really helps others discover the show. Thanks for tuning in.

We'll see you next time for the next handshake.

Safety in an `unsafe { world }`

2026-07-27T00:00:00+00:00

This post is a nearly verbatim adaptation of my RustConf 2024 talk. I've kept everything in the present tense as it was in September 2024. Towards the end, I've also included a bit of content that I had planned on presenting but had to skip during the presentation for time.

Watch the talk · See the slides · See the full references

Netstack3

So what is this talk about? Well, I'm going to bury the lede a bit.

First, let's talk about Netstack3. Netstack3 is Fuchsia's next-generation, pure-Rust networking stack. It aims to replace Netstack2, which is written in Go.

I started Netstack3 about six years ago, and led its development for four years, and then for the past two years it's been led by a different team. I haven't been on the project for two years, so obviously this is beyond me now – it's a huge team effort – I'm going to be bragging a lot about those folks. There are some really good engineers who work on that now.

Writing a networking stack is a serious undertaking. It's responsible for almost all network traffic in the entire operating system. It implements dozens of protocols, each of which is specified in documents that can run into the hundreds of pages. And it's the first line of defense against any attacker who isn't physically sitting in front of a device.

It's also just big. As I said a second ago, it's taken us six years to get to this point, there have been plus or minus ten developers that whole time, and it's split across 63 crates totaling 192,000 lines of code. That's more code than the top ten crates on crates.io combined.

Over the past year, the team has been preparing to launch Netstack3. Those of you with networking backgrounds will know that you don't just deploy networking code into production.

Networking code is famously difficult to test, and so you have to assume that, despite your best efforts, your code is riddled with bugs.

For a project of this scale — deploying an entirely new, ground-up rewrite of a netstack — you would expect to dogfood in the field for months or maybe even years before shipping to real users. In that time, you would expect to uncover tens to hundreds of bugs that you hadn't seen during development. And only once you had burned down most of those bugs and seen relatively stable behavior for a while, only then would you finally deploy to real users.

So let's talk about what the process has looked like for Netstack3. For 11 months, the team has been ramping up a dogfooding program. At peak, that program has seen about 60 devices running nearly 24/7 in developers' homes.

Again, if this were any other netstack, we would have expected to uncover a giant mountain of bugs in that time. So, over the past year, how many bugs did the team uncover in the field?

Three.

Note from the future: As of posting, Netstack3 is now in production, running on millions of devices. Immediately after its first deployment, the team observed a ~20x lower rate of crashes per million devices per day and a 50% reduction in memory usage compared to its predecessor, Netstack2.

The three dogfooding bugs (actually four – the last observed failure was due to two separate bugs) were:

419219829, fixed in 1282284: An operator-precedence mistake defined WindowSize::MAX as 1 << (30 - 1) rather than (1 << 30) - 1. Netstack3 therefore imposed an artificially low TCP receive-window maximum and could panic on connections with windows above roughly 500 MB.

420719068, fixed in 1286767: The DHCP client treated ENOBUFS as an impossible out-of-memory condition and panicked. On a packet socket, however, ENOBUFS can legitimately occur when the network device’s transmit queue is full.

422307328 was one observed failure caused by two bugs:

Fixed in 1334968: Netstack3 discarded IPv6 Neighbor Advertisements that omitted the Target Link-Layer Address option, even though RFC 4861 permits the omission in some unicast cases. This prevented otherwise valid reachability signals from updating the cached neighbor state.

Fixed in 1335018: Netstack3 propagated a Path-MTU error to TCP only when the error changed the PMTU cache. Repeated Packet Too Big or Fragmentation Required errors were therefore suppressed after the first cache update, preventing TCP from updating its MSS correctly.

So this talk is going to be something of a flashback.

Netstack3 was designed around a particular methodology for how to architect robust systems. Hopefully I've convinced you that we did at least something right on the robustness front.

In a sentence, that methodology says:

Buggy programs don't compile.

Obviously, I am far from the first person to suggest this methodology. Here is just a small sample of crates that use this methodology somewhere in their APIs:

ghost-cell, session_types, nalgebra, mundane, indexing, zerocopy, and bytemuck.

And here is a small sample of what has been written on the topic, both about Rust and about other languages:

References are like jumps, by withoutboats
Parse, don't validate, by Alexis King
Type Safety Back and Forth, by Matt Parsons
Ghosts of Departed Proofs, by Matt Noonan
The Typestate Pattern in Rust, by Cliff Biffle
Compiler-Driven Development in Rust, by No Boilerplate
Some notes on Rust, mutable aliasing and formal verification, by Graydon Hoare

So my goal for this talk is not to introduce a new idea.

Instead, my goals are:

First, I'm going to propose a concrete but general framework that attempts to unify all of the different ways this methodology shows up in practice, and explain them in terms of the same basic concepts.

Second, I'm going to walk through two examples — one from the standard library and one from Netstack3 — and show how we can explain them in terms of this framework.

Unfortunately 25 minutes isn't enough time to present more than two examples in the depth I'd like. I had to leave a lot on the cutting-room floor when I was writing this talk. But I promise that these two examples only scratch the surface of what's possible. The full list of references contains a surprising diversity of problems which this methodology has already been applied to.

What do we mean by “buggy programs don't compile”?

On its website, Rust advertises three properties: performance, reliability, and productivity. Let's zoom in on reliability.

The website says:

Rust's rich type system and ownership model guarantee memory-safety and thread-safety.

In other words, out of the box, Rust guarantees that "buggy programs don't compile," but only if we restrict our definition of "bug" to memory bugs or threading bugs.

Don't get me wrong: memory safety and thread safety are huge on their own. For example, Netstack3 is able to pull all sorts of crazy buffer-sharing tricks in the name of performance that would be wildly dangerous in C or C++. I gave a talk on those techniques at Rust Belt Rust in 2018 if you're curious about the details.

But that's only half the story.

Many critical bugs that we'd like to prevent are neither memory bugs nor threading bugs.

Take TCP, for example. If you want to implement what RFC 4614 refers to as "basic functionality" for TCP, you need to implement six different standards that run a combined 270 pages.

Add in the "recommended enhancements" and you're up to a total of 18 standards spanning 476 pages.

And that's just one protocol. Add Ethernet, ARP, NDP, IPv4, IPv6, ICMP, IGMP, MLD, UDP, and a host of others, and all the subtle interactions between them, and you start to realize that memory and threading bugs are just the tip of the iceberg of all the fun and exciting ways that you could fuck it up.

That's why uncovering only three bugs in almost a year of dogfooding is so surprising. There's just no way that memory safety or thread safety alone would get you that level of robustness.

So the points I'm trying to make are:

Out of the box, Rust guarantees that "buggy programs don't compile" only with respect to a small subset of the bugs we actually care about.
Netstack3 would not have been able to achieve the level of robustness that it has by relying only on what the language provides out of the box.

However, what Netstack3 has done — and what I believe any project can do — is to extend the Rust language to guarantee freedom from almost any class of bugs, not just memory or threading bugs.

This is what I mean by "X-safety" in the talk's abstract.

As it stands today, Rust provides a core set of abstractions — lifetimes, references, RAII, traits, etc — and guarantees various safety properties related to those abstractions.

But the majority of abstractions in the ecosystem are provided by libraries: JSON, random-number generators, cryptography, regular expressions, logging, time, TCP, PNG, SQL, command-line interfaces, linear algebra, browser APIs, and so on. Those libraries generally don't provide the same degree of safety with respect to their abstractions that we might expect from the language.

But it doesn't have to be that way.

As we'll see in the following examples, Rust's core abstractions alone are sufficient to express almost any safety property that we might care about. I claim that every library can and should provide the same level of safety with respect to its abstractions that we already expect of the language itself.

A general framework for safety

To explain the framework, we're going to turn to the humble binary tree:

struct Node<T> {
    left: Option<Box<Node<T>>>,
    right: Option<Box<Node<T>>>,
    value: T,
}

For our binary tree to be correct, we of course need memory safety, but we need another safety property as well. We have to ensure that the tree is ordered: every value in the left subtree is less than our node's value, and our node's value is less than every value in the right subtree.

Rust can guarantee memory safety, but it has no visibility into this ordering requirement, and so it can't enforce it for us. So how do we make sure that we never produce an invalidly-ordered tree?

We use the concept of an invariant.

First, we document an ordering invariant that we expect to always hold for any Node:

struct Node<T> {
    // INVARIANT: All values in `left` are less than `value`.
    left: Option<Box<Node<T>>>,

    // INVARIANT: All values in `right` are greater than `value`.
    right: Option<Box<Node<T>>>,

    value: T,
}

Next, we ensure that every time we create a new Node, the ordering invariant holds:

impl<T> Node<T> {
    // Post-condition: `Node`'s internal field invariants hold.
    pub fn new(value: T) -> Node<T> {
        Node {
            left: None,
            right: None,
            value,
        }
    }
}

And finally, we ensure that every time we modify an existing Node, we preserve the ordering invariant, assuming it held to begin with:

impl<T> Node<T> {
    // Pre-condition: `Node`'s internal field invariants hold.
    // Post-condition: `Node`'s internal field invariants hold.
    pub fn insert(&mut self, value: T) -> Option<T> {
        // ...
    }
}

Now imagine that you are code outside of this module. (We'll restrict ourselves here to safe code; obviously unsafe code can do whatever it wants.)

I claim that, from the perspective of safe code outside of this module, there is no difference whatsoever between the safety invariants provided by the language and the ordering invariant provided by this module.

If you write code which, if run, would violate Rust's memory-safety guarantees, Rust guarantees that the code will not compile. By a similar token, if you write code outside this module which, if run, would violate the ordering invariant, then similarly Rust guarantees that the code will not compile.

In other words, by structuring our code in this way, we have in some sense "taught" Rust about a new safety property that it didn't know about before.

We can use this Node example to explain the framework. It has three components.

Definition

First, definition.

We define an object that the Rust type system can reason about. In this case, we define the Node struct. We attach to this object a safety property that Rust cannot reason about. In this case, that is the ordering property.

Because Rust can't reason about this safety property, it's up to us, as the authors of the abstraction, to make sure never to violate it. This brings us to the second component.

Enforcement

Second, enforcement. Here, we enforce that our safety property is upheld – that responsibility is on us as the programmers.

In this case, we use field privacy – we make sure that our struct fields are private. If they were public, external code could modify them and violate our safety property.

Next, we make sure that all the code we write preserves the safety property. In this case, the methods that modify the tree are responsible for maintaining its order.

Consumption

Finally, consumption. This is where we get the bang for our buck.

We write code that consumes the safety property as a precondition, and is only correct in virtue of that safety property being upheld. In this case, our methods can find the correct location in the tree for a particular value in O(log N) time, without having to traverse the entire tree.

That optimization is only valid because we are guaranteed that the tree is ordered. And the tree is only guaranteed to be ordered because we structured our code to define and enforce this ordering property so that we could later consume it as a precondition.

With that framework in mind, Let's get to our first example.

Example 1: Thread safety

Our first example is from the standard library and concerns thread safety.

What I'm going to demonstrate is that not only can libraries use the language's core features to build new safety guarantees, but the standard library itself already uses this technique.

In fact, the thread safety that Rust advertises as one of its core features is not actually a language feature at all. It is implemented in the standard library and, in principle, could have been implemented in a third-party crate. It's just convenient to have it in the standard library.

Imagine that you're writing spawn. spawn is a real function from the thread module in the standard library (although I've stripped down its signature for simplicity):

pub fn spawn<F: FnOnce()>(f: F) { ... }

It takes a function and spawns a new thread that will run that function.

If we think about how we might implement spawn, we immediately run into a limitation in the language: the Rust language itself doesn't provide any mechanism to interact with threads. Threads are an operating system concept, and so creating a new thread requires going outside the language and making use of an API provided by the operating system itself.

For example, on a POSIX system, this might require calling libc's pthread_create function:

pub fn spawn<F: FnOnce()>(f: F) {
    unsafe { libc::pthread_create(/* ... */) };
}

This is where we run into the first core abstraction that we'll need to use as a building block.

Since libc functions like pthread_create are implemented outside the language, Rust has no visibility into their behavior, and so it has no way of guaranteeing that calling such a function won't violate memory safety.

This is why Rust introduces the notion of unsafe code. unsafe code is code that Rust can't guarantee won't violate memory safety. That doesn't mean it definitely will violate memory safety. It just means that, on its own, Rust isn't smart enough to prove that it won't. Instead, Rust has to rely on the programmer to do the heavy lifting and confirm that the unsafe code is correct — or sound.

This is the purpose of an unsafe block like the one above. Inside an unsafe block, Rust gives the programmer free rein to perform operations that Rust can't reason about. In exchange, the programmer takes responsibility for upholding memory safety. If they mess it up, all bets are off.

This is our first example of using a core abstraction to extend the power of the language. Rust itself provides unsafe functions and unsafe blocks, but it doesn't provide any mechanism for interacting with threads. However, by using unsafe in combination with external APIs like pthread_create, we can "teach" the language about threads.

As written, though, this implementation of spawn is unsound.

Not all values are thread-safe, meaning that not all values are sound to send between threads. Think of the difference between Arc and Rc. As written, there's nothing to stop you from passing a closure that captures some value that is not thread-safe.

So the standard library encodes the notion of thread safety in the type system.

First, we introduce a new trait called Send:

/// # Safety
///
/// `Self` is thread-safe.
pub unsafe trait Send {}

This is the standard-library Send that you all know and love.

Note that Send is an unsafe trait. Just like unsafe functions, unsafe traits tell the type system: "This trait has safety implications that you can't reason about." Just as calling an unsafe function can only happen inside an unsafe block, implementing an unsafe trait requires an unsafe impl block. The programmer must opt into taking responsibility for upholding memory safety.

Since the language can't reason about the meaning of Send – it's just an opaque trait with some name – we have to document its meaning in prose like we've done here. In this case, we say that a type can only be Send if it's thread-safe.

Now that we've introduced this trait, we can use it as a bound elsewhere in our program:

pub fn spawn<F: FnOnce() + Send>(f: F) {
    unsafe { libc::pthread_create(/* ... */) };
}

By requiring that F implement Send, we can make spawn sound. There's no longer a risk that a programmer could accidentally call spawn with a non-thread-safe value, because that wouldn't type-check.

So we've introduced this Send trait that encodes the notion of thread safety, but we haven't actually implemented it for anything. Let's do that.

First, we can implement Send for any built-in type that we know is always thread-safe:

unsafe impl Send for u8 {}
unsafe impl Send for u16 {}
unsafe impl Send for u32 {}

From Rust's perspective, this is axiomatic. Using these unsafe impl blocks, we've declared by fiat that the Send property holds of these types.

Again, Rust has no way of knowing what Send means, and so it has no way of knowing whether these implementations are correct. It's just taking our word for it.

We can also implement Send for more complex types. For example, a mutex is thread-safe by design, and so it's okay to send mutex references between threads:

unsafe impl<'a, T: Send> Send for &'a Mutex<T> {}

Finally, we can permit arbitrary composite types to implement Send by writing a custom derive:

#[derive(Send)]
struct Foo<T, U>(T, U);

Of course, in reality, Send uses a special mechanism known as an auto trait to accomplish the same thing for ergonomics reasons. My point here is that there is no reason in principle that Send couldn't have been developed as a library maintained by somebody other than the Rust project. Again, the core abstractions provided by the language are enough on their own to permit somebody to build an abstraction as powerful as Send outside the Rust language project itself

Sticking with the hypothetical, our custom derive would automatically emit an unsafe impl block with the appropriate bounds. In particular, a type can be Send if all its fields are Send:

#[derive(Send)]
struct Foo<T, U>(T, U);

// Derive-generated code:
unsafe impl<T, U> Send for Foo<T, U>
where
    T: Send,
    U: Send,
{}

Since Send is just a normal trait, once we've written our implementations, it works like any other trait:

let foo: Foo<&Mutex<u8>, u16> = /* ... */;
spawn(move || {
    // use `foo`
});

We know that this Foo type is Send because:

We derived Send for Foo.
We manually implemented Send for Mutex references.
We manually implemented Send for primitive types.

How does this example fit into our framework?

Definition

We define Send, teaching Rust about a new trait. We use prose to document that Send carries a special safety property that Rust can't reason about: thread safety.

Enforcement

We define Send as an unsafe trait so users can't violate the safety property without writing the unsafe keyword.

We manually implement Send for types that we know satisfy the safety property.

And, in our hypothetical version, we write a custom derive that generates an implementation that guarantees thread safety by requiring every field to be Send.

Consumption

In spawn, we use a Send bound to justify that our internal call to pthread_create is sound.

And that's how Rust guarantees thread safety without any support from the language itself.

Example 2: Deadlock prevention

Our second example is about preventing deadlocks. This is work by Alex Konradi, who was on the Netstack3 team at the time – I'm basically just bragging for him.

Imagine you're writing a netstack and you want to make your stack multithreaded. You care about performance, so instead of putting one mutex around all your state, you do fine-grained locking, using small mutexes around different pieces of state:

struct Stack {
    ip: Mutex<IpState>,
    device: Mutex<DeviceState>,
}

As we saw in the previous example, Rust will guarantee that anything you do with these mutexes is thread-safe, but it won't prevent you from deadlocking.

For example, imagine two different threads acquire these mutexes in different orders:

// Thread A
stack.ip.lock();
stack.device.lock();

// Thread B
stack.device.lock();
stack.ip.lock();

This is a classic deadlock scenario. If we're unlucky, Thread A will successfully acquire the IP mutex and Thread B will successfully acquire the device mutex. Now, to make progress, they're each blocked on the other.

You can conceptualize this as a graph of the locks we might want to acquire. All threads start in an Unlocked state, and can take different paths through the graph in order to acquire different locks:

%%{init: {"theme":"base","securityLevel":"loose","flowchart":{"htmlLabels":true,"padding":1,"diagramPadding":0,"wrappingWidth":520},"themeCSS":".diagram-box{stroke:none!important}.box-unlocked{fill:#e0e0e0!important}.box-ip{fill:#f2d4d3!important}.box-device{fill:#d3d2f5!important}.diagram-black{fill:none!important;stroke:#000!important;stroke-linecap:butt!important;stroke-linejoin:miter!important}.black-left{stroke-width:1.4px!important}.black-right{stroke-width:1.4px!important}.diagram-red{fill:none!important;stroke:#db3b26!important;stroke-width:2.8px!important;stroke-linecap:butt!important;stroke-linejoin:miter!important}.diagram-arrow{fill:#db3b26!important;stroke:none!important}.diagram-label{fill:#000!important;font-family:Liberation Mono,Menlo,Monaco,Consolas,Courier New,monospace!important;font-weight:400!important}.diagram-node-label{font-size:38px!important;letter-spacing:0.375px!important;stroke:#000!important;stroke-width:0.8px!important;paint-order:stroke fill!important}.diagram-thread-label{font-size:26.75px!important;letter-spacing:0.125px!important;stroke:#000!important;stroke-width:0.5px!important;paint-order:stroke fill!important}"}}%%
flowchart TB
  canvas["<svg xmlns='http://www.w3.org/2000/svg' width='520' height='399' viewBox='0 0 586 450' style='display:block'><g transform='translate(-2,-1)'><rect class='diagram-box box-unlocked' x='197' y='23' width='198' height='128'/><rect class='diagram-box box-ip' x='21' y='240' width='198' height='198'/><rect class='diagram-box box-device' x='373' y='240' width='198' height='198'/><line class='diagram-red' x1='127' y1='307.6' x2='365' y2='307.6'/><polygon class='diagram-arrow' points='361.5,301.5 373,307.5 361.5,313.5'/><line class='diagram-red' x1='230' y1='375' x2='465' y2='375'/><polygon class='diagram-arrow' points='231,369 219,375 231,381'/><path class='diagram-black black-left' d='M197 86.75 H127.25 V307.75'/><path class='diagram-black black-right' d='M395 86.75 H465 V376'/><text class='diagram-label diagram-node-label' x='295.75' y='101' text-anchor='middle'>Unlocked</text><text class='diagram-label diagram-node-label' x='120.5' y='352' text-anchor='middle'>IP</text><text class='diagram-label diagram-node-label' x='471.75' y='353' text-anchor='middle'>Device</text><text class='diagram-label diagram-thread-label' x='295.5' y='298' text-anchor='middle'>Thread A</text><text class='diagram-label diagram-thread-label' x='296.25' y='401' text-anchor='middle'>Thread B</text></g></svg>"]
  style canvas fill:transparent,stroke:transparent,stroke-width:0px

To avoid the risk of deadlocking, we need to make sure that this graph is acyclic. The two acquisition orders above form a cycle, so we need to remove at least one of those red edges.

This might seem like a fairly trivial problem in this tiny example. It's pretty obvious from looking at the code that it might deadlock.

But Netstack3 has 77 different mutexes protecting all manner of different state spread across 192,000 lines of code. You can imagine that keeping track of the order in which you're supposed to acquire mutexes in that environment would get pretty difficult pretty fast.

And indeed, Netstack2, its predecessor which is written in Go, has historically been plagued by deadlocks.

So the Netstack3 team decided to try to prevent deadlocks statically, at compile time. To do this, they needed a way to encode a lock-order graph in the type system and a way to ensure that code could only acquire locks consistent with that graph.

Let's see how they did it.

Name every mutex

The first step is to name each mutex:

struct Mutex<Id, T> {
    mtx: std::sync::Mutex<T>,
    _marker: PhantomData<Id>,
}

This new Mutex type is identical to the standard-library mutex, except that it carries an extra type parameter used to identify the mutex.

Returning to our example, we define one type per mutex and use those types as names:

struct Stack {
    ip: Mutex<IpLock, IpState>,
    device: Mutex<DeviceLock, DeviceState>,
}

enum IpLock {}
enum DeviceLock {}

These types are never constructed at runtime. We only need them for type-system purposes.

Encode the lock-order graph

Next, we define the LockAfter and LockBefore traits:

/// # Safety
///
/// No cycles!
pub unsafe trait LockAfter<M> {}

pub unsafe trait LockBefore<M> {}

unsafe impl<B: LockAfter<A>, A> LockBefore<B> for A {}

These traits encode the lock-order graph. They're unsafe to implement because the user has to promise not to introduce a lock-order graph with cycles. If they do, they violate our safety property.

Next, we define a macro that implements these traits on behalf of a user:

macro_rules! impl_lock_after {
    ($A:ty => $B:ty) => {
        // SAFETY: The blanket impl will cause any cycles
        // to result in a blanket impl conflict, and thus
        // won't compile.
        unsafe impl LockAfter<$A> for $B {}

        unsafe impl<X: LockBefore<$A>> LockAfter<X> for $B {}
    };
}

impl_lock_after!(TransportLock => IpLock);
impl_lock_after!(IpLock => DeviceLock);

Note that the macro expansion adds a blanket implementation. Don't worry about following the full logic here, but the consequence of that blanket implementation is important: if a user invokes the macro in a way that introduces cycles into the lock graph, those blanket implementations will conflict with one another.

Cyclic graphs therefore fail to compile.

Track the current position in the graph

Next, we define the notion of a lock context:

struct LockCtx<Id>(PhantomData<Id>);

A LockCtx allows the type system to keep track of where in the lock graph you are at any given point in the source code. It is a zero-sized type, so it has no cost at runtime.

Using these building blocks, we can build a deadlock-proof locking API:

impl<Id, T> Mutex<Id, T> {
    pub fn lock<L>(
        &self,
        ctx: &mut LockCtx<L>,
    ) -> (MutexGuard<'_, T>, LockCtx<Id>)
    where
        L: LockBefore<Id>,
    {
        (
            self.mtx.lock().unwrap(),
            LockCtx(PhantomData),
        )
    }
}

Let's walk through the components of this signature one by one.

First, we require an existing LockCtx. Note that we borrow it mutably, which disables the LockCtx for further use until we've unlocked the mutex by dropping the MutexGuard. So long as we're holding the mutex, that context is effectively unusable.

Second, we require that the existing LockCtx has a lock ID upstream of our lock ID in the graph. This prevents us from locking a mutex which we've already locked.

L: LockBefore<Id>

Finally, we return a new LockCtx that the caller can use to continue locking mutexes (recall that the original LockCtx is now unusable so long as the current mutex is held). This new LockCtx has our lock ID. So long as the returned MutexGuard exists — in other words, so long as this lock is held — this new LockCtx is the only usable lock context, and it only permits locking mutexes downstream of this one in the lock graph.

One last bit of boilerplate: to represent the root of any lock graph, we introduce the Unlocked type and permit anyone to construct a lock context that starts off unlocked:

pub enum Unlocked {}

impl LockCtx<Unlocked> {
    pub const UNLOCKED: LockCtx<Unlocked> = LockCtx(PhantomData);
}

Use the graph

Let's go back to our example and see how we can use this machinery to prevent deadlocks.

As we showed before, we introduce one type for each mutex and use the macro to encode our graph in the type system as implementations of LockAfter and LockBefore:

struct Stack {
    ip: Mutex<IpLock, IpState>,
    device: Mutex<DeviceLock, DeviceState>,
}

enum IpLock {}
enum DeviceLock {}

impl_lock_after!(Unlocked => IpLock);
impl_lock_after!(IpLock => DeviceLock);

Remember that this code will only compile so long as the graph is free of cycles.

Now let's walk through how we use this graph at runtime. First, we construct a new context that starts in the unlocked state:

let mut ctx = LockCtx::UNLOCKED;

Next, we lock our first mutex — the IP mutex. This generates a MutexGuard and a new LockCtx that shadows the old one:

let (ip, mut ctx) = stack.ip.lock(&mut ctx);

Finally, we repeat the same process to lock the device mutex:

let (device, mut ctx) = stack.device.lock(&mut ctx);

Putting it together:

let mut ctx = LockCtx::UNLOCKED;
let (ip, mut ctx) = stack.ip.lock(&mut ctx);
let (device, mut ctx) = stack.device.lock(&mut ctx);

Now take a look at what happens if we try to violate the lock order. As before, we start in an unlocked state and directly lock the device mutex:

let mut ctx = LockCtx::UNLOCKED;
let (device, mut ctx) = stack.device.lock(&mut ctx);

On its own, that is fine. We haven't introduced a cycle yet, so it compiles.

But if, having acquired the device mutex, we now try to acquire the IP mutex, that won't compile, because there's no path in the graph from the device mutex to the IP mutex:

let (ip, mut ctx) = stack.ip.lock(&mut ctx);
//                                ^^^^^^^^ ERROR

How does this fit into our framework?

Definition

First, we define the LockAfter and LockBefore traits. We document in prose that they carry a safety property: they always encode an acyclic graph.

Second, we define the Mutex type. We document in prose that it carries a safety property: it is only locked consistent with a predefined lock-order graph.

Enforcement

We make the LockAfter and LockBefore traits unsafe so that a user can only violate our safety property by writing the unsafe keyword.

We design impl_lock_after! so that it guarantees that only acyclic graphs can compile.

And we use a LockBefore bound to ensure that the lock method can only be called consistently with the lock-order graph.

Consumption

The lock method uses its LockBefore bound to justify that locking the inner mutex won't deadlock.

What I have presented here is a simplified version of what exists in practice. There are actually two subtle ways that I'm aware of that you can violate deadlock safety using this simplified version. And there's an interesting conversation to be had about whether safety properties like these should be literally impossible to circumvent, or whether they only need to be hard enough to circumvent that you could't shoot yourself in the foot by accident.

In reality, the lock-ordering library takes the latter approach. I'm not going to get into that discussion here, and it doesn't really affect the point I'm trying to make.

The results

How has this played out for Netstack3?

In the beginning, Netstack3 was entirely single-threaded for simplicity. But over the past few years, the team has been working to thread mutexes and mutex guards through the entire stack. Finally, just a few weeks ago, they had finished that process and were ready to flip the switch — to go from running on one thread to running on multiple threads.

So they changed the default number of threads from one to four:

- Self(NonZeroU8::new(1).unwrap())
+ Self(NonZeroU8::new(4).unwrap())

That's it. That was the whole change. It was bug-free on the first try.

Conclusions

Those are my two examples.

I'd like to encourage you to try this methodology for yourself, and especially to bring it to new domains and new classes of bugs that nobody in the Rust community has tackled yet.

Let me give you a few pieces of concrete advice to help get you started.

Treat partial functions as a code smell

On a practical note, think of partial functions in your API as a code smell.

Your public functions should never panic, and they usually shouldn't return Option or Result unless that is inherent to the behavior you're modeling.

Returning an error because an I/O operation failed is unavoidable and totally fine. Returning an error because the caller passed an illegal value? That code shouldn't have compiled in the first place.

For great primers on this way of thinking, I recommend Parse, don't validate and Type Safety Back and Forth.

Make your APIs exactly match the problem

Make your APIs exactly match the structure of the problem you're modeling.

Here is the error type that Netstack3 returns when it fails to parse an IP packet:

pub enum IpParseError<I: IcmpIpExt> {
    Parse {
        error: ParseError,
    },
    ParameterProblem {
        src_ip: I::Addr,
        dst_ip: I::Addr,
        code: I::ParameterProblemCode,
        pointer: I::ParameterProblemPointer,
        must_send_icmp: bool,
        header_len: I::HeaderLen,
        action: IpParseErrorAction,
    },
}

pub enum IpParseErrorAction {
    DiscardPacket,
    DiscardPacketSendIcmp,
    DiscardPacketSendIcmpNoMulticast,
}

pub enum ParseError {
    NotSupported,
    NotExpected,
    Checksum,
    Format,
}

This may seem surprisingly convoluted, but it is a faithful representation of the complexity of both the IPv4 and IPv6 protocols. To do anything simpler would cause us problems down the line.

For example, let's zoom in on the parameter problem pointer field.

If you receive a malformed packet, you have to respond with your own packet containing an error message. That error message identifies the byte offset of the malformed field that caused parsing to fail. That byte offset is known as the "parameter problem pointer".

Note the generic type here. In IPv4, the field in the error message that stores this pointer is one byte long. In IPv6, it's four bytes long. Using generics ensures that we always store the byte offset using the right size of integer.

If instead we uniformly stored the pointer as, for example, a u32, we would have to perform a fallible conversion elsewhere in the codebase. If we had a bug, that conversion would either silently produce incorrect error packets or panic and crash the stack.

By ensuring that our error type faithfully models the behavior of IPv4 and IPv6, we can make that sort of bug impossible.

Expect diverse solutions

Don't expect the solution to always be the same.

Just because the catchphrase is always that "buggy programs don't compile" doesn't mean that there's a recipe. In my experience, different problems call for wildly different solutions that leverage a hodgepodge of language features.

Simplicity is the art of hiding complexity

Don't expect it to always fall out naturally.

One of my favorite quotes is from Rob Pike, talking about Go's design:

Simplicity is the art of hiding complexity.

Your responsibility is to do the hard work so that your users have a simple mental model of your API and they don't have to understand how awful the internals of your library are.

Rust wasn't explicitly designed to support this methodology, at least not in its full generality, and you may have to really bend the language to your will.

For example, if you haven't seen the trick, it is not at all obvious that it would be possible to prevent graph cycles using the trait system. The internals of the lock-ordering library are kind of ugly as a result, but the mental model that a user needs in order to reason about the API is very simple:

Cyclic graphs don't compile.

That's it.

Turtles all the way down

When it comes to your internal APIs, use the same level of discipline and rigor that you would use with a public API.

If a function has safety requirements, make it an unsafe function. If a function can panic, document it clearly.

It may seem obvious in the moment how to call an internal API correctly, but it won't be so obvious to the new developer on the team four years and six refactors from now. If you keep up this discipline, then no matter how much time passes, your internal APIs will be just as easy to use correctly as the day you wrote them.

We adhered to this discipline strictly in Netstack3, and the larger the project grew, the more it paid off.

A bit of speculation

We're late in the podcast now, so we can let our hair down a bit and speculate.

– Sean Carroll

Finally, if you'll permit me a bit of speculation, I think this methodology has the potential to fundamentally reshape how we engage in the process of software engineering, and I think that so far we've only scratched the surface of how far we can push it.

One reason is that there are domains where correctness is a much bigger deal than it is for most of us in this room. Rust is only just starting to make inroads into high-assurance domains like automotive, aerospace, medical devices, and so on. Those are domains where bugs are so unacceptable that the pace of development is absolutely glacial.

I'm not suggesting that we're going to be able to stop doing code reviews or anything crazy like that. But imagine how much faster we could move if we eliminated entire classes of bugs that we normally have to catch through reviews or testing.

Another reason I think this methodology could be a big deal is that it allows software to scale more effectively.

Software complexity often scales superlinearly as a function of the size of the codebase, and much of that complexity is incidental rather than inherent. Humans only have so much mental capacity. As incidental complexity increases, it crowds out our ability to reason about the inherent complexity that we're actually interested in. That puts a natural upper limit on the inherent complexity of the problems we can tackle.

My guess is that this methodology stands to make the biggest difference in large, complex codebases, where small savings in complexity across every module compound on one another.

That has certainly been our experience in Netstack3.

As I mentioned at the beginning, I wasn't able to provide nearly as many examples as I would have liked. I strongly encourage you to check out the references and do more reading on your own.

I really believe that, thanks to Rust, we have the opportunity to fundamentally improve the state of software engineering, and I intend to spend the next phase of my career proving it.

I hope you'll join me. Thank you.

I May Have Been Targeted by a Spear Phishing Attack

2026-06-26T00:00:00+00:00

Matthew Mastracci recently published a writeup of an apparent targeted supply-chain attack against Rust maintainers. This was posted to r/rust.

I was one of the maintainers who received similar outreach. In my case, I assume the target was my maintainer access to zerocopy. According to Matthew, the payload he received was a remote access trojan (RAT):

The drop is a full remote-access trojan that seems to have been put together by someone who knows what they are doing. It sets up an RSA key locally and uses AES-256-CBC as a session key.

On startup it calls a checkin routine that harvests and exfiltrates a host fingerprint:

primary IP address (enumerates all non-internal interfaces), plus all IPs

username (os.userInfo().username)

hostname

OS type + release + platform + architecture

process PID and full process.argv

Node version

It generates an RSA-2048 keypair and a random AES-256 session key (aes_psk), then all subsequent traffic is AES-256-CBC encrypted with an HMAC-SHA256 integrity tag.

It supports the following commands:

env — JSON.stringify(process.env) dumped and sent back.

upload — reads an arbitrary file path and exfiltrates it.

download — writes attacker-supplied bytes to any writable path.

spawn — runs an arbitrary process with optional shell expansion.

ls / cd / pwd / cp / mv — general filesystem primitives.

dns — makes the host resolve arbitrary names through a specified resolver (for DNS tunneling?).

dismantle — self-removal.

Here are the emails I received:

The approach was professionally plausible: someone claiming to represent a VC firm reached out about (presumably paid) technical diligence/consulting. While in my case the conversation petered out before moving beyond email, other targets were eventually directed toward a software project which, as Matthew's post describes, contained malicious code.

I don't know who was behind this, and I don't have independent evidence for attribution. But it appears to have been a coordinated attempt to compromise the maintainers of widely-used Rust crates. A similar attack – presumably carried out by the same actors – is described by Piyush Shukla roughly four months before Matthew's post. Piyush was himself the target; he works in the cryptocurrency space, so this may go beyond just Rust.

This post is just to add my data point to the public record: maintainers of high-impact open-source Rust projects are being targeted through ordinary-looking professional outreach.

The Coming Programming Language Renaissance

2026-06-08T00:00:00+00:00

For decades, programming language researchers have invented ways to make programs more explicit, more checkable, more constrained, more secure, and more verifiable. Refinement types. Dependent types. Effect systems. Session types. Typestate. Linear types. Contracts. Capabilities. Model checking. Proof assistants. Program synthesis. Typed holes. Semantic editing.

While watered-down versions of some of these have occasionally made it out of the academy ("technology from the past come to save the future from itself"), these are the exceptions that prove the rule. For the most part, what we know how to do in theory makes our industrial practice look like it's stuck in the stone age.

Some powerful programming language research directions are hamstrung by fundamental constraints like computability, but my sense is that most of what stops great programming language ideas from escaping the academy are human factors constraints. Human cognition is weird and, in many ways, extremely limited. Empirical work shows, for example, that even apparently superficial language syntax choices can materially affect programmer accuracy and learning.

Coding agents, on the other hand, have few of the same limitations. Research already shows that they excel where humans struggle, benefiting from their ability to read verbose error messages, author precise specifications, and tolerate the iteration required to generate valid proofs of program correctness.

We've been here before. In the 1930s and 1940s, between the Great Depression suppressing demand for new technology, wartime secrecy requirements preventing discoveries from being made public,^[1] and military demand driving increased R&D spending, a huge backlog of research built up. Some economists have suggested that America's postwar economic miracle^[2] – which brought us innovations from the microwave to commercial aviation to mass-market penicillin – was in part the natural consequence of corporations simply working through this research backlog.

Programming language development in 2026 looks very similar. Decades of theoretical innovation have built up in the academy due to human limitations that are about to be lifted en masse at exactly the time when demand for the production of new programming artifacts is skyrocketing.

Concretely, I predict that agentic programming will lead to a renaissance in programming language design. Language designers, faced with a fundamentally new type of user, suddenly have decades of theoretical research to draw from in designing new, powerful language features uniquely suited to the strengths of coding agents. Whether innovation takes the form of new language features in existing languages, DSLs, entirely new languages, or even a reimagining of what counts as a programming language, change is coming.

During WWII, the USPTO placed more than 11,000 patent applications under secrecy orders, then rescinded them en masse after the war. ↩
That video is about Japan, but the reference was too good to resist. ↩

Memory Safety Is a Matter of Life and Death

2026-06-02T00:00:00+00:00

In 2018, Saudi journalist Jamal Khashoggi's fiancée's phone was hacked. Later that year, Khashoggi walked into the Saudi Arabian consulate in Istanbul to obtain documents related to his planned marriage, and left two hours later in a body bag.^[1]

It's easy to forget that the work we do affects real people. It's easy to feel, if only unconsciously, that the point of programming is to be mathematically beautiful or fun to use. It's easy to forget that buffer overflows aren't just bad in the abstract, but that sometimes, they get real people killed.

And it's about to get much, much worse.

In my day job, I work on a security team, so I have access to hard data and water cooler talk with colleagues at Google and elsewhere about agentic bug-finding models, which are behind what the security industry has coined the "vulnpocalypse". For the past few weeks, I've been repeatedly asked the same question: "Mythos. Is it really that bad?" In a word: Yes.

The point of this post isn't to adjudicate this claim, so in leiu of a serious argument, I'll just leave you with this graph courtesy of Firefox:

Many factors will determine the long-term equillibrium of vulnerability discovery,^[2] but that equillibrium will take many years to reach. Regardless of how that plays out in the long term, the medium term impact is clear: memory-unsafe^[3] open-source^[4] software is not ready for high-quality bug-finding agents to be made widely available. It is expected that broad availability will happen at some point this summer,^[5] and when it does, few open source programs written in memory-unsafe languages will be safe from catastrophic exploits. Many actors will use these exploits to steal identities, or steal money, or commit other run-of-the-mill cyber crimes. But some will use these exploits to kill people.^[6] I am not being hyperbolic when I say that when these agents are made available, more people will die.

While memory safe languages are not a panacea, they will prevent the majority of these vulnerabilities (a typical estimate is 70%), and they will prevent the highest-impact of these vulnerabilities.^[7] This makes switching to memory safe languages a moral imperative. Alternatives such as Carbon are being developed, and other languages with different performance characteristics like Go or Java exist. However, when it comes to memory safe languages which are already in production and which impose no overhead relative to C or C++,^[8] Rust is the only option. Therefore, it is a moral imperative: Rust must succeed.

I am honored to call many people in the Rust community some of my closest friends. I have heard their stories at conferences, and over drinks, and on Zulip threads and Jitsi meetings. We have come to Rust for many different reasons (although they usually involve some form of nerd sniping). While the community has had its struggles, as any open source community does, it has always been an absolute blast – in our better moments, we have been kind to each other and written amazing software. To paraphrase Scott McNealy, we have "kicked butt, had fun, and changed computing forever."

I don't see why we can't keep having fun with Rust for the rest of our lives. But at the same time, we must acknowledge a hard truth: of the billions of people our software touches, only a tiny fraction know what a pointer is. To these people, it's irrelevant whether Rust is beautiful or fun. But for some of these people, if Rust doesn't succeed, they will die.

My dad and sister are both doctors, and I've always admired how they show up to work every day and face the weight of real, life-and-death consequences. The connection between their work and the lives of real people is immediate. One contrast that I've seen between their industry and our own is how they measure what is important.

Imagine that your parent had a worrisome lump on an annual physical and went to the radiologist to get it checked out. Imagine that the radiologist could choose a highly-accurate imaging technology, but instead they chose one with a high false-negative rate. You would be furious that this radiologist was jeopardizing your parent's life, and it wouldn't make you feel any better if the radiologist tried to reassure you that the technology they chose used a cleverer design, or had a better user interface, or made them feel nostalgic for when they were in med school. The radiologist's job is to treat your parent, and everything else is secondary.

Whatever the reason we each got into Rust, and whatever keeps us going, we have a new responsibility that most of us never asked for: people are now depending on us for their survival. They depend on Rust being secure. But more importantly, since Rust is already the most secure systems language in existence, they depend on Rust succeeding.

I wrote this post as a celebration that we have built a language so secure that people's lives depend on projects choosing it. We should be immensely proud of what we've accomplished.

But I also wrote it as a call to arms. Now more than ever, it is important that Rust succeed. In the coming months and years, we will need to make hard choices. We cannot be all things to all people. We will need to prioritize some use cases over others. We must think of progress and consensus as virtues in and of themselves. We must learn to recognize when having a consensus is more important than having the right consensus, and in these cases, to pick progress over stagnation. Whether it's C++ interop, or auditability, or any other un-glamorous feature which will nonetheless increase Rust adoption, we will have to hold our noses, wade into the muck, and get the work done.

I want to keep having fun, and thinking deep thoughts, and writing great code. I don't want to give up what has made Rust a joy to participate in.

But I also don't want people to die. If that means having a little less fun and being a little more serious, then I believe in our community's ability to rise to the occasion.

The exact chain of events is complicated, with the intelligence services of multiple countries targeting multiple people close to Khashoggi for surveillance, both before and after his murder. Publicly-available information is insufficient to prove that Khashoggi would not have been murdered had certain hacking attempts been unsuccessful. However, many other examples, such as the ransomware attack against Synnovis in 2024 and the WannaCry malware in 2017, demonstrate that hacking often leads directly or indirectly to the deaths of innocent people.

I chose to open with the Khashoggi example as it makes the moral obligation that I talk about in the rest of this article very clear. However, if you want to read this article with a less emotional and more analytical mindset, replace "this one person was brutally murdered thanks to a memory safety bug" with "in expectation, many people are injured, killed, or otherwise harmed thanks to memory safety bugs, although it's usually difficult to attribute a single incident to a particular bug." Beyond physical harm from nation state actors, estimates of the economic damage of cyber crime range from $100B to $1T annually. ↩
If you had asked someone in 1995, "what will be the long-term social consequences of the proliferation of cellular phones," nobody would have answered, "we will see a rapid rise in the suicide rate among teen girls." Anyone who tells you they know how this will play out is wrong (or lying).

To be less glib, some people have argued that the long-term effect of high-quality bug-finding agents will be to make software more secure, as teams will use them to catch potential vulnerabilities before they are merged. This is a real possibility, but so is the opposite, and in any case it may take a long time reach any equillibrium. The point of this article isn't to argue that we can predict any particular outcome, but rather that we have to consider that catastrophic outcomes are sufficiently probable to be worth worrying about and preparing for. ↩
While memory safe languages are not free from vulnerabilities, they are generally free from memory safety vulnerabilities. Non-memory safety vulnerabilities tend to permit exploits which are significantly less impactful than the exploits permitted by their memory safety-related counterparts. ↩
Some have suggested that agents are good enough at binary analysis that even closed-source software won't be safe – as long as an agent can inspect the binary, it can find vulnerabilities. ↩
Specifically, it is expected that Mythos itself or equivalently powerful models will become available for public use. In fact, ChatGPT 5.5, which is available to the public, may already match Mythos in capability. While such models may have guardrails, these usually only pose a serious impediment to casual users, while serious actors are usually able to find ways to bypass them. ↩
More accurately, they will use these exploits to steal information which will allow them to kill people. ↩
Per Android:

Memory safety vulnerabilities disproportionately represent our most severe vulnerabilities. In 2022, despite only representing 36% of vulnerabilities in the security bulletin, memory-safety vulnerabilities accounted for 86% of our critical severity security vulnerabilities, our highest rating, and 89% of our remotely exploitable vulnerabilities. Over the past few years, memory safety vulnerabilities have accounted for 78% of confirmed exploited “in-the-wild” vulnerabilities on Android devices. ↩
...which is a non-negotiable for many teams when switching languages. ↩

How to Run When You're Not Being Chased

2026-05-18T00:00:00+00:00

Have you heard this one before?

I obviously don't like being anxious, but it's how I drive myself. If I (meditated / went to therapy / got on anti-anxiety meds / etc), I would lose my edge.

I've had multiple close friends use some form of this argument to justify why they avoid working on their mental health: that their demons (anxiety or anything else) fuel their success, and that they couldn't succeed without them. Even if we grant that "keeping your edge" is a worthwhile goal, in my experience this argument doesn't hold up factually. The reality is instead:

In childhood, your demons did shape your ability to be successful, but that's done. As an adult:

They're not helping anymore.

They're actively holding you back.

I once heard someone articulate the argument succintly:

Imagine that you had schizophrena, and that you frequently hallucinated being chased by monsters. As a result, you spent a lot of time running away from these monsters. What would happen after you got treatment, stopped hallucinating the monsters, and stopped needing to run? You'd still be in great cardio shape.

The problem with the "my demons give me my edge" perspective is that it fails to distinguish between developing skills and using those skills. If, as a child, your demons made you feel like you had no choice but to succeed, then you spent your childhood pushing yourself, becoming comfortable with hard work and stress, and generally cultivating habits, traits, and skills that allow you to succeed as an adult. But now that you are an adult, the development is done. Your success is predicated on your having certain skills, not on the factors that led you to develop those skills in the first place.

What's more, your demons are likely holding you back by driving you to waste your energy on unimportant things.

The more severe the mental illness is, the more people believe [the false projections of] their mind.

– Dr. Alok Kanojia

Mental illness impairs our ability to rationally assess what is required in a given situation. To expand on the hallucination example, while it's useful to be in great running shape, it's actively harmful to feel driven to run all the time. If you're always running, even when it's not necessary, then you'll always be exhausted. On the occasions when it really is necessary, your energy will already be spent. What you really want is to have the judgment to know when running is justified, and otherwise to conserve your energy. To paraphrase the Serenity Prayer:

Give me the calm to rest when I can, the fitness to run when I must, and the mental health to be able to tell the difference.

So if you find yourself avoiding working on your mental health because you're worried you'll sabatoge your own success, consider that not working on your mental health may be the real act of sabotage.

How to Brush Your Teeth at Work

2026-04-14T00:00:00+00:00

When you were a kid, you had to learn to brush your teeth. It may be easy now, but at one point it was a skill. You would get toothpaste everywhere, and your mom or dad would have to remind you to brush every tooth.

You also probably hated doing it. You'd conveniently "forget" unless reminded, and then make excuses for why you hadn't done it yet, or didn't actually need to do it tonight in particular, or that actually your teeth were better than everybody else's teeth and thus brushing was unnecessary and beneath you.

The problem, of course, is that if you don't brush your teeth, your teeth will fall out of your head. At first, your parents were responsible for keeping your teeth in your head. Eventually, as you got better at brushing your teeth and got used to it as an inevitable, minor daily annoyance, you stopped thinking about it.

Now, as an adult, imagine if someone told you that brushing their teeth was too much of a hassle to be worth doing – that it would take too much effort to get good at and that, besides, it's ridiculous that we need to brush our teeth. Shouldn't they just be healthy all on their own? Why haven't we evolved that? You would rightfully think that they were shooting themselves in the foot over a very minor inconveience. Surely a few minutes of mindless chore per day is worth the upside, right?

This is the pitch that someone gave to me a few years ago, and it changed my relationship with my career.

At the time, I was feeling what is, I think, a very common feeling: That pointless bureaucracy was making my job a slog, that I was being held back from doing good, interesting work, being asked to make it worse in order to make it legible to a bureaucracy that didn't understand what I was doing, didn't understand why the way I was doing it was actually better, didn't value quality, didn't reward the right behaviors, and generally made us all do paperwork that few people would ever read and that would have no meaningful effect on what we actually produced. It all felt so pointless and unfair. Why can't we just let people do good work and get on with their lives?

As an aside, there are actually good reasons for all of this that tend to be harder to appreciate when you're earlier in your career. A mentor once told me: "Organizational dysfunction is never completely avoidable. Sure, it might be possible to do things better, but it's always downstream of a genuinely hard coordination problem. The fact that all these different people and groups, with their competing and misaligned interests, can collaborate at all is a miracle of game theory. The fact that it isn't perfect should be the least surprising thing in the world." If this is interesting to you, read The Evolution of Cooperation by Robert Axelrod. It will change how you think about human organizations.

But let's ignore that and assume, for the sake of argument, that bureaucracy really does exist for no good reason at all.

While this feeling is nearly universal at some point in a career, consider the analogy to brushing your teeth. No matter how stilly it may feel, feeling that it's silly isn't going to magically make us evolve past the need to brush our teeth. You have two options: You can either do what most of us do – accept a very minor inconvenience and get the benefits of having teeth – or you can avoid a very minor inconvenience, but at the cost of feeling constantly angry at an unfair world and not having teeth.

Think of the bureaucratic bullshit as teeth-brushing. The child perspective entails constantly dreading the impending request to brush, clumsily brushing and getting toothpaste all over your face, and having poor dental health from all of the times you managed to avoid doing it. Imagine, instead, if you got good at navigating the bureaucracy. It would take some up-front investment to develop the skill, but once you did, you could put it all on auto-pilot. Just like brushing your teeth, as you got good at it, it would take less mental effort. It would feel less like a big, looming obligation and more like just another small cost of doing business that you get out of the way so you can move on to the interesting work you really care about doing.

Beyond the mental peace, you would also have more control over your career. As you learned how to speak bureaucrat-ese, you would be more effective at demonstrating – in a language that your manager's manager's manager can understand – that your work is important and should continue to be funded. You would get to pick more interesting projects because you would know how to convince your leadership that they were worth investing in. You would get less oversight and micromanagement because you would be trusted to use your judgment about what to work on.

The next time you encounter somebody with a really cool job and think, "fuck, what did they do to get a job that cool," consider how good they are at brushing their teeth. Maybe a lot of what separates you from them is just a few minutes per day and some toothpaste.

Force Your Macro's Callers to Write Unsafe

2025-04-05T00:00:00+00:00

Imagine you're writing this macro:

/// Types that can be frobnicated.
///
/// # Safety
///
/// It must be sound to frobnicate `Self`.
unsafe trait Frobnicatable {}

/// Implements `Frobnicatable` for `$t`.
///
/// # Safety
///
/// `$t` must satisfy the safety invariant of `Frobnicatable`.
macro_rules! unsafe_impl_frobnicatable {
    ($t:ty) => {
        // SAFETY: The caller has promised that `$t` satisfies
        // the safety invariant of `Frobnicatable`.
        unsafe impl Frobnicatable for $t {}  
    };
}

Of course, you'd never actually write a macro to emit code this simple, but it's a meant as a stand-in for macros that emit more complex code.

A caller might write this:

struct Foo;

// SAFETY: It is sound to frobnicate `Foo`.
unsafe_impl_frobnicatable!(Foo);

But imagine that your caller wants to practice good safety hygeine, and they use #![deny(clippy::undocumented_unsafe_blocks)]. Because Clippy doesn't realize that the macro call requires a safety comment, it won't complain if your caller forgets their safety comment:

#![deny(clippy::undocumented_unsafe_blocks)]

struct Foo;

// Clippy: 🙈
unsafe_impl_frobnicatable!(Foo);

Instead, you should write your macro so that it invokes unsafe code. This also means that we can drop the unsafe_ prefix from the macro's name – it was just there to make it less likely for your callers to miss the safety requirement, but that's no longer a concern. Also, make sure to have the macro's right-hand side contain a block – ie, {{ ... }} – see the appendix for an explanation of why this is important.

/// Implements `Frobnicatable` for `$t`.
///
/// # Safety
///
/// `$t` must satisfy the safety invariant of `Frobnicatable`.
macro_rules! impl_frobnicatable {
    ($t:ty) => {{
        // Force caller to wrap in an `unsafe { ... }` block.
        $crate::__unsafe();
        
        // SAFETY: The caller has promised that `$t` satisfies
        // the safety invariant of `Frobnicatable`.
        unsafe impl Frobnicatable for $t {}  
    }};
}

#[doc(hidden)]
pub const unsafe fn __unsafe() {}

Now, even when Clippy isn't used, Rust will force your caller to put the macro invocation in an unsafe block. In other words, this will no longer compile:

// SAFETY: It is sound to frobnicate `Foo`.
impl_frobnicatable!(Foo);

To fix this, your caller must wrap the invocation in an unsafe block, for example:

// SAFETY: It is sound to frobnicate `Foo`.
const _: () = unsafe { impl_frobnicatable!(Foo) };

And now Clippy will catch a missing safety comment since your caller is using a normal unsafe block.

Appendix: Macro expansion

Note one subtlety: You're going to want to have your macro emit a block (ie, use (...) => {{ ... }} rather than (...) => { ... }). You're now requiring your callers to call your macro in expression or statement position (ie, inside of a block) rather than in item position (ie, at the top-level, outside of a block). Thus, it has to play nicely with code before and after it in the same block. Expanding to a self-contained block is a good way of ensuring that this will happen.

Introducing Mundane, a New Cryptography Library for Rust

2018-11-06T00:00:00+00:00

Mundane, meet the world. The world, meet Mundane.

Mundane is a cryptography library written in Rust and backed by BoringSSL. It aims to be difficult to misuse, ergonomic, and performant (in that order). It was originally created to serve the cryptography needs of Fuchsia, but we've decided to split it off as a general-purpose crate.

Why Mundane?

Mundane's raison d'être is to provide an API which is difficult to misuse. Experience shows that one of the most common failure modes of cryptography is incorrect implementation, and often, that failure occurs at the boundary between an application and a cryptography library.

Given this experience, Mundane takes the approach of giving the programmer the fewest degrees of freedom possible. Doing the right thing should be easy and feel natural. Doing the wrong thing should feel difficult and ideally be entirely impossible.

Rust is the perfect language for a library with this design philosophy. Its expressive and strict type system allows us to ensure that, across a wide array of correctness properties, an incorrect program will not compile.

Consider, for example, the humble signature. A signature consists of two components: a hash of some input data, and a digital signature of that hash. Mundane's Signature trait looks like this:

/// A cryptographic signature generated by a private key.
pub trait Signature: Sealed + Sized {
    /// The private key type used to generate this signature.
    type PrivateKey: PrivateKey;

    /// Sign a message.
    ///
    /// The input to this function is always a message, never a
    /// digest. If a signature scheme calls for hashing a message
    /// and signing the hash digest, `sign` is responsible for
    /// both hashing and signing.
    fn sign(key: &Self::PrivateKey, message: &[u8])
        -> Result<Self, Error>;

    /// Verify a signature.
    ///
    /// The input to this function is always a message, never a
    /// digest. If a signature scheme calls for hashing a message
    /// and signing the hash digest, `verify` is responsible for
    /// both hashing and verifying the digest.
    fn verify(&self, key: &<Self::PrivateKey as PrivateKey>::Public,
        message: &[u8]) -> bool;
}

Right away, we see an example of a mistake which is impossible with Mundane's API: It's impossible to incorrectly combine hashing and signing because they're not exposed as separate operations. Instead, both on the signing path and the verification path, hashing is handled by the API automatically.

This allows us to unlock more powerful and subtle correctness properties. For example, since we are responsible for hashing, not only can we ensure that hashing is done properly, we can also ensure that a secure hash function is used. Consider, for example, the EcdsaSignature, which implements Signature:

/// A DER-encoded ECDSA signature.
pub struct EcdsaSignature<C: PCurve, H: Hasher + EcdsaHash<C>> { ... }

The ECDSA signature algorithm can operate over any elliptic curve (the C type parameter) and any hash function (the H type parameter). If we look closely, we see that H is not just a Hasher (the trait implemented by hash functions), but is, in addition, an EcdsaHash<C>. This trait is only implemented for valid hash/curve pairs. Some hashes are large enough that, in order to sign them using ECDSA, they would need to be truncated. For example, using a SHA-512 hash with ECDSA over a P-256 curve would require truncating the hash before signing, reducing the security of the hash.

Thus, we only implement EcdsaHash for hash/curve pairs where no truncation is necessary. Hash/curve pairs which require truncation aren't supported, and code which tries to use them won't compile.

For more examples, see the documentation. For an in-depth discussion of our design philosophy, see the design doc.

Implementation

Of course, there's more to cryptography than the API. Mundane is based on the well-vetted and actively-developed BoringSSL. Mundane does not provide any implementations itself, and does not currently have any plans to do so in the future.

A common problem with relying on native libraries from Rust crates is compatibility - while Rust and Cargo are able to compile multiple crate versions into a single binary, C is not so flexible. In order to avoid this problem, Mundane vendors a copy of the BoringSSL source, which is shipped along with the crate on crates.io. It uses symbol prefixing to ensure that the version of BoringSSL compiled for a particular version of Mundane will not conflict with either a) the version of BoringSSL compiled for a different version of Mundane or, b) any version of BoringSSL compiled for any other Rust crate. Because the source is shipped with Mundane, users do not need to have BoringSSL installed when building.

The approach we use to prefix BoringSSL's symbols is complex, and makes compilations take longer than we'd like. If you're interested in helping improve the situation, please get in touch!

Why not <other cryptography crate>?

There are a host of fantastic cryptography crates in Rust, each with its own focus and design tradeoffs. Mundane has its focus - to be misuse-resistant at all costs - and it makes design tradeoffs with that focus in mind. If that's not what you're looking for, then one of the other cryptography crates might be a better fit for your use case.

Additionally, as the primary Rust cryptography crate used by the Fuchsia OS, Mundane has a very high bar for security and active maintenance. While Rust is our ideal language for almost everything, it still has some shortcomings when it comes to writing cryptography code, such as the lack of guaranteed constant-time operations. Moreover, to be blunt, we don't trust ourselves to implement cryptography correctly. That's why we rely exclusively on BoringSSL, which is developed full-time by a team of professional cryptographers, and which provides strong guarantees about fixing vulnerabilities quickly if they arise.

How can I help?

Use Mundane! Use it for high-level applications and low level applications. Use it in all different environments. The more users we have, the more feedback we'll get, and the better Mundane will be. Tell us what you like, and what you don't. Tell us what features you want to see in the future. Submit issues.

Rust Has Higher Kinded Types Already... Sort Of

2018-10-18T00:00:00+00:00

Refresher

First, a very quick refresher.

In Rust, a type which takes type parameters (Rc<T>, Vec<T>, HashMap<K, V>, etc) is only a valid type when all type parameters are specified. In other words, Rc, Vec, and HashMap<K> are not types. You can't have a variable of type Rc. You can't pass Rc as a parameter to other types.

The ability to have such things be actual types is a feature called higher kinded types (HKT). It's primarily useful when combined with traits which also take type parameters. Consider, for example, a hypothetical trait to abstract over different kinds of reference-counted pointers.

trait RcPtr<T>: Clone + Deref<Target=T> {}

You can easily implement this for Rc and Arc:

impl<T> RcPtr<T> for Rc<T> {}
impl<T> RcPtr<T> for Arc<T> {}

However, with HKTs, RcPtr (as opposed to RcPtr<T>) would be a valid trait, and Rc and Arc would implement it (as opposed to Rc<T> and Arc<T>).

This would allow you to do things like:

struct TreeNode<T, P: RcPtr> {
    val: T,
    right: P<TreeNode<T, P>>,
    left: P<TreeNode<T, P>>,
}

This is a tree data structure where the caller gets to decide whether it's thread-safe or not. There are many applications which would benefit greatly from allowing the caller to specify some property such as thread safety, caching behavior, etc - all things which require HKTs in order to leave type parameters unbound until later than is allowed in normal Rust today (I say "normal" because, well, read the next section).

HKTs in Rust Today

One way to view the limitation that prevents us from writing this code is that, a) when declaring a trait bound, all type parameters must be bound (i.e., P: RcPtr is not a valid trait bound since RcPtr takes a type parameter) and, b) when declaring a type, all type parameters must be bound (i.e., Rc and Arc aren't types; Rc<T> and Arc<T> are).

However, it's not quite true that there's no way to define a trait which allows some type parameters to remain unbound, or that there's no way to implement those traits with types which also leave some type parameters unbound. Consider, for example, the following function from the standard library's Iterator trait:

trait Iterator {
    fn sum<S: Sum<Self::Item>>(self) -> S;
}

Even once a type has implemented the Iterator trait, the S type parameter on sum is still unbound - it's only bound once the method is called. This implies a general pattern - if you want to allow specifying a trait bound while leaving type parameters unbound, just move them into the methods on that trait. Let's give it a try:

trait RcPtr {
    fn clone<T>(&self) -> Self;
    fn deref<T>(&self) -> &T;
}

If this trait gives you an uneasy feeling, you've got the right intuition. Normally, with a trait with a type parameter like:

trait RcPtr<T>  {
    fn clone(&self) -> Self;
    fn deref(&self) -> &T;
}

You declare variables like:

let foo: RcPtr<T> = ...;

Then, whenever you operate on foo, the compiler guarantees that the correct type T is used. This is important, or else you'd be able to operate on the same variable as different types at different points in the program:

let foo: usize = 0;
(foo: &str) = "hello there";
(foo: &[u8])[0] += 1;

In other words, the fact that Rust tracks the type of variables is what allows us to have memory safety! So if the type parameter to RcPtr is moved from the trait itself into each of the methods, then you could do something like:

let foo: RcPtr = ...;
println!("{}", foo.deref::<&str>());
let bar = foo.clone::<&[u8]>();

In other words, you could violate memory safety. No bueno. I guess we're out of luck...

Although... There is a giant, hairy, awful, scary escape hatch in the language, so what if we just...

unsafe trait RcPtr {
    unsafe fn clone<T>(&self) -> Self;
    unsafe fn deref<T>(&self) -> &T;
}

That's better :)

Of course, using unsafe means that you're opting into manually verifying certain properties. So what properties does the programmer pinky-swear to uphold here? Essentially, it becomes the programmer's responsibility to track the types of variables and only ever access them consistently. In other words, for a given instance of a variable, it must only ever be accessed with the same T parameter.

Luckily, Rust is pretty good at encapsulation, so we can just encapsulate this in a safe type that uses unsafe under the hood, and guarantees to always use the same T:

struct RcWrapper<R: RcPtr, T> {
    rc: R,
    _marker: PhantomData<T>,
}

impl<R: RcPtr, T> RcWrapper<R, T> {
    fn new(t: T) -> Self {
        RcWrapper {
            rc: R::new::<T>(t),
            _marker: PhantomData,
        }
    }
}

impl<R: RcPtr, T> Deref for RcWrapper<R, T> {
    type Target = T;
    fn deref(&self) -> &T {
        unsafe { self.rc.deref() }
    }
}

Given an R: RcPtr and a T, instead of writing R<T> as we would if Rust supported HKT natively, we write RcWrapper<R, T>. We can then treat the RcWrapper<R, T> just as we would R<T> if we had native HKT - we can pass it around, call methods on it, clone it, etc.

OK, so we can define the trait, and we can even make it safe to use. But how on earth could we implement the trait? What would such an implementation look like?

Awful, that's what. While one of the benefits of giving variables types is that you can guarantee that you always operate on them as the proper type, another benefit is that you always know what size they are. What size is something whose type isn't known until you call methods on it? God only knows...

And when only God knows something, you've got no choice but to figure it out at runtime, and stick it on the heap.

struct SingleThreadedRc {
    // Actually *mut Rc<T>, but we don't know
    // what T is until runtime...
    inner: *mut (),
}

impl SingleThreadedRc {
    // Dear SingleThreadedRc, I pinky-swear that
    // you're actually holding a pointer to an Rc<T>.
    // I swear on my mother's grave.
    unsafe fn inner<T>(&self) -> &Rc<T> {
        &*(self.inner as *const T)
    }
}

unsafe impl RcPtr for SingleThreadedRc {
    unsafe fn clone<T>(&self) -> Self {
        // Get access to the Rc<T> we knew we had 
        // inside of us deep down all along...
        let inner: &Rc<T> = self.inner();
        let new_inner: &Rc<T> = inner.clone();
        let new_inner = Box::new(new_inner);
        SingleThreadedRc {
            inner: Box::into_raw(new_inner) as *mut (),
        }
    }

    unsafe fn deref<T>(&self) -> &T {
        let inner: &Rc<T> = self.inner();
        inner.deref()
    }
}

Does anybody have any Purell?

But it works. You can see a full example including usage here.

Conclusion

To recap, here's the general pattern. You would like a trait, MyTrait<T>, to be higher-kinded, but we don't support that in Rust right now. Instead, you create a trait, MyTrait, which takes no type parameters, but each of its methods take a type parameter, T. Then, if you have an M: MyTrait, instead of doing M<T> as you normally would, you create a concrete wrapper type, MyTraitWrapper<M: MyTrait, T>. M<T> is then written as MyTraitWrapper<M, T>.

Of course, the code presented here is far from pretty or optimal. One obvious drawback is the heap allocation. If you could guarantee an upper bound on the size of the internal type (e.g., if you knew an upper bound on the size of Rc<T> for all T), you could stack- instead of heap-allocate.

This also doesn't cover everything you might want HKT for. For example, how do you emulate the following trait, where you'd like to be able to do M: MyTrait and then, later, M::<T>::MyType?

trait MyTrait<T> {
    type MyType: Clone;
}

I'm pretty sure it's possible, but I haven't actually gotten it working in code yet. In the interest of keeping this post short (and also reclaiming some of my free time), I've decided to leave it an exercise to readers to either get it working, or argue that it can't be done (and thus that the approach I've proposed here falls short of full HKT).

Finally, in fairness, calling this "HKT" is a bit of a stretch. It's not a general-purpose mechanism - you have to manually implement it for every trait and every type - and it doesn't provide the same soundness guarantees that a language feature would. But, for any given application that needs a particular trait to be HKT-ified, it gets the job done.

Rust has HKT right now, but only the sinners can use it.

Soylent: In Review

2016-09-18T00:00:00+00:00

[This post is part of a series on my month-long experiment with Soylent. See the original post for an introduction.]

Just over two years ago, I started what I thought was going to be, as the note above suggests, a month-long experiment with Soylent (if you don't know what Soylent is, read the original post). At the time, I was considering Soylent as a potential full or partial replacement for normal food, but since I was about to head into a school year with a full meal plan, I wasn't going to be able to start right away. So I wanted to try it out for a month to see if it was something I'd want to do full-time after graduation, and I decided to write this blog series on it - one post per day - to track my progress and let others benefit from my experience.

Not only did I not end up switching to Soylent, I didn't even finish the blog series (or, for that matter, the experiment itself). And for the past two years, I've continually putting off writing a final post. Better late than never, I guess?

In this post, I'm going to explain my main takeaways from the experiment, and a number of tips, tricks, and insights that I learned along the way.

UPDATE (09/18/2016): I should note that the version of Soylent I used for this experiment was 1.0. Some results may vary with other versions.

You Need to Need It

By far the most important lesson that I learned is that Soylent would never work for me unless I need it to. As I described in the original post, my reason for considering Soylent was convenience - I'm lazy when it comes to shopping, cooking, and cleaning, and Soylent reduces the burden of all of these. But the timing of the experiment meant that I never needed the convenience. For the first two weeks of the experiment, I was on summer break, and I had all the time in the world. Cooking was, if anything, a way to fill the time. After that, I was at school, and on a full meal plan. Instead of shopping, cooking, and cleaning, I simply walked into a dining hall, ate already-prepared food, and left my plates to be cleaned by somebody else.

For me, Soylent would only work if the alternative was time-consuming or annoying. To be fair, though, I never got to try, so I can only speculate. Maybe even then it wouldn't be enough.

But that begs the question - why didn't I enjoy Soylent in its own right?

Soylent is Medicine

After you get used to it, Soylent doesn't taste actively bad, but it certainly isn't appealing in its own right either. I found myself craving real food not because I was hungry - Soylent certainly made me full - but for the taste and the texture. As I described in my third post, "I still feel full and satisfied after I drink it, but I have to will myself to drink it knowing that that will be the outcome; I don't find myself actively wanting Soylent itself." In short, you have to treat it like medicine - you eat it because you need to, not because you want to.

Again, you need to need it. If it's your only option, or if it has upsides that normal food doesn't, then it's worth it. I also know that a lot of people only use Soylent for one meal in the day - for example, a quick breakfast if they're rushing to get to work in the morning. It makes sense to me that that might work better since you can still enjoy lunch and dinner, but I haven't tried it myself, so I can't say for sure.

Tips, Tricks, and Insights

Even though I didn't see the experiment through to its conclusion, I did eat it for more than two weeks, and picked a few tips, tricks, and insights along the way.

The taste of Soylent is mildly sweet, and it was a bit off-putting to me at first. The texture is very thick, and I was not used to drinking liquid that dense. It takes a few days to become fully comfortable with drinking it, so be patient.
Drink slowly. I can't emphasize this enough. Each glass of Soylent has the caloric content of a full meal. Imagine how full you would feel if you scarfed down a cheeseburger in thirty seconds. Soylent may be a liquid, but if you drink it quickly, you will feel just as full. I made this mistake the first day, and I felt queasy afterwards.
Not only should you drink slowly when sitting down to a meal, but it may be a good idea to sip continuously throughout the day instead of treating each drink as a single meal. I found myself naturally falling into this rythm, and I think the reason is that, being a liquid, Soylent doesn't have to be broken down by stomach acids before its nutrients get absorbed into the blood stream. Thus, in order to get the continual feeling of having energy that you get from eating a meal of solid food, just drink small quantities of Soylent continuously.
If you use Soylent for a large part of your diet, and you adopt the approach of drinking small amounts continuously, you may find that you become accustomed to eating small quantities, and that eating a large quantity of food (such as at a restaurant) makes you feel much more full than it normally would.
A common problem that people report when first trying Soylent is flatulence. "Something something gut biome," I guess; I'm not a biologist. But it's something people experience. I never had it that badly, but one thing I can say is that drinking Soylent quickly makes it worse - again, always drink slowly.
Always drink cool or cold Soylent, and never water it down. To this day, my friends cite the most entertaining post in this series as the day 15 post in which I describe the horror of drinking warm, watered-down Soylent. "For the love of all that is good and holy, do not water down your Soylent... I don't know what it is about watered down or warm Soylent, but just... just don't. Dear god, just don't." I felt really queasy - on the verge of throwing up - for the better part of the morning. However, I had cold, not-watered-down Soylent for lunch, and it made my stomach feel much better, which confirmed that the temperature and dilution were the problem.

Well, that's all I've got! It feels like a relief to finally get this thing written - I've seriously been putting it off for two years. Hopefully this dinky blog series on a dinky personal website will find someone in the future with questions, and give them some answers. It will be my little piece of "Wisdom of the Ancients."

Wisdom of the Ancients, xkcd

Soylent: Day 17

2014-08-28T00:00:00+00:00

[This post is part of a series on my month-long experiment with Soylent. See the original post for an introduction.]

Nothing noteworthy to report today.

Vitals

Objective Vitals

Weight: N/A
Heart rate: N/A

Soylent: Day 16

2014-08-27T00:00:00+00:00

[This post is part of a series on my month-long experiment with Soylent. See the original post for an introduction.]

Nothing noteworthy to report today.

Vitals

Objective Vitals

Weight: N/A
Heart rate: N/A

Soylent: Day 15

2014-08-26T00:00:00+00:00

[This post is part of a series on my month-long experiment with Soylent. See the original post for an introduction.]

For the love of all that is good and holy, do not water down your Soylent. This morning, not having a fridge to store a full pitcher in, I decided to make one serving for breakfast. I thought that I could get away with fitting one and a half servings in my water bottle, so I put one and a half serving's worth of water in. Then I realized that it wasn't going to fit, so I only put in one serving of Soylent. Oh, and since I didn't have a fridge, it was room temperature (another big no-no). I forced myself to drink the warm, diluted substance because I needed to eat, but each sip made me feel like I was going to puke. I don't know what it is about watered down or warm Soylent, but just... just don't. Dear god, just don't.

UPDATE: I spoke too soon! After initially posting this article, I had some Soylent that had been sitting in the fridge all day. It was actually a huge relief. The taste of the watered down Soylent had been in my mouth all day, and the cold Soylent was like cool sweet relief. Not only was it refreshing, but it actually washed away the taste of the watered down Soylent from earlier. Soylent to the rescue, I guess?

Vitals

I still don't have access to a scale in my dorm, and I forgot to take my heart rate (that one's on me).

Objective Vitals

Weight: N/A
Heart rate: N/A

Soylent: Day 14

2014-08-25T00:00:00+00:00

[This post is part of a series on my month-long experiment with Soylent. See the original post for an introduction.]

Yesterday was my last day in Boston for the summer before heading to Providence. Being me, I decided to make the trip by bike, and, also being me, decided to bump said trip up to 2:00 am last night on a whim.

Besides preening my adventurer's feathers, I mention this because I had to finish all of the Soylent that I had before I left, or else throw it away (I took the pitcher with me, and I didn't want to have a half-full pitcher of Soylent sloshing around on the back of my bike). Naturally, I picked the first option.

There was a lot of Soylent left - about two very full glasses' worth. Having made the decision to bump up the trip about 30 minutes before I was going to leave, I had 30 minutes to drink it all. So I chugged.

I expected to feel over-full and maybe in a small amount of pain. But that's not exactly what happened. I felt pretty full, but not so much that it bothered me. Instead, I think largely from drinking the Soylent so quickly and giving my esophagus a workout, I got a heartburn-like pain near the bottom of my sternum that lasted all through the night and even persists faintly now at 9:00 pm the next day.

The moral of the story: Drink. Soylent. Slowly.

Vitals

I don't have access to a scale in my dorm, so I was unable to weigh myself. I may buy a cheap one so I can keep taking weight measurements. I also didn't want to take my heart rate after a night of biking and no sleep since that would kind of be like slapping statistical significance in the face.

Objective Vitals

Weight: N/A
Heart rate: N/A

Soylent: Day 13

2014-08-24T00:00:00+00:00

[This post is part of a series on my month-long experiment with Soylent. See the original post for an introduction.]

Nothing noteworthy to report today.

Vitals

Objective Vitals

Weight: 164 lbs
Heart rate: N/A

Soylent: Day 12

2014-08-23T00:00:00+00:00

[This post is part of a series on my month-long experiment with Soylent. See the original post for an introduction.]

Nothing noteworthy to report today.

Vitals

Objective Vitals

Weight: 165 lbs
Heart rate: 64 bpm

Soylent: Day 11

2014-08-22T00:00:00+00:00

[This post is part of a series on my month-long experiment with Soylent. See the original post for an introduction.]

Nothing new to report today.

Vitals

I wasn't at home last night, so I couldn't weigh myself in the morning (no scale available), and I forgot to take my heart rate (that one's on me).

Objective Vitals

Weight: N/A
Heart rate: N/A

Soylent: Day 10

2014-08-21T00:00:00+00:00

[This post is part of a series on my month-long experiment with Soylent. See the original post for an introduction.]

Today I went out to dinner and ate a fairly large amount, and I was less full than I had been in previous days. I think my straying from a pure-Soylent diet the past few days has gotten me more accustomed to large quantities of food in bursts as opposed to small quantities repeatedly.

Vitals

I've decided that, given the fact that my physical health hasn't fluctuated at all (it's just been listed as "good" on all but one of the days), I'm going to remove it from the list of vitals. If any noteworthy changes happen, I'll note them, but otherwise I won't mention it.

Also, I pulled an all-nighter last night, and so I couldn't take my heart rate just after waking up, and I omitted it today.

Objective Vitals

Weight: 164 lbs
Heart rate: N/A

Soylent: Day 9

2014-08-20T00:00:00+00:00

[This post is part of a series on my month-long experiment with Soylent. See the original post for an introduction.]

Today, I went out to dinner and desert, I didn't fill up as much as I expected. I have a suspicion that this may just be because of the increased amount of normal food I've had over the past few days (with going out to dinner on Sunday and snacks the past few days). I just had a glass of Soylent as I sit here writing this (at 3:00am, so it's been a while since I ate), and I was actually hungry before I had it. Really the take-home here is that this stuff makes for good blog posts but really shitty empirical science. Hooray for small sample sizes!

Vitals

Objective Vitals

Weight: 162 lbs
Heart rate: 72 bpm

Subjective Vitals

Physical health: Good

Soylent: Day 8

2014-08-19T00:00:00+00:00

[This post is part of a series on my month-long experiment with Soylent. See the original post for an introduction.]

Today, I had a turkey sandwich with two slices of sharp cheddar cheese on a relatively empty stomach, and now I'm very full. Normally this would not be the case - I'd be satisfied at best. I'm getting the sense that I'm really getting used to eating a bit at a time, and I'm getting used to it pretty quickly (after all, it's barely been a week).

Vitals

Objective Vitals

Weight: 168 lbs
Heart rate: 62 bpm

Subjective Vitals

Physical health: Good

Soylent: Day 7

2014-08-18T00:00:00+00:00

[This post is part of a series on my month-long experiment with Soylent. See the original post for an introduction.]

Nothing noteworthy to report today.

Vitals

I forgot to take my vitals this morning... sorry.

Objective Vitals

Weight: N/A
Heart rate: N/A

Subjective Vitals

Physical health: Good

Soylent: Day 6

2014-08-17T00:00:00+00:00

[This post is part of a series on my month-long experiment with Soylent. See the original post for an introduction.]

Today, I went out to dinner with a friend and had a restaurant-sized cheeseburger. I normally have a bottomless pit of a stomach, but this filled me up to the point of being uncomfortable. My hypothesis is that I've gotten used to not eating a large amount at a time. Because Soylent is a liquid, it is absorbed immediately, and thus must be consumed bit by bit during the day. Thus, over the course of the past few days, my stomach has never been particularly full. I guess I've gotten used to it.

I also didn't even finish the leftover Soylent from last night (there was about a fifth of a day's worth), let alone make a new pitcher. I'll be curious to see if I wake up very hungry tomorrow (or have more after posting this article and before going to bed).

Vitals

Objective Vitals

Weight: 162 lbs
Heart rate: 80 bpm

Subjective Vitals

Physical health: Good

Soylent: Day 5

2014-08-16T00:00:00+00:00

[This post is part of a series on my month-long experiment with Soylent. See the original post for an introduction.]

Nothing particularly noteworthy from today. However, it's been four full days now, and I haven't had nearly the amount of crime-against-humanity farting that many others have reported. I've had a little, but it's been pretty infrequent (a few farts a day) and not particularly smelly. I expect that it's been some combination of following the advice I wrote about on day two (not chugging the Soylent) and my body's unique sensitivity (or lack thereof) to the ingredients in Soylent. There's no obvious single answer; I guess the most I can say is that I got lucky.

Vitals

A friend suggested that I add heart rate to my daily objective vital tally since it's trivial to take yourself and doesn't fluctuate too much.

Objective Vitals

Weight: 161 lbs
Heart rate: 74 bpm

Subjective Vitals

Physical health: Good

Soylent: Day 4

2014-08-15T00:00:00+00:00

[This post is part of a series on my month-long experiment with Soylent. See the original post for an introduction.]

The Soylent itself has been getting steadily more palatable. Though I'm making the same mixture each day, it seems thinner to me now than it did at the start, and more enjoyable. I'm still not to the point where I'll want it if I'm hungry (though I'll convince myself to eat it in that case), but it's better than it was.

Vitals

Objective Vitals

Weight: 164 lbs

Subjective Vitals

Physical health: Good, but I pulled an all-nighter last night, so any subjective data from today is untrustworthy

Soylent: Day 3

2014-08-14T00:00:00+00:00

[This post is part of a series on my month-long experiment with Soylent. See the original post for an introduction.]

Today I made another observation about eating quickly. Don't. Just don't. I tried sipping more slowly like I wrote about in yesterday's post, but I guess I didn't slow down enough because now I feel almost uncomfortably full. In retrospect this shouldn't be surprising. Imagine eating a plate of spaghetti in a minute. Of course you're going to be full.

I've also noticed that a full pitcher seems like too much for a day. I'll admit that I've been snacking on normal food a bit, so maybe that's been making up for some of my calorie count, but I finished the pitcher I made for dinner two nights ago for breakfast this morning (that is, it lasted me almost two days). I haven't been very active these past two days, so maybe I'm just not burning much energy on a daily basis.

Lastly, I've found myself being actively hungry for something other than Soylent. As in, the idea of eating tasty food really appeals to me. I'm not sure if it's because the bland taste is, well, bland, or because I'm missing some key nutrients (or at least nutrients that aren't necessary but I had a habbit of getting a lot of before). I still feel full and satisfied after I drink it, but I have to will myself to drink it knowing that that will be the outcome; I don't find myself actively wanting Soylent itself.

Vitals

Objective Vitals

Weight: 162 lbs

Subjective Vitals

Physical health: Good.

Soylent: Day 2

2014-08-13T00:00:00+00:00

[This post is part of a series on my month-long experiment with Soylent. See the original post for an introduction.]

In my previous post, I mentioned that I had found drinking Soylent to be unpleasant, and that it gave me a queasy feeling in my stomach. I now think that that had mostly to do with the large quantity of candy I had eaten just beforehand because, when I had my first Soylent today (even without the strawberries that I blended in yesterday to make it palatable), I didn't find it nearly as unpleasant. Since I was more hungry, the sweetness was more enjoyable.

Besides the sweetness, I had made another mistake. I found a post online by a guy discussing the flatulence issue. He said that he hadn't had it so bad, but his wife had. He figured out that it was because she was drinking it quickly like she might a normal beverage, and they found that when she stopped drinking so quickly, and instead sipped it slowly like one might eat a normal meal, the issue subsided. I haven't yet had any issues with gas, but his point about drinking quickly was nonetheless relevant: I had drunk my Soylent quickly, and that probably contributed to my feeling queasy - it was like eating a big hamburger in under a minute. Today, I tried more slowly, and found that it helped.

Also, I should note one thing that I forgot to mention in the original post: I'm going to set a rule for myself that I can eat less than a day's serving of Soylent if I want to (though not by supplementing it with other food), but I won't ever eat more than a day's serving. If I end up getting really hungry this way, I may stop after a week or so, but I want to try it for at least some amount of time to see how it feels.

Vitals

Today is the first day I'm taking any vitals. I probably should have started yesterday - oh well. I don't have much equipment, so for the time being I'm just going to take my weight, and also record some subjective stuff like how healthy I feel (not mood, but physical health).

Also, I should note that weight fluctuates a lot day-to-day based on things like how late you ate the night before and how early you weigh yourself and so on. For that reason, I'd advise you not to read too much into the minor fluctuations, but pay attention rather to the overall trends. I'll try to write about these trends in any multi-day summaries that I write.

Objective Vitals

Weight: 166 lbs

Subjective Vitals

Physical health: Decent. I've got a bit of a headache, but I didn't sleep a full night, so that's probably why.

Soylent: The Blog Series

2014-08-12T00:00:00+00:00

For the next month, I will not be eating normal food, but will instead be subsisting almost exclusively off of Soylent. Soylent is a a powder which contains all of the nutrients that a human body needs to be healthy (at least, as far as modern science knows). When mixed with water, it yields a bland, brownish liquid that you drink.

What exactly do we call this concoction? The side of the box it was shipped in bears the tagline "powdered food." Some have referred to it as a "food alternative," or have suggested that they are eating Soylent instead of food. Most importantly, there's no shortage of articles and blog posts discussing the various philosophical aspects of Soylent - is it food? Will people want to eat it? Isn't food meant to be social? I have thoughts on these questions, but that's not the point of this blog series. I may write a post on the subject, but for the most part this series is about my personal experience with Soylent, and the practicalities of using it as a replacement for a traditional diet.

The Blog Series

As I mentioned, this blog series is focused primarily on my experience with Soylent and the practicalities of using it as a replacement for a traditional diet. I'm going to use it as a sort of scientific log book or journal, documenting how the Soylent tastes, how it's making me feel, whether I've stripped the paint off of any of my walls with my farts [1], and so on. I hope to post once a day. I expect that most posts will be fairly brief, and not much will change day-to-day ("Still haven't died; my housemates still hate my intestines."). From time to time I may do a review post noting trends that I've seen over the past x amount of time.

The Why

A reasonable question to ask is why - why am eating Soylent for a month? That's easy - I'm doing it to see how it affects me so that in the future I'll know if it's a viable option (I'm on meal plan at college this year, so I won't be using Soylent long-term for at least another nine months). The harder question is - why would I want to do it long term?

I have two main reasons. First, it's easy. I'm really not a fan of the chores of life - the stuff that you do so you can keep living the parts of life that you look forward to. While I don't consider eating to be a chore, it's a different story for cooking, cleaning, and shopping for food. When my life is relaxed and not too busy, this isn't such a big deal, but when it does get busy, time spent on those chores is time that I invariably don't spend on more enjoyable activities.

Second, it's healthy. I'm something of a garbage truck when it comes to food - I basically just eat whatever tastes good. This is fun while I'm young, but it's clearly going to come back to bite me later in life. And, yes, eating a normal healthy diet is obviously an option, but I know myself, and I know that planning, executing, and, most importantly, paying for a healthy diet isn't something I'm realistically going to do. Soylent is (as far as we know) very healthy, and it's cheap and requires no effort or planning.

Now, there's more to it than just that, but those are my primary reasons - the rest is just minutiae. Hopefully this little primer on my thinking will give context to this blog series and my opinions on and reactions to different aspects of Soylent.

The What

"So you're really eating nothing but Soylent for a month?" Well, no. Not exactly. What I want to do is test the efficacy of living off of it exclusively. But I don't think that occasionally eating normal food is going to mean that my results suddenly are wildly inaccurate. I'm not trying to get precise numerical results - I'm just trying to get a general sense of how an all-Soylent diet affects me. In light of this caveat, I'm going to allow myself a few exceptions:

An occasional snack or desert (so long as it doesn't constitute a meaningful portion of my daily intake)
Social eating (going out to dinner with friends and whatnot). I'll try to have Soylent in social situations whenever possible, but I'm obviously not going to bring a water bottle of Soylent to a restaurant.
Additives to make the Soylent itself taste better (so long as they don't constitute a meaningful portion of my daily intake)

Nonetheless, I'm going to try to stick as closely as possible to the spirit of going "all Soylent." I may adjust how much Soylent I eat (for example, if I'm not getting enough calories each day), but in general, I won't introduce non-Soylent food as a significant part of my diet.

The Unboxing

When you order Soylent, the first thing that you get is not the Soylent itself, but a "starter kit." It includes a pitcher for mixing and storing the Soylent, and a measuring cup which holds one half serving's amount of Soylent powder.

Then comes the real behemoth - the Soylent itself. I ordered a month's supply of Soylent, so it came in a box big enough to uncomfortably smuggle a human. Inside the box were four smaller boxes, each containing a week's supply of Soylent and a small instruction booklet.

The booklet is labeled "Soylent 1.0 Release Notes," a clear nod to Soylent's open source ethos.

The Soylent itself comes in seven packages - one per day - and seven small bottles of oil blend. A few of the ingredients in Soylent are sourced from animal products, and in an effort to remain vegan-friendly, these are packaged separately in the oil blend. Soylent's makers claim they hope to eventually find vegan alternatives so that they can sell Soylent as the powder alone.

The pouches themselves are, like everything in the box and the box itself, styled minimalistically - plain black text on a white background. They bear only the logo and nutrition facts.

The powder is beige and quite dense. If you drop the pouch on the counter, the powder settles with a single muted thud. It gives off a pungent, sickly sweet smell (I had just eaten five servings of Twizzlers as a last hurrah, so that probably lowered my tolerance).

The Preparation

Once Soylent has been mixed with water, it has a recommended fridge lifetime of 48 hours. The instruction booklet suggests ways to make single servings if you aren't going to eat a full day's worth within that time. However, I'm going all in on this, so it's not a problem.

The pitcher that is included in the starter kit holds one day's worth of Soylent - one pouch and one vial of oil blend mixed into the proper amount of water. This means that it conveniently serves as a measuring device - just dump the powder and the oil in and then fill the rest up with water.

The instruction booklet recommends a slightly more nuanced procedure. First, dump the powder into the pitcher. Then fill most of the way up with water. Shake vigorously for 30 seconds. Then add the oil blend and fill the remaining space with water. Shake vigorously again for 30 seconds.

When I first added the water to the pitcher of powder, it looked as though the powder was somehow resistant to mixing with the water. However, probably due to the fine grain of the powder, after 30 seconds of vigorous shaking, the mixture was completely mixed. The whole thing was pretty heavy, and after adding the oil blend and shaking for another 30 seconds, my arms were pretty tired.

The Consumption

The resultant liquid has the same sickly sweet smell as the powder, and I was afraid that drinking it would be quite unpleasant. However, it didn't taste nearly as bad as it smelled. I'd heard reports that it had a grainy texture, but I didn't find that to be an accurate description. I could certainly tell that it had been mixed from a powder, but the powder was fine enough that I couldn't make out the individual grains. Instead, it had a smooth but heavy texture somewhat like a well-blended milkshake made from dense ice cream.

However, there was something off-putting about it. I couldn't place my finger on what it was, but I found that drinking it gave me a sensation like what I was drinking tasted actively bad, though it didn't actually taste that way. In order to make it more palatable, I blended in four frozen strawberries with a glass (about one serving, or one third of a full day's worth). This helped, but I still found it off-putting. When I drank water shortly after drinking the Soylent, I got a mild queasy feeling in my stomach.

I have a few hypotheses here. First, maybe it just takes some getting used to. Second, maybe I shouldn't have eaten it on a full stomach (remember those Twizzlers?). Or maybe it's something else. I'm going to look around online tonight and see what other people say about it. I'm sure there's no shortage of advice online about what to expect and how to make Soylent more palatable. I'll write about what I find in tomorrow's post.

Conclusion

That wraps up this first post. I expect that future posts will be much shorter. Please comment or write with questions, thoughts, or suggestions!

[1] Many users have reported having pretty bad gas while their digestive tracts adjust; it usually goes away pretty quickly (a few days to a week).

Boston Go Meetup

2014-03-08T00:00:00+00:00

Yesterday (Thursday, March 7th, 2014), I gave a talk on reflection in Go at the Boston Go Meetup. It was a blast, and there were a bunch of great people there (and other great talks). Here are the slides from that talk.