Along the way, they explore the tradeoffs between simplicity and flexibility, the role of Docker and Kamal in making deployments seamless, and why ease of use matters more than raw capability. This conversation is a look at packaging complexity into something approachable and how ONCE aims to make running your own software feel fast, fun, and accessible.

Watch the full video episode on YouTube.

Timestamps

• 00:00:00 — Introduction
• 00:01:06 — What ONCE is and why it became open source
• 00:03:42 — Running multiple apps on a single machine
• 00:10:16 — Why single-machine deployments work today
• 00:17:47 — Balancing simplicity with flexibility
• 00:26:11 — Installing apps in seconds
• 00:36:47 — When to use ONCE vs. Kamal

Links & Resources

• ONCE GitHub Repo

Transcript

Episode Highlights (00:00:00): The vast majority of people, they want something that’s convenient. They want something that’s quick. They want something where they can get the value without all the hassle. Making it a little bit more flexible is really useful, but not losing sight of that super targeted flow was really important. Making something that’s possible, accessible, easy, fun, aesthetically pleasing, that’s the mission we’ve been on with ONCE.

Kimberly (00:27): Welcome back to Recordables. This is where we’re sitting down with the programmers from 37signals to hear a little bit about what they’ve been working on behind the scenes so you can learn a few things and learn from some of our mistakes. I’m Kimberly, your host, joined by my trusty co-host, Fernando. Hello, Fernando.

Fernando (00:41): Hello, hello.

Kimberly (00:42): And then we have back with us, Kevin McConnell, who’s been with us on Recordables before, and then a special guest. You know him, David Heinemeier Hansson. Our team has recently released a new app server. We’re going to talk about it today. It is called ONCE. You’ve heard that name before. We’ve rebranded a couple of things, so we’re going to dive into all those details today. David, why don’t you kick us off on what this new open source tool is?

David (01:06): Sure. So this is a continuation of all the work we did originally to launch the ONCE brand and the applications that sat under it, like Campfire and Writebook. And those applications were intended for the ONCE business model where we would sell you a piece of web application software that you could install and run on your own machine. And it worked pretty well, but it wasn’t kind of the rocket ship that it needed to be for us to continue in that direction as a commercial branch of what we do here at 37signals. And instead we thought, you know what? We’ve recouped the investment that we did into Campfire. We had already released Writebook just as a free tool under this setup, but there are factors in how you run these services or how you did run them that weren’t the best if you wanted to run multiple.

(02:02): So we thought, you know what? Let’s just make all of this open source. Let’s just make all of it free. And then let’s really upgrade the platform itself that these applications run on so that you can run all of it at the same time on a single machine. Because when we first launched Campfire, it was really easy to install. You got one bash command that you’d paste into your terminal and it would set everything up, but then it would kind of dedicate that machine or that VM if you were hosting in the cloud to just that application. And getting anything running next to it was kind of cumbersome. So we actually already encountered this feedback when we launched Writebook because there were a bunch of people who were already running Campfire and who liked to run Writebook as well. And we kind of had to tell them, “You need a second machine.

(02:52): You need a new VM.” Which I don’t know, for us who are very familiar with this stuff, perhaps wasn’t the biggest deal in the world, but it was a real blocker for a ton of people. So we thought now that we’re releasing all this stuff as open source and that we ended up releasing Fizzy under the same paradigm, it had a dual license, the O’Saasy license where we allowed you to do basically whatever you wanted on your own installation as long as you didn’t compete with our SaaS setup, but it functioned the same way as Campfire and Writebook. And now you really have three applications that someone might want to run and you’re not going to ask someone to use three machines to dedicate to that. That’s wasteful, it’s cumbersome, all of that. So as we were moving into that direction of making all this free and open source, we thought, you know what?

(03:42): We could do better. And one of the reasons we could do better is because we’ve already built many of the underpinnings that we’re using for ONCE for our commercial products. We built something called Kamal, which is the main deployment tool that we used to get out of the cloud. Where in the cloud, we were running some Kubernetes and all this other stuff that was just way too complicated. So when we moved Basecamp and HEY and all the other services onto our own machines, we built something called Kamal. And Kamal has kept growing over the last few years. And one of the things that it has gained is something called Kamal Proxy. And Kamal Proxy allows us to do a lot of the really nice things you need for commercial services like deploying a new version of it without taking the application down, not losing any in-flight requests.

(04:35): And we thought, you know what? That piece, that proxy would fit really well if we could put it into the ONCE setup such that when you’re updating any of these applications, now there are multiple of them on your setup, you also get that nice no downtime deployment. But just as important, you can run multiple applications on a single machine. So that was really the impetus to this. But even broader than that, I had from the beginning with ONCE, once we kind of corralled around this vision of turning it into complete open source deployment setup, that it should work more like Nintendo, like an old Nintendo with cartridges where you could have multiple cartridges. The older version of Nintendo was you bought Donkey Kong on this handheld and that was the one game. And Donkey Kong is a phenomenal game, but at some point you’re probably going to get bored with that.

(05:32): And well, here’s a piece of hardware that only works with that. The Nintendo, Super Nintendo systems and even the original NES, they all have cartridges. You could change them out and you could plug in a new game. And what I really loved about that setup was just the simplicity of it. Cartridge out, cartridge in, it just works. There’s almost nothing to it, but it is interchangeable. And I thought, can we bring that level of simplicity to something like web applications? Now, we’d sort of already done that with ONCE, but that was kind of like Donkey Kong and we wanted to do the full console experience with multiple games here, multiple web apps. So that was the drive, right? Thinking about this as a console experience, which actually when Kevin is going to show how it really works, it was a little bit of the inspiration too about the aesthetics.

(06:25): Can we make this feel like a bootscreen? Like you’re starting up a new game and it has some interesting visual effects to it. Can we lean something like a web app server into the direction of something that’s a little fun, something that perhaps blurs the line a bit between developers and consumers where tech savvy, not even that savvy, just marginally savvy individuals could actually use this, could actually run their own applications on it, even if they weren’t programmers. I think this is something we’ve seen now a lot with the AI revolution, that there’s suddenly a bunch of people who were not capital P programmers who are programming because we’ve reduced the barriers of entry. We’ve made it so easy, or these agents have made it so easy, to create everything from an iOS app, to a web app, to a Go App, to whatever app. So I think we’re getting more people in that direction.

(07:23): And then nicely enough too, that whole flow has been through its terminal. So Claude Code and OpenCode and Gemini CLI, they’ve all been tracking people who might’ve been a little unwilling at first into the terminal. And that’s basically all we’re asking anyone to do when they set up this ONCE web app server. We’ve even made it interesting point that by the time you set up ONCE, you can install all our three basic applications. So you can install Campfire, you can install Writebook and you can install Fizzy. They’re just selectable from a multi-select menu. Or just as importantly, in my little Nintendo metaphor here, those three applications are our first party. That’s our Mario Kart, that’s our Super Mario, that’s our Pilot Wings. But then you can also bring your own software. You can bring your third party Docker containers to the setup and they can run the same way right next to the three applications we’re shipping with.

(08:22): Anyone can make open applications, open source applications that are distributed this way, comply to a few small rules. Basically, I think we have two rules. One is your application has to answer on slash up or you’d have to configure where that is. If you answer on slash up, we know when your application is ready to serve, that’s very useful when you’re both setting it up in the first place. And then also when you’re doing updates that we can do that gapless deploy. And then secondly, that in almost all cases that it’s based on SQLite because we give you one directory called slash storage where you can put anything into and then it’s very nicely contained. So in that setup, it’s really designed for single computer deployments, which is also this trade-off. If you look at something like Kamal like I was talking about that we’re using to deploy Basecamp and HEY with, that’s deploying on dozens of machines at the same time.

(09:19): And there’s a lot of coordination, there’s a lot of auxiliary services. We need to have MySQL running, we have Redis running, we have all these other things running, which is great. Kamal is ideal for that kind of bespoke internal SaaS operations setup. ONCE is much more sort of the console stage. There’s just one machine that you dedicate to run ONCE on, and you can run as many applications on that as you can fit on a single box, but it is constrained to one box and the applications are constrained to one directory that they can save into, which means that in almost all cases they’re going to use something like SQLite. Now, the great thing about SQLite is it’s ideal for this purpose that these applications are meant for single user if it’s a personal application or single company deployments. So it’s not like you’re going to deploy Campfire and then run it for, I don’t know, a million different companies at once.

(10:16): No, when you run ONCE, each company has their own ONCE console. They click in the cartridge for Campfire and then they use it within their organization. And it just so happens that technology has gotten really good. Servers have gotten incredibly fast. Even tiny little mini PCs, let alone droplets or other micro VMs in the cloud have gotten so good that almost any size organization can run one of these applications even with thousands of people on that single box. So it’s kind of trying to take advantage of all these advantages that we’ve seen over just even the last few years. Docker is phenomenal for packaging this stuff up. We have counted technology with something like Kamal Proxy. Server hardware has gotten really good just that you can dedicate a single machine to it and that’s everything you need for even quite large deployments. The missing bit has been to make this so easy that almost anyone can do it.

(11:18): And when we were talking about this and originally said, no, I wanted to make it so easy in all the instances. I wanted to make it really easy if you just want to run this on your laptop. If you just want to run it for yourself, Writebook is a wonderful way of writing a book even if you’re the sole consumer of it. You can use Campfire with just a study group. You can do Fizzy just with you and your agents to coordinate work. These applications scale down really well to single individuals or tiny groups. But then of course, you can also take that and start it on your laptop and move it into the closet. You might have an old PC or something like that. You stick into the closet and suddenly you have all the firepower in the world. They can stay up all the time even when you close the lid on your laptop.

(12:06): And then at some point you might want to move this onto the cloud or a bigger service. So that’s also kind of the magic of the constraints around this slash storage directory because we’ve also built in backup and restore. So you can start an application on the laptop, do a backup, get it running in the closet, restore that backup, get it and run it on a cloud server if you want or move in any other direction that you care to. And I think that model of web applications that they are for single organizations fulfill a enormous amount of checkboxes that people are increasingly starting to care about. And a key part of it is you own your data. Now in the EU, for example, that’s become a really hot topic that lots of organizations actually can’t host their stuff on American servers or are feeling queasy about it and would prefer to have it on their own setup.

(13:06): And with something like ONCE, it’s completely trivial to do. The second thing is this software is yours forever. So if Fizzy, for example, 15 years from now is no longer operational because we as a business no longer are around, that Docker container can run literally forever. This is the other parallel of why I like this console metaphor. People are playing the original Atari Pong from 19, whatever it is, 76 or 79 or whenever that was from, they’re vintage machines. Now, if you go into the 80s, there’s tons of them, right? There’s tons of operational Nintendos and Segas and whatever that are perfectly fine and they still work. Do you know how many sort of SaaS style games today that were launched even five or 10 years ago that are no longer running because the company behind them went out of business and then everything that depended on the operational servers that’s just gone and then all this stuff disappears.

(14:08): And I think, do you know what? Web applications is such a great form of making applications, such a great platform, but it has been tied intrinsically to the SaaS paradigm that there’s some company that needs to be around and you need to be, in almost all cases, a continuous customer of that company, paying them every month or every year for the privilege to use those servers. And at any point you’re at the mercy of them going out of business, yanking the application, handing your data over to someone who asks politely for it with a summary judgment. None of those things are true with ONCE. So there are all these different factors that I find this is really interesting. Even if we couldn’t make the original ONCE business model work, maybe we were just too early and I want to make sure that we keep this medium, this approach alive because I think back of Campfire, it’s funny we launched that….

(15:04): Campfire in its SaaS version was launched in 2005 and we realized quite quickly that, oh my gosh, chatting with your own team, especially if you distribute is a complete game changer. Now, the market wasn’t ready for that at all. It wasn’t until Slack took the business world by storm in, what was that? 12 or 11 or 13 or whenever they really took off. And you’re like, do you know what? You can just be too early. I don’t want to give up on this. I think there’s a ton of really valuable properties to being able to run your own applications on a super easy to operate console-like platform, both the ones we make, the ones you might make, the ones anyone might make. And this could become a really powerful platform. And especially now that the cost of producing these new applications have really dropped.

(15:57): I did a screencast for introducing ONCE and during the screencast, I made two applications using AI. You did a HTML application that was just a snake game. And then I did an application that kind of didn’t work on the first go, well, written also in Go, but we can make it so easy now that the imagination of all these new people who weren’t capital P programmers can suddenly create these apps and let’s see where it goes. But that bit, having the console, I think is really a prerequisite because if it’s too complicated for you to share your vibe coded app with someone else, it’s not going to go anywhere. But if we make it really easy, and if we build it into things like Omarchy is going to ship with ONCE just pre-baked because it’s such a tiny little CLI setup, we can simply just have it as part of it and now there’s an install platform to use this.

(16:57): So anyway, this is why we built it. This is why we dreamed it up. This is why it has these influences. This is why the TUI interface forward looks really playful and fun and throwback to a Commodore 64 game. It’s drawing on all those kind of highlights of programming and our industry and trying to put it forward to something that could only happen today because all these moving pieces are now coming together, the hardware, the software, the packaging, the vibe coding, all of it.

Kimberly (17:32): Okay. Because Kevin, I know you worked on the technology for self-installing Campfire, Writebook, our original ONCE products. You’ve worked on this. Tell us a little bit about the challenges that you faced or anything that we should know about the changes with this version.

Kevin (17:47): Sure. Yeah. So some of the components are the same. Some of the ideas are the same. We were able to look at what worked well with that first Campfire version and kind of roll that into the next one. So the first version, as David kind of mentioned, the first version was we focused so much on how simple can we make this that we deliberately restricted a lot of the things that you could do. So there was only one path to go down. You went through one install flow, you told it one piece of information, the host name you were deploying to, and it would run, but that was kind of all you could do. And I think what we learned from that was that making it a little bit more flexible is really useful, but not losing sight of that super targeted flow was really important.

(18:33): So we kind of hung onto that idea. We still use some of the same sort of underlying components, like the way manages Docker containers to run is based on the same idea, but it’s just expanded a bit of flexibility. So where we used to just run a single container that contained the app and everything, and that’s all you would run on that machine. Now the tool basically makes us as its own Docker network and installs a proxy along with one or more applications that it can run. It sort of manages that group of things together. So it’s kind of the same ideas from the first one, but it’s just like leveled up a little bit to become a bit more flexible.

Fernando (19:11): That’s so cool. I think what’s really interesting about this is that we went from like, oh, the cloud, there’s this massive sprawling of dependencies things. And we’re like, no, let’s simplify that. And then out of that comes Kamal, then the obvious necessities of running Kamal, it’s like, oh, we need a proxy to just do no downtime deploys. And then we’re still like, you know what, this is great, but what if we can make it simpler? And out of all the tiny pieces and all the open source programs that we ship, we’re like, you know what, at some point it’s almost like Lego, which I find fascinating about ONCE this new platform. It’s like, well, we have this and we have this and we have this. We just plug it in here, you just run a Rails app with up and the other requirements and that’s it, right? Am I getting it right?

Kevin (20:03): It is. And also a lot of it is continuing that same convention over configuration idea that Rails has always done is like, rather than you have to set something up and say, “Here’s where I’m storing the data, here’s what you would need to do if you make a backup or if you need to restart it.” Instead, we just say, here’s a small set of conventions. If the apps adhere to those, then you don’t have to tell anything. There’s virtually nothing to do when you set it up because we know all the decisions, we already made the decisions. So I think that sort of philosophy continues into the design of this.

David (20:37): This is something I was really keen to focus on since there is actually already a fair amount of open source software out there that you can run on your own server. And usually when you go to check it out and you go to the install page, it’s like freaking three pages of this dependency and this piece of configuration and all this stuff. And I always thought, this seems unnecessary. There’s got to be a way where we can boil this down to just putting in, as we do now, we ask for your host name, where do you want to put it? You can put something in as simple as chat.localhost if you want to install Campfire on your own laptop. And then in that case, we even already know the Docker container because this is one of our applications. But if it’s another application, either a third party or something you wrote yourself, you just put in the Docker image URL and that’s it.

(21:30): Literally everything else is just taken care of for you. And some of this infrastructure that is actually quite important, you can do things like compose up, right? You can do your own Docker Compose file and then you go like, well, what if I need to upgrade it? Well, sorry. Well, I guess you could put Caddy in front and you could do something. Yeah, you could do a lot of things. All of this stuff was always possible if you wanted to dig the trenches yourself. Now, as I’ve found literally for 20 years, most people don’t want to dig the trenches themselves. They would like trenches just to appear the place where they want to have some water flowing. And as with Ruby on Rails, the magic is not necessarily just in making something possible. Almost everything is possible. This is something we say at 37signals, or I at least reply whenever Jason asked me or I wanted to design, “Can we do this?”

(22:28): I always say, “We can do whatever we want. It’s just software.” Literally everything is changeable. It’s a factor of how long is it going to take? How much concrete do we have to dig up if the architecture is really sticky in this area and in a way, everything’s possible, but that’s not how you get people excited about things. I mean, I had this exact experience with Omarchy. So Omarchy takes a bunch of components that already exist and just glues it all together. Anyone could have done that if they had 300 hours to spend on Linux by themselves. And lots of people did, right? That’s what that subreddit that I got hooked into our Linux point was all about. It was people who’d spend, if not hundreds of hours, then tens and dozens of hours on really perfecting their own system. And it’s beautiful when there are folks who are that engaged with the super nitty gritty of it and want to go real deep.

(23:28): It’s just a tiny niche. The vast majority of people, they want something that’s convenient. They want something that’s quick. They want something where they can get the value without all the hassle. So ONCE is trying to do that for something I kind of believe just should exist, like all this technology, Docker is a huge part of it. Without Docker, without this way of being able to containerize applications and bundle up all the dependencies, none of this would work. This is why, for example, we couldn’t have had ONCE in 2005 when Rails was first released. That was not going to be a thing because we did not have the technology. And then even when Docker first appeared in, I don’t know, was that early 2010s, computers were just not fast enough actually to run something like Campfire for a few thousand people on a single machine if you were going to build in these kind of technologies on a tiny box.

(24:20): So sometimes you just got to wait until all the little Lego pieces are ready, but then you also got to build the castle. You can’t just dump it all on people. Well, actually Lego is a bad example because I find it a lot of fun to put Legos together. I don’t know, a million piece puzzle. I can appreciate a finished puzzle. You’re not going to get me to put a million piece puzzle together. I don’t have 300 hours for that. I have 300 hours for putting a Linux puzzle together, but not an actual puzzle. So a huge part of this is just fiddling away all the little annoyances, all the little gotchas, all the things you’d have to document. This is one of the things that both Rails and Omarchy and now ONCE live under is this document driven development. If I have to write something down, we’ve done something wrong.

(25:11): I should not have to explain anything. You know what? When you unwrap that cartridge of Super Mario 3, you’re not reading through the manual. You’re like, here’s the slot, bam, jam it in. And if it doesn’t work, pull it out, blow on it for a second and jam it back in. It’s going to work. It’s going to boot. You don’t need to actually read anything. And that should be a model for software of all kinds. In fact, it’s been a model for our kind of software for a very long time. We didn’t use the cartridge metaphor. The metaphor we used to use was Fisher Price. It should be Fisher Price easy. Literally a two-year-old should be able to make this stuff work. And if it’s colorful, all the better. So getting all that stuff together, making something that’s possible, accessible, easy, fun, aesthetically pleasing, that’s the mission we’ve been on with ONCE.

Kimberly (26:06): Okay. We’ve talked about how easy this is. Kevin, do you want to show us actually how easy it is?

Kevin (26:11): Sure. Yeah. Let me just show you what it looks like to install an app or two with it. Okay. So if you can see my terminal okay here. So installing the tool is basically, there’s a command in the ONCE repo and on the website, we share this command with you that you just need to copy and paste into a terminal to sort of bootstrap the install. We have direct versions of this command for specific apps. So like Fizzy can have its own Fizzy specific link, which will take you straight to installing Fizzy, but there’s also just a general one for installing the tool, and then you can choose apps from there. So if you run that, that will pop up like the first running experience of what the TUI looks like with some of this rather 80’s aesthetic that David mentioned.

(27:02): So as we said, it knows about the three ONCE apps already that are built in. So if you want to install one of those, you can just pick one from a menu and go onto the next stage. If you want to install something else, you can pick this other option here, which would let you put in the path to any Docker image that contains a web app that would work with ONCE and you can install there. And one of the nice things about that, which we could probably talk about a bit more later is that this is also a path for you to take one of the built-in apps. And if you want to make your own changes, you can make your own fork of that app and just make a few changes here and there and publish the image up to your own path.

(27:40): And then you could use it in this sort of custom Docker image path to do your customized version of Fizzy or whatever. But to start with, we’ll just pick one of the built-in apps like Campfire. And then the only thing required piece of information that we have, much like with the old ones, is telling it what the host name is you’re installing onto. So if you’re installing onto a server on the internet, you would’ve set up a DNS entry that points to that machine, and that’s where you would type in here. And by doing that, we can make sure that we’ve got SSL certificates set up and everything and make sure the app’s accessible on that path. If you want to just run it local in your machine, you can just do a localhost address. So if I want to install Campfire chat on localhost, I can just put that in there and that’s basically it. It’ll go and fetch the container. It will set up the proxy, set up the Docker network and everything and land you back into this dashboard. And this dashboard is where you get to see all of the apps that you’ve installed and whether they’re currently running or not, how much memory and CPU they’re using.

Fernando (28:47): Wait, that’s it.

Kevin (28:48): That kind of thing. And that’s it. That’s all you need to do to install it. So these bits over here that list the app names are links. So I could click on that, which would launch the app that I just installed in a browser and you could start using it. You can also from this dashboard, as well as see how things are running, this is also sort of the central place where you can then carry out other operations or make changes to your app. So for example, if I wanted to just stop this app from running, if I wasn’t going to use it for a while, then I can use the menus in here to do that. So there’s a little actions menu that lets you stop and start or delete apps. If I wanted to install a second app, so let’s say I wanted to add Writebook as well, it would be just a case of selecting that, giving it its own name. It will run through the same dance. I guess I didn’t have this one downloaded locally, so it’ll just take a second longer, but you’ll see when this returns. So now we’ve got two apps running here side by side in the same machine because they have different names, the traffic to these will route automatically to the correct app.

Kimberly (29:56): Kevin, you said that took a long time or it took longer. Like that was really fast.

Kevin (30:01): Yeah, it’s all relative, I guess. I must have recently installed Campfire on this machine, which is why that one was like a second. Sometimes it might be a few seconds, but the whole process should always be really pretty quick. It’s a little bit dependent on your internet connection obviously, but these Docker containers are not typically massive. I think our apps are sort of in the hundreds of megabytes size normally, I think. So it’s basically the time it takes to download that is the slowest part of setting up an app. The whole thing should definitely be up and running under a minute.

David (30:37): What’s really nice about using Docker as the underlying mechanism here is that we get to rely on the Docker layers when we update. So if someone like us puts out a new version of Writebook and we update this books.localhost, you’re only going to get whatever layer changes, which is typically just the Rails layer. And those updates are even tinier. We’re not talking anything close to hundreds of megabytes to get that up. Docker is really efficient at sort of separating the things that change from the things that don’t change and then making those updates really quick to do. Which, Kevin, maybe you can show that just the calling for an update and just the fact that we actually do poll for updates too. So these are not just like a one and done install and you can never get any updates to the applications. These are somewhere between the cartridge and the steam game and the fact that they can actually get updates.

Kevin (31:35): Yeah. So there’s a bunch of background tasks that work to take care of all that stuff. And if you look in the settings menu from this dashboard, you can see some of the things that happen. So the updates part there that we’re talking about is that by default, this feature is checked on. And just by the nature of the fact that you’ve installed it, it knows that it should check periodically for those updates. So once every 24 hours, it will go do that Docker pull. If there’s changes, it will fetch the new version and then redeploy new version. As we were saying earlier, that’s like a gapless process. So there’s no downtime where it does that. It’s just a case of get the change Docker layers, reboot onto a new container that runs the newest version. You can also check anytime you want just by clicking that button.

(32:19): It’ll see if there’s anything there to get. There’s a similar kind of thing with backups. So if you want to have backups happen automatically, you can just specify a path on your machine where you want those backups to go to and then click on this option. And then once a day, it will make a backup of all the things that are in storage and all the app settings and everything like that. And put that in a file in that directory. It’ll keep the last 30 days. So you can just sort of set this on and forget about it, and you’ll know that you’ll always have a month’s worth of daily backups without having to do any other kind of maintenance on that.

(33:00): Some of the other things you can do is customize settings. So it’s all designed so that you shouldn’t … In the common case, you don’t really have to customize settings most of the time, but you might want to for some other reasons. So some applications might have special environment variables you could set to control how features in the app work or something like that. So we have a screen where you could enter some of those. If you want to do that, you can do things like for apps that need to be able to send emails, Fizzy is one of these. Fizzy use these emails as its way of logging in through Magic Links. So when you set up Fizzy, you’ll actually want to tell it where it can use an SMTP server to send email. So we’ve got a screen that you can pop open where you can put in the details of whichever SMTP service you’re using, which could be something you run locally, or it could be like a third party email provider that you use.

(33:57): Oh yeah, sorry. It’s got these application settings, which is basically the things that you entered when you first installed it. You can see that by installing … In this case, this was Writebook. So by picking Writebook from the menu, it knew that this is where it can get the official Writebook image. And so what I was saying earlier about being able to fork the apps if you want to make your own changes is that since all of these built-in apps are on GitHub and they’re open source, you can use GitHub to just create your own private fork, make some changes, commit those. And when you do that, the same CI setup that we have in that repo for building images, that will apply to your fork. And so you’ll end up getting your own version of this. So if I forked it, then I would probably have a path like this or something.

(34:47): So you can use this as a way to, a really low friction way to start making changes to some of those apps, try your own little customizations and just point your installed version of the app at your fork. And then when you apply that, it will go grab that container boot onto that with all your data. And it gives you quite a low barrier way to start trying out customizations or getting an AI to do some customizations perhaps. One other thing I was going to show here is just you can also use this to watch the logs of the running apps. There’s no traffic going to this one just now, obviously, but if it was, this would be like a scrolling screen where you can see the logs that are happening in real time and you can filter down certain things that match certain bits of text. So you can use that kind of as a way to just any, I don’t know, inspect what’s running, I guess, diagnose any problems that you might run into.

David (35:45): Now what’s pretty cool with this setup too is, as Kevin showed, you can install your own stuff. And Rails applications out of the box ship with all these kind of setups for using slash storage for storing the database and then having slash up. But you can use any language, whether you want to create something in Go or you want to create something in Node or whatever backend language you want to use, if you are able to package it up as a Docker container, which you can do with anything, you can ship it and run it on this setup. Now, this is where the question of, should I use Kamal or should I use ONCE sometimes comes in. And as we talked about earlier, Kamal is really for deploying your own private applications in a more sort of involved manner. I mean, you can use it for small things too, but it’s meant to scale all the way up to a Basecamp or HEY or even bigger applications where with ONCE it’s meant to be a console.

(36:47): It’s meant for one-machine deployments where the applications are open and public and where they can be shared more so than just for your own setup. So in the spirit of this, as you’ll see, at least right now, we also don’t have any form of authentication. The Docker URL that you put in has to be a public one. It’s intended to be a console system where we share all the cartridges. So if you fork Writebook and you make a really cool special version of it, you should have that available for anyone else to try it. So in that sense, there’s a little bit of almost like GPL-like spirit, even if none of the software’s actually licensed under that. But I like this idea that here we can have a common platform where we can collaborate and make it really easy to fork and try different things on and see where it goes.

Kevin (37:42): Yeah. So one other thing I thought would be useful to mention here, which comes from this integration between the proxy and the ONCE tooling is the fact that on the dashboard, we’re able to show things like the traffic that’s reaching your service and the number of unique visitors that go to your service. So these are being done in a way that’s, it’s not specific to the application. Like any application you install through this, we can get this information, but we do it because we can hook into some of the stuff that comes through at the proxy level and kind of inspect the traffic and figure these things out. So the traffic, obviously these apps I’ve just installed have zero traffic because I just installed them on my laptop, but normally you would be able to see a ray of requests here and which percentage of them are errors.

(38:28): Those are actually coming out of Prometheus metrics that Kamal Proxy exports. So the tool is able to just kind of query those the way a normal Prometheus dashboard would be. It’s just like a very mini version of it and it uses it to render this little needle that would go up and down as the traffic varies. And then on the visitor counters, we actually have a process that looks through the proxy logs to see all the requests that are being processed and it can count the unique IP addresses or client IP addresses that sees come through there. So client IP is kind of a reasonable sort of approximate measure of users. Obviously we can’t tell exactly which users are which without, there could be an application specific thing, but I think counting client IP addresses is a fairly good approximation of that. So we have a process that processes the logs, grabs the client IP addresses. It uses a thing called HyperLogLog, which is a way, kind of an efficient way to count unique values over time ranges without having to use tons of memory to store them all. And so that lets us calculate these unique visitor accounts on a daily and weekly basis. And we can do that efficiently, even if there was millions of them, it wouldn’t take inordinate. It’s like kind of a fixed amount of memory regardless of how many there are.

Kimberly (39:45): Okay. So we’ve talked a little bit about what we’ve done already. Do you guys have any ideas of what’s coming next for this that we can share?

David (39:52): Well, we have some secret plans for some of the things that could be coming next that we’re not going to go into, but there are a bunch of ways you could take this. I mean, Kevin and I talked about a bunch of ideas for, for example, having offsite backups as something that’s built in. Could you do something with SCP or whatever? There’s a million ways we could take this, but I also just do want to see if, are we, do you have the right timing for this? Is this now that something like this could work? And what we have now is everything you need to start doing this cartridge sharing. But what I love about all of this stuff is, as with the earlier versions of ONCE, we keep rolling forward. So there’s some experiment that have some degree of success or doesn’t have some degree of success or as logical says, I’m like, we just pile it all in.

(40:39): It just all goes into the mixer and then we put out something that’s the sum of all of those experiments and attempts. And I think this is going to inform and already has informed and work in parallel with all this stuff that we’ve been doing for Kamal or the ways we can make Kamal simpler. So there’s this symbiotic relationship where we get to experiment with this kind of stuff, especially since it’s open source. And now there’s this base of applications that people can use. It’s not just one thing. There’s literally three applications right now off the gate that you can install of ours. That’s enough to get things rolling.

Kimberly (41:15): Okay. Well, this has been an episode of Recordables. To hear more from our technical team, you can read their blog at dev.37signals.com.

But as launch day approached, the tradeoffs became harder to ignore. Kevin shares what worked, what got complicated, and the pivotal decision — days before release — to unwind months of work and revert to a more conventional setup. This conversation is a candid look at architectural bets, emotional attachment to big ideas, and knowing when to change course.

Watch the full video episode on YouTube.

Timestamps

• 00:00:00 — Introduction
• 00:02:18 — Exploring infrastructure options
• 00:09:08 — Making the app feel fast everywhere
• 00:14:17 — The per-customer SQLite experiment
• 00:31:05 — When the architecture started to feel heavy
• 00:42:22 — Choosing Plan B
• 00:46:00 — What we kept and lessons learned

Links & Resources

• Kevin McConnell’s Rails World 2025 talk: SQLite Replication with Beamer

Transcript

Episode Highlights (00:00:00): A year from now when the app’s growing and we have lots of customers and we’re trying to add new features, are we going to be kicking ourselves for setting it up in this particular way? The only other option would be to kind of go ahead and do it while not feeling confident that it was the right choice, if you see what I mean, which just that option just isn’t really an option. We risked having an app that broke, which no one wants.

Kimberly (00:23): This is Recordables, a place where the 37signals team shares their behind the scenes work building products like Basecamp, HEY, Fizzy, and open source products. We are sharing the behind the scenes of what we’ve done, how we’ve done it, so you can learn from us and avoid some of the mistakes we’ve made. I’m Kimberly with my trusty co-host in tech, Fernando. Hello Fernando.

Fernando (00:45): Hello. Hello.

Kimberly (00:46): We talked recently with Mike Dalessio about Rails’ multi-tenant structure of working with databases. This week we’re diving a little bit deeper, specifically into Fizzy and the infrastructure that we investigated with that product and ended up using. So to do that, we have Kevin McConnell from our programming team. Kevin, welcome to Recordables. Well, before we dive into the Fizzy infrastructure, tell us a little bit about you, how long you’ve worked here, and then we’ll dive into the topic at hand.

Kevin (01:14): Sure. So yeah, I’m a programmer here at 37signals. I’ve worked here for I think coming up on four years, something like that.

Kimberly (01:23): Nice.

Kevin (01:25): For probably about the first year I worked on kind of across a few of the products, worked on HEY for a while, worked on Basecamp a little bit, actually a lot, a little bit. And then I kind of joined the team building the ONCE products, which was part of what, well we’ll learn more about this I think when we talk a bit more about the architecture, but some of the things we explored in Fizzy, they sort of started out life as part of the things we explored in ONCE and kind of grew from there. So yeah, so that was my first, I would say the first three years we’re I’m working on some of the products and working on the ONCE for a couple of years and then the last sort of year in a bit has been more Fizzy.

Kimberly (02:03): Okay, awesome. And I feel like we could do a whole episode just about the ONCE products and how, if that all came to be, but we’re going to talk a little bit more about Fizzy today. Let’s start maybe with why we wanted to even investigate a different infrastructure, what we were trying to accomplish, and we’ll go from there.

Kevin (02:18): Sure. So there were really two reasons for it. I think there was one initial reason, and that’s actually the part that ties back to the ONCE stuff like I was saying. So the ONCE products, the idea behind those was that where most of our products are normally SaaS based and people subscribe, we run all the software on our hardware and people use it. The ONCE project was sort an experiment to see maybe people would like to buy software and run it themselves, kind of the way people used to buy software back in the day.

(02:50): So the notion was instead of subscribing, you’d pay once, but there’s also this, that was part of it, kind of the business side I guess was pay once rather than subscribing, but the technical side is run it yourself rather than us running it for you. And when we did those products it was quite interesting, but I think we found that some people really gravitated towards the idea of running things themselves. And so when it came time to start working on Fizzy, having done that a little bit, we had this idea about should we make a product that you could do either. So rather than be like Basecamp is subscription, but Campfire is ONCE, it’s like what if Fizzy was a thing where you could choose, you could pay for it once upfront and you would own it your own copy and you could run it yourself or you could just use a normal SaaS subscription thing.

(03:38): And so that’s what led us into exploring architecture as a way to answer that because normally you would build things slightly differently for those two use cases, if you see what I mean. So Campfire, the first ONCE product that shipped is very self-contained as a Docker container, it runs SQLite, so it’s kind of like maintenance free on the database side. Everything that it needs to do is built into a single Docker container. It’s really easy to run it, but for a SaaS application, that’s not what you would typically do. You would normally use database servers that could hold data for lots of accounts at the same time. They would be separate from your application server, you’d spread things out a lot more. Your job servers would be different than your application servers and that kind of thing. And so this was one of the angles for looking at new architecture for Fizzy was if we want it to work in both places, what should we build to do that? Should it be the same thing that works in two? Should it sort of be switchable in some way where you could change it depending on how you could package it differently for sales as a one-off versus us running a SaaS. So those were the questions for one part.

Kimberly (04:43): I would imagine too, there’s the maintenance question of keeping two versions for multiple people in different ways consistent with each other.

Kevin (04:53): Yeah, that’s definitely part of it. Anywhere you’ve kind of made a branch where you say it in this situation for doing it this way and the other situation we’re doing the other way, then you’re kind of potentially making things difficult for the future because you’ve got to make sure why any change you make has to work in both sort of thing. But that was half of the reason. The other reason for exploring the new architecture was really more about speed and performance. And I think it wasn’t the first reason for us to look at new architecture, but once we had the other reason, the ones thing, and we started thinking about the ways we might solve it, it occurred to us that some of them would have an impact on performance. And then I think we sort of changed gears a little bit and start chasing this idea of how can we make this really fast?

(05:35): And that became just as much or if not more of a driver of exploring new architectures. A big part of that really is just about, it’s about where you put data, locality of data to people. So normally when we run SaaS applications, the database lives somewhere geographically and we can use things like read replicas to make copies of it near different locations. But essentially you have one main copy of your data somewhere. So that could be in something like Chicago. The further you are from that, the slower the app’s going to feel to you just because of the latency, the speed of light limits and so on. And this is something that comes up a lot I think, because the kinds of speeds that you have to reach for an app to feel fast are short enough that common distances are too long for that to feel fast, if you see what I mean.

(06:27): So for me in Edinburgh, if I have to use an application that’s running out of Chicago say then realistically there’s probably like 100 to 150 milliseconds of latency for me to send anything there and back before the server even does any work, just the time for my request to get there and come back. The theoretical speed of light latency would be a bit less than that. It’d be like 60 or something probably, but in practice it’s going to be like 150 milliseconds to do nothing. If you’re actually doing something like rendering a page that takes 50 milliseconds to render, then that’s 200 milliseconds for a request. And in practice, I think it usually feels like if you can get your pages to render, like all your requests to complete in a hundred milliseconds or less, then the app feels nice and fast. More than a hundred starts to feel slow. More than by 200

(07:22): You really feel like something slow is happening there. And so this is an issue that’s, it’s hard to avoid if you have all your data in one place just because there’s going to be a bunch of people where no matter how fast you make the bits of your app, it’s going to feel slow to them just because far away. But if you took, for my situation being in Edinburgh, one of the other places that we run servers out of at the moment is Amsterdam. So if we moved everything to Amsterdam, it would be fast for me because my round trip to Amsterdam and back is like 25 milliseconds or something. So that 50 millisecond page that we talked about plus 25 milliseconds of roundtrip is nicely under that a hundred milliseconds. Does that kind of make sense?

Fernando (08:03): Yeah, yeah. The threshold.

Kevin (08:04): So that was a little bit of a tangent, but thinking about that stuff was what got us to think about as we’re exploring different architectures for running self hosted applications and SaaS applications, maybe we could take something from the model of the self-hosted ones where people would run things on their own servers next to themselves. Their data was generally right next to them. If you’re going to self-host something, you might be running on a server that’s in your office or closet or if you rent something on a cloud provider, you can pick a region that’s near you or something and you get the fastest data. When it’s SaaS typically you plop it in one place and that’s where the day is for everybody. For some people it was fast. For some people it’s slow. So we started to think about maybe we could take something from the self-hosted model. So even when we run it as SaaS, can we set it up somehow so we put people’s data near where those people are and then everyone gets a faster application.

Fernando (09:03): And how did that go?

Kevin (09:05): Answering that is jumping right to the end of the whole story probably.

Fernando (09:08): Spoilers.

Kevin (09:10): Yeah, I don’t know if you want to go through there bit by bit or just answer it, but I will say that you can quite easily make reading data local to people because you can take databases and replicate all the data out. So whenever you make a change in the place where the data is normally kept, the copies can go to different data centers and then whenever someone has to read their data, they can just read from one of those read copies and it’s near them and it’s fast. This is basically what we’ve ended up doing. We do this in other apps as well and Fizzy where we landed in the end is what we do. So all the writes are still happening centrally in one place, but all your reads come from the closest reader that there is to you. It actually works out pretty good in practice because most web applications do far more reads than writes. A lot of it’s just because the way you use apps, you tend to click around and look a lot of things and every now and again you’ll change something, but the ratio of reading to writing is really heavily skewed to reading. I think Basecamp I once lookeded and it was something like 94% reads, like 6% writes.

Fernando (10:17): That makes sense.

Kevin (10:18): So even if all you make faster is the reads, you’ve already helped most of the things most of the time. There’s also I think a bit of a perception about it where you kind, like changes that you make, sometimes it’s okay for them to feel a bit slower because it feels like you did something if you see what I mean. You’re sending an email, if you click a button to send the email and it took half a second or something to happen, it wouldn’t feel that bad. But if you’re trying to read through a set of pages on a website and every time you click on a new page it takes half a second to come up, that does feel slow. So yeah, we ended up focusing on speeding up reads by making them local and not worrying too much about trying to divide up the data and move it around to make loca writes because that’s where a lot of the complexity is from. But I can say that’s jumping to the end of this story because the way what we set out to do at the start was exactly split up everything and pre-reads and writes closer to people.

Kimberly (11:17): So Kevin, let’s start there on what did we try or what were the things we thought about in terms of here are our options for this new infrastructure?

Kevin (11:26): There were three, I think kind of three things that we thought were worth trying initially. One of them wouldn’t have helped so much with the speed, but more just about the packaging of the two different ways of use the software. And that was just to basically squeeze all the things we need on the SaaS side into the self-hosted side. So when we made the ONCE applications, they ran on SQLite because it’s nice and easy and convenient. We don’t run SaaS applications on SQLite typically because it’s a lot easier to sort of scale things when you have separate database servers and we have a lot of experience with making MySQL run really fast. So we usually use that. So one of the options that we had would be just take the way we do in SaaS, take by MySQL Everything and squeeze those into this self-hosted version so we don’t have to have two different ways of running software, but people can still buy it and run it themselves.

(12:23): That would’ve helped with the packaging sort of side, but it doesn’t really change anything about the speed. So that wasn’t the one that we decided to go to first. Another one that we kind of considered but quickly I think decided not to pursue is that because we knew how these self-hosted apps worked when running as individual Docker containers, which is the way the ONCE stuff works, we could always just basically host everyone’s own copy of the app on the SaaS site in the same way we could run a whole bunch of Docker containers so that everyone who bought it would have their own little container. And that’s what we would run to access things, which is in theory quite nice conceptually, but in practice quite complicated because it’d be really inefficient, like to do it naively would be very inefficient because there’s a lot of overhead in every container and if you have a hundred thousand customers running a hundred thousand Docker containers would be a huge waste of resources because most of them wouldn’t be active all of the time and you’d be using up a bunch of memory and CPU for things that are really being used.

(13:31): If you see, even for customers that are active, there’s a lot of idle time between actions usually. So you do something and then you don’t do anything for a few seconds and then you do something else. And if you kept a container running all the time, it’s just going to be very inefficient to do. And so to make that work, you end up having to invent ways to do things like have containers that you can run but then put to sleep between request and then wake them up really fast when the next request comes in, which some things do you basically be reinventing the fly.io service, which is kind of what they do is like run containers everywhere. The thing that would’ve been nice about that would be that we could put them anywhere and they would have the data alongside all the app servers and job servers and everything they need to run.

(14:17): So we could spread them around and let people run their version just like they would self-host it, we would just be kind of self-hosting it for them, if you see what I mean. But as I say, that isn’t something we pursued because I think getting that to work in an efficient way, it’s quite a big project. So then the third way, which is the one that we did decide to pursue and looked at way seriously for a while was the idea of giving everyone their own database but running those databases inside the normal sort of setup of app servers that we would normally have. So we might have half a dozen servers running Rails that would normally talk to one MySQL database with everybody’s data, but instead we would have half a dozen web servers running Rails that would talk to lots of little MySQL databases. Each one would be one customer’s data and we could put those databases on those app servers wherever we want.

(15:09): They don’t have to be in the same central place because they’re separate files. And it felt like a nice thing to look into because then we get this sort of isolation of data. It’s like one customer’s data is all in a single file and wherever you want to put that only affects that customer, if you see what I mean. And so that’s what this is part of what we pursued. That’s what became Mike’s Active Record Tenanted, which I know you talked to him about that recently and he’ll have gone and tell the details of that. But that was kind of the idea behind it was it lets you put data wherever you want without having to also chop up the rest of your infrastructure in order to do so.

Kimberly (15:47): And that also gets you the speed that you’re looking for because the data can be closer to each customer.

Kevin (15:51): Exactly. The data can be closer to people so they get less latency. It’s also quite nice because it was one of the routes that could make it practical to run SQLite as the database server. Like SQLite can scale to very large amounts of data and in the right circumstances, but if you want to run SaaS on it, there’s some challenges about concurrency, how many actions you can do at the same time on the same database. Like SQLite limits you to one write at a time. For an individual customer it’s probably fine. You can do a lot of writes very quickly and one customer is only going to be able to change so many things in a certain space of time that you can keep up. But if you used one SQLite database for all your customers, you can only do one write in your whole application at one time, you’re going to start actually getting locked contention bumping up against that just because lots of customers will try and do things at the same time. They’ll end up waiting on locks. But if you split the databases up into individual databases, those types of locks and those types of kind of limits, they apply a database level. So now it’s back to each customer can do one thing at a time. Not the whole system could do one thing at a time.

Fernando (17:05): So I have a question about that. I know this is Fizzy infrastructure, but isn’t Campfire running with SQLite?

Kevin (17:12): But we don’t run Campfire SaaS, so Campfire, right?

Fernando (17:16): But even what I’m wondering about is that each, I assume each message needs to be a write.

Kevin (17:23): Yeah.

Fernando (17:24): So…

Kevin (17:26): You can scale quite far and that’s not a problem because they’re all so fast. With Campfire, we did run into it sometimes when we were first developing Campfire and we would notice when we had done something where a write operation was slow, if a right operation took too long or if you start a transaction, try and do too many things before you the transaction or something like that, then you will start to see that. You’ll start to see the requests get held up behind the one that you’re trying to process.

Fernando (17:57): How does that look like for someone as a user?

Kevin (18:01): How would you know if that was happening?

Fernando (18:03): Yeah, yeah, because what you’re telling me is a little bit of programming magic, right? Oh, I can see the transaction and the requests queued up behind it, but is that noticeable for the end user?

Kevin (18:16): It would become noticeable when it’s bad enough. You would feel the sluggishness of the app is probably what would happen. So if you start to have requests waiting on locks, they’re sort of queuing up. And so a thing that you would try and do posting a message should normally be really fast, but you might post it and be waiting for the response to come back because it’s waiting to try and get the lock on the database, if you see what I mean.

Fernando (18:41): Yeah, that makes sense.

Kevin (18:42): So generally when working with, this is one of the things we learned with Campfire I think was when working with SQLite, it’s good to be conscious of how long you’re likely to be taking locks for. So you would keep your write operations short and efficient, don’t hold transactions open while you sort, if you want a bunch of statements to work as an atomic unit with a database, you can start a transaction, you can do some work, then you could go off and do some other things in your Ruby code while you finish processing things and then you can do more and then commit the transaction.

(19:13): And that whole time in the SQLite world, that whole time you would have a write lock that you’d be blocking anybody else from writing to the database. So it just makes you be a bit more careful about that.

Fernando (19:24): And one more question about that is that I assume that Rails is prepared for that. It just queues up the requests. This isn’t a Rails limitation, it’s a SQLite limitation.

Kevin (19:37): Yeah, it’s a SQLite limitation. Exactly.

Fernando (19:39): So you could have the write log whatever and Rails is happy to wait until you release that log and then continue processing the requests.

Kevin (19:48): Exactly. Rails doesn’t even really know about it. I mean as long as you don’t hit some kind of time out or something that you’ve set up, Rails wouldn’t even really know. It just asks the database to do something and that takes some amount of time. It’s just that would slow down. But as I say, if you keep your transactions short, it doesn’t tend to come up a lot in SQLite, at least if you’re using SQLite with the write mode, you have to use it in wall mode and that kind of thing. But you can go pretty far with it. But once you get into the world of running SaaS with lots of customers, then you would start hitting those limits. And so that was part of what was, I think interesting about this idea of giving everyone their own SQLite database because that meant since we’re not having these logs, then it could be practical to run SQLite as our SaaS database rather than MySQL.

(20:40): And that has some really nice properties and one of them that’s nice for speed as well as the thing of getting the data close to people to cut it and latency. There’s another effect that’s happening, which is more about getting the data close to the app server. So normally if you use something like MySQL, you’ll have a Rail server somewhere and then MySQL server running somewhere else, and whenever it needs to do anything with the database, that’s a network call between the two, and they’ll be close together in the same data center usually it’s usually a very fast network call, but still it does add some time.

Fernando (21:14): But why is this, I don’t know, a CPU limitation where you want to run them separately?

Kevin (21:21): Usually you’ll want to run them separately partly for resources, partly for scaling because you want to be able to add more of one without necessarily the other, if you see what I mean. So for example, you might start with having two app servers in one database and that’s enough load, but then as you get more customers, you might find your app servers are quite busy, but the database server is fine. And so you can add more app servers and get more capacity there rather than, I mean if they were just on the same one box and it got busy, you’d just have to keep making that a bigger and bigger box.

Fernando (21:56): That makes sense.

Kevin (21:56): So yeah, so this thing where because SQLite works, it’s not on a separate server, it’s not even a separate process from your Rails process. So anytime you need to access the database, it doesn’t have to go across the network. It doesn’t even have to go from one process to another and make some kind of IPC call. It can just directly carry out the query, it just grabs the data from disk or if quite often a lot of the active data will be cached. So it’s basically pulling stuff out of memory. So it’s really, really fast and it’s enough of a difference to be noticeable I would say. I think often in a typical web application, a request might do a handful of database operations, now a handful of queries, things that have to be separate queries really because they’re unrelated things, if you see what I mean.

(22:47): Quite often you’ll be loading up the user record to make sure they’ve got permission to see what they’re looking at. You’ll be querying for whatever it is they’re trying to see, like the Fizzy card that they want to look at or something. And maybe there’s some sidebar content with a menu in your pulling out. So you probably do a handful of different database operations and with a client server database like MySQL, that’s a handful of back and forth, back and forth between machines. When you do it with MySQL and you’re nodding it back and forth, it is a noticeable difference. MySQL is super fast, sorry, SQLite sorry is super fast when you use it that way. So that was one of the other things that was nice was like, okay, with this architecture we’ll get all the speed of having our databases inside the process and we’ll be able to move them close to the people, so it’ll be fast and yeah, that was one of the things that kind of drove us to chase this for a while I think.

Kimberly (23:42): I mean it sounds like a big win. Tell us when we started going down that path, what happened or what did we find?

Kevin (23:49): So a lot of it turns out to go quite well, but I think it starts to get a little bit complicated just because although those are the nice advantages to that kind of architecture. Basically the nice parts of using SQLite means that the data is right there inside your Rails process. But the flip side to that is that now your data has to live on the same machine as your app server. You can’t just put a database server somewhere and then have however many app servers you need that all talk to it. And as you get more load and more customers, you can add more app servers. They have to be on the same machine. So if you want to increase capacity, then you can add servers, but then you also have to move the data. So you have to take some of your customers that were stored on server A and put those on server B, which starts to become a bit more complicated.

(24:46): You also have to have a way to not have each particular server be the only place that customer’s data lives. And you need to have some kind of redundant copies because that machine could fail suddenly and then you don’t have access to customer’s data. If you want to spread out the reload to multiple machines like we talked about before, then that also needs a way to replicate data from one server to another because otherwise it’s always in that one place. One thing I probably should have mentioned about this idea of making data close to people is it sort of assumes that people tend to be accessing the data from the same place. So you expect all your customers should… if all your customers work in Edinburgh and you put your data in Edinburgh, that’s great for them. But if they’re like 37signals and everyone’s spread all over the world, there’s not really a good spot that’s good for everybody.

(25:40): So you can’t avoid still trying to find some way of doing the thing of like, well, we can spread the read on the copies around and everyone gets fast reads. You still need to do those types of things if you want to work well in situations like that, if you see what I mean. When you want to do that with SQLite, there wasn’t really a good way that we found to just do that with existing software that kind of fit our use case well. So we had to build that part too. So there’s one of the places where the seemingly simple architecture starts to get a bigger project, a bigger project like, well, this bit is good, but we also need to build this other thing. We also need to build this other thing. And so it starts to grow arms and legs a little bit.

Fernando (26:23): And some of these concepts seem deceptively simple, right? From what we learned yesterday with Mike SQLite is just a file. So at first it seems like quite simple, you just copy the file everywhere. You just do it right and then copy the file somewhere else.

Kevin (26:42): Yeah

Fernando (26:44): That should suffice. And then I assume that when you start doing something simple like that, oh, there’s this edge case that you know when this happens.

Kevin (26:51): Yeah. Exactly. It is just a file, but it’s a file that is being changed in various different parts of it, potentially very quickly. And you need, so in the case of replication, you want another copy of it that you can use for reads. Then whatever changes in that first file has to change in that second file. But it has to do it really quickly, and it has to only do the parts that change. It can’t recopy the whole file because it’ll get too big. You would end up try to send far too much data across the network if every time you made a change, you copy the whole funnel.

Fernando (27:23): How big is the SQLite file, like in your experience while we were testing this?

Kevin (27:29): So it really depends. Since it’s per customer, I mean I think a lot of customers will be quite small. So megabytes probably. Big customers might be a gigabyte or something. They’re not massive, but they’re big enough that you couldn’t copy the whole thing each time someone changes. One thing I was getting to that you need to at least find out what’s the part that changed and then just apply that change. And you want to do it as close to instantly as possible because you want people to be able to use those read only copies to access all the reads without them getting behind and showing stale information, if you see what I mean. So we built a system for replication. We also did build in something about that notion of stale information because you can’t make the replication actually instantly. There’s always going to be some amount of delay and usually it should be short, but I think there’s always a potential that it gets held up for some reason because essentially you’re sending changes across networks between servers.

(28:35): If a lot of changes happen in one place, it’s possible that it might take a moment for all of those changes to get across the network, get applied in the other side. It shouldn’t normally be long. It should be well under a second most of the time, but there’s always a chance that a lot happens suddenly or something. Or just computers. It might just get slow for a little bit. So you usually need some mechanism to make sure that what people are looking at is not stale data. You wouldn’t know it’s stale data really if you weren’t the one making the change. If you’re looking at a page that someone’s updating and it takes two seconds for the change to show up, you’re probably just not going to know. You didn’t know it was there until you saw it, and that was in two seconds is nothing.

(29:22): But the case that quite often comes up is if you change something and then you go to another page where it’s you make a new post, add a new card in Fizzy or something, and then you go to the list of cards. If yours isn’t there because you read from a server that didn’t have it yet, it’s going to look broken. Normally, or I think commonly the way people handle that, which we do in some other apps is just have a short delay where you pin your activity to the place where you wrote it. So when you make the change, it has to go where the writer is. When you go to make a read, normally you would read it from the reader, but because the system knows that you just wrote something, it’ll make sure that you actually read from the writer for the next second or something, which works well.

(30:06): But it does mean that more of the requests go to the writer than have to, because it’s a pessimistic kind of approach, right? It’s like we don’t know if it’s in the other place yet, so we’ll just assume the worst and keep serving you from the writer for a couple of seconds until we’re sure it would be there. So in the architecture that we were building for Fizzy, we were quite conscious of trying to avoid doing any more work on the writer than we absolutely had to because we have these kind of tighter restrictions about how many writes we can push through in each app server and how many customers can go on each app server as a result. So we took a different approach there and instead we actually track what the idea of the last transaction is that you wrote. And then every time when your request comes in, we are able to assume that it’s probably safe to serve it from the reader and send you to the reader, but detect if we were wrong.

(31:05): And so in those rare cases where your next request goes to the reader and it doesn’t yet have the transaction you just wrote, at that point, it can quickly resubmit the request to the writer to get the fresh copy, which means it potentially might be a wee bit slower if that happens, but that very, very rarely happens. So it gives you the protection to never seem broken, but with the more optimistic behavior, that is fine to just read from the reader all the time. I feel like we just went down a really deep rabbit hole there. I forgot what we were talking about before we went down there.

Fernando (31:38): No, no, no. I think this is great. The reason is it seems like, wow, it seems so simple at first, and I’m sure you felt this and you and the team felt this, but I’m sure the desire to make something like SQLite work was really, really high.

Kevin (31:56): It was. I mean, I think it was at the start because it seemed cool basically.

Fernando (32:02): Yeah, it is cool.

Kevin (32:03): The advantages seemed good also. It just seemed like a fun thing to do and we all grew to really SQLite a lot. I think a few of us had used it and quite liked for other things, but the more we worked with it we’re like, this thing is cool. We just want to make this work and use this. And I think also there’s a thing where the more time you spend on a project, you also get a little bit emotionally attached to it and there’s a why get to the point where we’re like, I’m going to make it work because I said I was going to make it work. That like, it’s not going to beat me. I’m going to do this. Which actually, I think probably for me at least, it led me to chase it for a bit longer than… I think in hindsight, I think there was a point where I probably, it would’ve been better to stop, realize that some things weren’t going quite the way we wanted to and then change tech earlier. But instead, I was so determined to make it work because it mostly did as well. That’s the thing. Most of this did turn out to work really quite well.

Fernando (32:57): Right, and I was going to say, there doesn’t seem to be a huge wall in front of you, right? It was just tiny pebbles that made you like, oh, clean this up. I need to move this here. But they just kept coming.

Kevin (33:15): Yeah, well…

Fernando (33:18): Or was there something huge where you’re like, oh.

Kevin (33:21): I don’t think there was anything so much huge. I actually think it was a bit more of, it became a timing issue to get it done because it did turn out to be quiet quite big and quite hard to crack. So the way it played out in practice was that we built some of the parts of this first. We had the tentanted SQLite part. We had replication working. We set things up so we had one app server that was the writer, and then we had a couple of readers that replicated from that. We set up this geolocation routing stuff. We use. CloudFlare is a load balancer service that you can use to sort of direct traffic to one of your, like to the nearest data center from your data centers. So we put these servers in different places. We had traffic routeing to the right one, and that worked really well.

(34:15): And that’s what we used internally for a long time. We had this sort of small group, this sort of invite only early testers group that were using Fizzy for a while, we’re running on this. And it was great. It was super fast and worked really reliably, but as we started to add the multiple writer part, there’s a lot of things in there that are quite difficult to get right. So it took a while, I think to land on a design for that that we liked. We got there in the end, but we were already, we’re getting later and later in the project. And so we were working on the infrastructure part while it was basically for the longest time it was just Mike and myself working on this, and then Stanko joined. So there’s three of us towards the end. But while we’re doing that, there’s other people working on the product and so early on it’s great.

(35:05): You have all the time in the world, they haven’t built the product yet, so you can take your time figuring out the infrastructure, but we get to a point where the product’s good to go, ready to go out the door, and we are not quite ready with the infrastructure. We still have some things to figure out there, and for a little while you could kind of get away with that. We can be like, well, if we need a couple more weeks to finish something up, then they’ll add more things to the product for a couple more weeks. There’s always more things to add, but I think we were getting to the point where it started to feel like we’re going to end up slowing this down. We’re going to end up not being able to release a thing we’ve built just because we still have questions on our side.

(35:44): We figured out how we wanted to do the multi-writer thing. We built that kind of towards the last minute in project terms, the last minute. We got that stuff all working. But I think for me, there were two problems that we were staring at. One was that although we kind of got this stuff working, we had intended to do a lot more preparation in terms of how we were going to run this and make sure that we were prepared for whatever happened. So having run books for how to handle operational situations like if a machine breaks or if something goes wrong with a replication, then it gets, there’s a lot of replication lag. How do we deal with that? There were a lot of things that we ideally would have practiced and researched and written up. So we knew that we were prepared so that when we launched this for real, if anything went wrong, we could quickly recover.

(36:42): And we hadn’t really done enough of that by that point, mostly because we were so busy figuring out these other questions that we had to keep pushing that further down the line. Same with benchmarking. We’d done some basic benchmarking to know how fast this was, and we knew that it was pretty fast, those tests. But I think there’s always the potential that there’s kind of limits and ceilings that you haven’t uncovered yet. And to be confident in releasing the app, I think we needed to have spent a bit more time trying out different benchmarking scenarios, knowing that if loads and loads of people sign up on day one, it’s not going to catch fire because of some limit that we didn’t know about or something. So that was half of the concern was just not being quite ready enough to do it in a responsible way. We could have shipped it and launched it. I think it would’ve probably been fine, but if it wasn’t fine, I think we would’ve had a bad time. And we risked having an app that broke, which no one wants, right?

Fernando (37:40): It’s all fun and games until you get the customer data loss.

Kevin (37:45): Exactly because all this time we’ve been running internally, it was working well, but the stakes were low at that point because if it broke, we would just go, oh, I wonder what went wrong. We’d fix it and we’d put it back together. But I think the point where it’s out in the wild and people are using it and also especially the early point where you’re trying to tell the world you’ve made this new thing and you want everybody to come and look at it at the same time, you don’t want it to break in that moment. And as I say, it’s not that I thought it would break, but it was more that I didn’t feel confident enough that we had made sure it wasn’t, or that we had made sure that we could fix it really fast if it did. There was an amount of preparation that we hadn’t been able to do in time, I think.

(38:29): So that was half of it. The other reason though was that for all that we liked about that architecture, the longer we worked with it, I think we start to feel like there were some parts we didn’t like about it as well. There’s some parts that become harder and a lot of it just boils down to that same, it’s that same constraint that you have where if you’re using SQLite and your data is where your app server is, then even though you can have your read only copies and you can read from multiple machines, if you want to change the data, you have to change the data on a specific machine. If you’ve divided your data up per customer, then most of the time that’s fine because usually you know for any specific request, this is only for this customer. I know which machine has their data, I can route the request to the right place and they can do the write there.

(39:19): But sometimes you have requests that aren’t like that. There’s some things that do span customers. So for us, it came up with things like the way login works. When you want to log in, you need a way to enter your credentials and get authenticated before we could show you, here’s all your accounts. Those accounts are essentially the different tenants. And so we need to find out, we need to get you authenticated, show your list of different accounts, even though they’re all on different machines. So the information about you and how you log in can’t be just in each of your tenants. It’s a layer above that. It’s across all of those. And so as we were working on some of that side of it, we started to run into these situations where things that seemed like they should be easy were hard because you’d have a feature…

(40:12): I think one was that we had, I’m trying to remember the exact detail, it was something like we wanted to make it so that your profile picture wasn’t per account but was per person. You have the same profile picture regardless of accounts. And initially we had built this into the tenanted database because it was a per account thing. Changing that normally would be super easy. But in this architecture, we’re like, well, we don’t even know which database is holding the information about what your profile picture is, and if it’s not the machine where your account’s on now the request of this machine has to talk to this other machine. And we kind of worked our way through it, but we just started to feel like we’ve made things harder for ourselves here. There’s a lot we liked where, I dunno, I was going to say where we made things easier for ourselves.

(41:00): I dunno if it’s exactly fair to say we made things easier for ourself, but more like there was a lot where we could do things the way we normally did and get benefits. But then there’s these other cases where we’re like, this is kind of awkward. I don’t actually know if we’re going to be pleased that we did this a year from now when the app’s growing and we have lots of customers and we’re trying add new features or are we going to be kicking ourselves for setting it up in this particular way, if you see what I mean? And I don’t think it’s super clear right now whether it would’ve been, I think we were happy about or I think we weren’t happy about, but there was doubt. And so I think the combination of, we’re not quite ready to go with this, but the app is ready, the product is ready to go, that combined with, and we’re not even really that sure we still want it anymore kind of thing. That’s why we decided that the right thing to do for that situation was actually unwind some of that and go back to a more traditional for us architecture and ship on that.

Fernando (42:05): And I believe what’s interesting about this is that it’s in the git history, right?

Kevin (42:09): It is. Yeah. So actually I can show you, I have bookmarked the right PR for this in case, so that I could kind of show…

Fernando (42:20) The exact point in time. We were like, yeah, nope.

Kevin (42:22): Yeah. So it was a dramatic week in some ways. So the decision to go from what we were going to do to what this PR here is called Plan B, because we always had this idea of behind that if this didn’t work out, the plan B was that we would just convert the app to run on MySQL, use the same kind of set that we typically use for our other apps. And so the decision to do that, I think the day on this says November 18, I can’t remember the dates very well, but if this is November 18th, that probably means that I think November 19th probably was the planned day for shipping the app. And I think the day before this PR is probably when I kind of went, I don’t think this is going to work. I think it was on a Sunday evening, I pinged David to say, I think we should change. I think we should do Plan B. We should bail on the new architecture. It was literally two days before we were supposed to ship it.

Fernando (43:25): That must have been difficult.

Kevin (43:28): It was, like I said earlier, I think in hindsight it makes sense. It still feels like it was the appropriate thing to do in that situation for all the reasons that we just talked about at the time. But at the time of doing that, it was quite a hard thing to do because we’d invested a lot of time in it. I was really, like I say, I was determined to make it work. I was kind of attached a bit to the idea of shipping. It felt like failure to not do it. I knew it was the right decision to do, but at the same time, you get for one thing, you get kind of emotionally attached a bit to the project you’re working on for a long time, I think. But also, we had talked about this a lot. I gave a talk at Rails World about Beamer and how we were using it for Fizzy, and David and his keynote talked about how we were doing this whole new architecture and we’re going to change the world with this new architecture. So all of a sudden to be like, we’re not actually going to be doing that. It’s hard to not have that in your mind when you’re deciding to change the plan.

Fernando (44:29): For sure. Just knocking on David’s door, like hey. Have a second?

Kevin (44:32): Yeah like, sorry I told you we were going to do this thing, now we’re not going to do this thing. But it did really feel like the choice between, you either just put your hands up and say, you know what? This hasn’t worked out. We need to do something else. Or the only other option would be to go ahead and do it while not feeling confident that it was the right choice, if you see what I mean. Which that option just isn’t really an option, I think.

Fernando (44:58): It’s already been a few months and Fizzy’s out. People love it. They really, really love it. Do you see it as a failure or is this just like software development?

Kevin (45:10): Mostly the latter. So I think exploring this made a lot of sense. I think we learned a lot from this. There’s things that we took out of this like Fizzy, although we reverted to a more conventional, for us, architecture, it’s not exactly the same. We did get to keep some things from our explorations on the other architecture. So for example, the thing I described there about dealing with replication lag by tracking the transaction that you wrote rather than pinning all your reads to the writer for some period of time, that we figured out how to do that when working on this new architecture and when we switched to Plan B, we kept the idea and we ported it to the MySQL side. So now Fizzy still has that improvement because we had already put the work in while we were doing the other stuff. And then there’s a couple other things like that.

(46:00): Part of that I didn’t really talk about much earlier, there was that as well as replication and the database location stuff. The other thing that you have to do in all of this is routing requests to the right machines, you have to have a more, I dunno the right word to call it, but more dynamic I guess sort of routing. So when requests come in on a particular data center, it has to know which is the right server for this customer for this action. And so there’s a bit more behavior going on at that kind of level. And so we built a bunch of stuff into Kamal proxy, which is the proxy server that we have in Kamal our deployment tool. We built a bunch of load balancing features into it so that we could build this original architecture. But when we switched to the other, the MySQL version, it turned out to still be really good to be able to do this at that level.

(46:55): So we still use Kamal proxy as a load balancer. We have six load balancers that run Fizzy that are all Kamal proxy using the same new stuff that we had built originally for the first architecture, if you see what I mean. So we did take some stuff out of it. We didn’t throw it all away, but we did change a lot. So I was going to show you the PR just to give you a sense of what the work was like to do it because it was quite a sudden and big change in a sense. So one thing I didn’t mention is that there’s 14 participants it says in here. A couple of these are just people who I think commented and discussed things. So it’s not exactly 14, but there are probably eight or 10 people involved in doing this change. It has tons of commits.

(47:43): It’s one of those things that GitHub is not very good at displaying. It has too many things, but it’s basically a week of work by the whole team. So it was kind of a intense week where we all said, all right, let’s make this work. Let’s go with plan B. And we all made the changes. The changes themselves, most of them are not actually that difficult or complicated, it’s just that there’s a lot of ‘em. So I dunno if it’s interesting to the sort of thing that we had to do to change this. I could probably point a couple of things.

Kimberly (48:18): Yeah, let’s do that.

Kevin (48:19): The way this played out, you could see at the start I actually dunno why this first commit that wasn’t on the PR before. That’s just a weird git history thing. But we basically started by pulling stuff out. So Beamer’s the replication system that we built. So we weren’t going to need the replication anymore. We weren’t going to need this test bed, which was there to test how replication worked across machines. We added the trilogy adapter, which is the MySQL adapter. with two active record tenanting. And so you could see, we basically started this project where we’re like, take out all the new stuff, set it up on MySQL, and then there’s a bunch of, and then make it work things. So there’s a lot of update queries, fixing tests and stuff like that until it worked against MySQL. And a lot of it was, as you just described, there’s a lot of places where in the tenanted world, the database had the data for just that one customer or that one account.

(49:18): And that’s usually the level that you query things at. So if you want to look up your account, we didn’t have to say find the account for the customer, we could just say account.sol. It would just be like the account. And that would give us the account information. If we wanted to get a list of all the boards in your Fizzy account, we just query for all the boards because they’re all your account because of the database they’re in. Obviously once you stop tenanting and you go into this model where one big MySQL database has everybody’s data, then you have to go through, if we find the models down there, if you look at something like board, the model, yeah, there’s tons of this sort of stuff. So board had to have a belongs to account because before it didn’t matter. There only was one account. Now it has to have that relationship. And then all the places where we do queries, we have to make sure that we’re actually accessing things through current account or current user rather than just the only account. So there’s a lot of mechanical changes that are of that sort of form.

Kimberly (50:31): Kevin, is this made any more complicated because we had beta testers, like people with actual account data, not just our team in there. Did that make any of this change unraveling any more complex or would it have been the same regardless?

Kevin (50:47): It was a bit more complicated because yeah, we had people using it. We wanted to make sure that everyone’s data was preserved properly and we also didn’t want to interrupt their use of it too much, even though it was pre-release. So it was sort of okay to say we’re taking it down for maintenance, but we wanted to minimize the interruptions. So yeah, we couldn’t just start fresh with a new database. We had to write scripts that imported the data from all the individual tenanted SQLite databases and then copy them into the MySQL world.

Fernando (51:17): I saw in one of the commits that it says Remove Beamer. Beamer is not yet open source, is it?

Kevin (51:26): So it’s not yet. I want to open source it, but there’s a couple of things I wanted to tidy up to make sure it was ready before it goes out. And because we were quite busy getting Fizzy launched, at the point where we stopped, we decided we weren’t using it, haven’t really had a chance to go back and just make sure it’s ready, I think. It works. We were using it for a long time, but there are just a couple of rough edges that I don’t want there, and I think I want to share it because we built it and I think it works quite well. It’s super fast. It turned out to be pretty reliable, but I think as soon as you share something, you kind of… you only want to do that if you’re ready for other people to use it. If you see, I mean, I don’t want to share it insane, but it’s a bit wonky over here or not be around to help if someone has questions or something. Do you see what I mean? Kind of want to do it, share it responsibly if you see what I mean. So I will.

Fernando (52:23): I think that’s really interesting. How do you marry that with David’s very famous, the gift philosophy.

Kevin (52:31): I do think it’s a gift. I think you can give things to people and they don’t… I don’t think people have the right to demand that you start doing certain things like add features that they want if you don’t want them. I think that’s kind of where the gift thing come from, right? If you make a thing and you think other people can benefit from it and you want to give it to them, then you can and don’t really, they’re not really entitled to then demand more things. It’s a gift they could take or leave it. They can build their own if they don’t like the one that you made. I think that’s fine. But I do think also there’s an amount of just giving people something in a good form. You don’t want to waste people’s time by saying, I built this thing, it’s 95% done. It’s up to you to finish it or something. You know what I mean? It’s sort of a balance though I think.

Fernando (53:21): I get it. And this is still your baby in a way., right? I know this is a team effort and everything, but from what I’ve heard, Beamer is something you’ve poured a lot of time into.

Kevin (53:33): Yeah, because although all these things are, they’re team efforts, but we’re a small team, so in practice, in practice you might find that there’s two people or three people working the thing and there’s two or three things that you need to make for it to work. So you tend to end up each having a thing. A lot of the things we build end up being mostly built by one person, especially those kind of supporting tools and stuff. It’s usually one person that came up with an idea and built a thing.

Kimberly (53:59): This path that we went down with Fizzy infrastructure that we didn’t end up using. Do you imagine that in the future we’ll revisit it? We’ll try to go back to it given more time. It seems like there was a time element where the product was ready, the backend wasn’t. Given enough time, do you think it’s something you’d want to re-explore?

Kevin (54:18): Think there’s parts of it that we want to re-explore. I think we learned a lot from it that I think informed how we would do it again if we were to do it again. And I don’t know that we would go back and exactly continue that same journey and finish the exact same thing. But I think there were some things that we thought that we were hoping to get out of it that we ended up because we didn’t ship on that version, we don’t have these things that we might want to go back and say, well, how would they apply in this new architecture? So one of the things that I’d really like to look at, we will try to do that soon, think is some other form of doing the local writers to people. So even though we don’t have individual tenanted databases per customer that we can move around to different places,

(55:07): right now we have one big MySQL database with everybody, but we could have four or five or something. You could have the European database, the East U.S., the West U.S. and segment customers into a small number of large databases and use a lot of the existing, the load balancer routing that we already built. I think we could apply a lot of what we built for the MySQL version onto, sorry for the SQLite version. We could apply a lot of that onto a sort of MySQL sort of form of this and still get that benefit of, now my data could be in MySQL at Amsterdam, and so it’s faster here in Edinburgh. So I think that’s more like where we might go back and look at, it’s like what could we pick and choose from the things that we did and apply them in this place.

Kimberly (55:57): That makes sense.

Fernando (55:58): Listening to what Mike had to say about Active Record being multi-tenant and all of this, it seems like there’s the gem that he is working on. The Kamal proxy load balancing stuff, the Beamer, which in the future may come up and Fizzy overall, there was no actual real harm done by this exploration. It seems to be that it was like a net win even if we didn’t get a hundred percent of everything that we wanted, right?

Kevin (56:31): Yeah. Yeah, I think so. I think it was a really useful exploration. I think we learned a lot of stuff and some of the things you just mentioned, there are things that we have kept. It’s not like none of it shipped. It was more that we shipped some parts, we didn’t ship other parts, but we did still get quite a lot out of it. I’m quite excited with some of the load balancing stuff that we built into Kamal proxy and that I think we’ll be genuinely useful to a lot of people. We still have a little bit of work there to make it easier for people to set up those load bouncers and we’re going to add some things to Kamal to do it, but I think that’s actually going to be really nice for people and that came out of that work. So I think it was all good.

(57:14): I think, like I said earlier, I think the one part, in hindsight, I would’ve liked to notice maybe one month sooner than I did that we were going to change our mind. So instead of having that two days before launch going, let’s change everything and then we had to have that crazy week, I think it would probably would be good to notice slightly earlier. But other than that, I don’t regret looking into it and I do think that it was the right thing to explore and it was also the right choice in the end to go where we went. So I think it’s all good.

Kimberly (57:46): Yeah. Well Kevin, thanks for sharing all of that with us. This has been an episode of Recordables, which is a production of 37signals. To hear more from our technical team, check out their blog at dev.37signals.com.

Upright runs health checks from multiple geographic locations and tells us when something breaks. It’s a Rails engine that you deploy to cheap VPS nodes around the world using Kamal. Each node runs probes against your services and reports back via Prometheus metrics, so you can alert with AlertManager and visualize with Grafana.

Site overview with world map

30-day uptime history

Probe status across all sites

Playwright probe video recording

Why we built it

We’ve used Pingdom for years and it’s generally worked fine, but has been somewhat of a black box. We couldn’t customize checks, couldn’t run authenticated browser flows without handing over more cash, and couldn’t control exactly where checks ran from. Occasionally, probes would fail and trigger a page, but the issue would resolve itself before we could investigate.

We wanted something we owned, something we could extend, and something that fit into the open source observability stack we already use.

Upright started as an internal tool to complement our existing monitoring. It now handles all of our synthetic monitoring across five global sites.

What it does

Upright supports four types of probes:

Playwright probes are the headline feature. These are real browser checks that can log into your app, navigate through flows, and verify that things work end-to-end. When a check fails, Upright captures a video recording and logs so you can see exactly what went wrong. You write them as Ruby classes using the Playwright API.

HTTP probes are simple health checks — hit a URL, check the status code. They run every 30 seconds from every site. Logs are in a familiar cURL format.

SMTP probes verify EHLO responsiveness, STARTTLS support, and SSL certificate validity.

Traceroute probes map the network path to your services with hop-by-hop latency tracking, thanks to MTR reports.

All probes run from every site you deploy to, optionally staggered by a few seconds. Upright uses this multi-site data to distinguish between real outages and regional blips. If one site in Amsterdam can’t reach your app but four others can, that’s a regional issue, not a full outage.

How it’s built

Upright is a Rails engine. You create a new Rails app, add the gem, run the install generator, and you have a working monitoring system. It uses:

• SQLite for probe results storage
• Solid Queue for background and recurring jobs
• Kamal for multi-server deployment
• Prometheus for metrics and uptime queries
• AlertManager for notifications
• OpenTelemetry for tracing and logging
• Dark and light mode for the dashboard UI

The architecture is simple: deploy Upright to a few cheap VPS nodes (DigitalOcean, Hetzner, etc.), point DNS subdomains at each one, and configure your probes. Our setup runs across five sites — four DigitalOcean droplets (2 vCPU, 4GB RAM, ~$24/mo each) and one Hetzner server (3 vCPU, 4GB RAM) — for about $110/mo total. You could run a minimal two-site setup purely on Hetzner for under $20/mo.

Each node identifies itself via environment variables and runs probes independently. Metrics are scraped locally with an OpenTelemetry collector, and then fed back to the primary site which runs Prometheus and Alertmanager.

We actually send the metrics to three Prometheus instances, one in each of our datacenters as well as the primary Upright instance, providing full redundancy.

Getting started

rails new my-upright --database=sqlite3 --skip-test
cd my-upright
bundle add upright
bin/rails generate upright:install
bin/rails db:migrate
bin/dev

The install generator creates everything you need: Prometheus and AlertManager configs, OpenTelemetry collector config, probe directories, Kamal deploy templates, and a Docker Compose file for local development services.

Define an HTTP probe in probes/http_probes.yml:

- name: "37signals.com"
  url: "https://37signals.com/"

- name: "Help Ticket Creation"
  url: "https://help.37signals.com/billing/tickets/new"
  expected_status: 301

Or write a Playwright probe that logs into your app and checks a real user flow:

class Probes::Playwright::Bc4ProjectProbe < Upright::Probes::Playwright::Base
  authenticate_with_form :bc4

  def probe_name = "Basecamp 4: view project"

  def check
    page.goto("https://3.basecamp.com/5228151")

    page.get_by_text("37s Monitoring HQ").visible?
  end
end

Check out the README for the full setup guide, including multi-site Kamal deployment, alerting rules, and authentication configuration.

Try it out

Upright is available now on RubyGems and GitHub under the MIT license.

During the conversation, Mike does a live demo showing what it takes to convert an existing Rails app to multi-tenanted and the safeguards built in to prevent accidental data leaks. You’ll also hear about the edge cases of globally replicated SQLite, and why we ended up launching Fizzy without it.

Watch the full video episode on YouTube.

Timestamps

• 00:00:00 — Introduction
• 00:02:56 — The Fizzy bet: one SQLite database per customer
• 00:06:57 — The challenge with SQLite and global writes
• 00:14:35 — Switching to MySQL (and fixing the fallout)
• 00:18:55 — Why the Apartment gem wasn’t enough
• 00:22:55 — Live Demo: Making Writebook multi-tenanted in minutes
• 00:31:36 — Built-in safety checks to prevent data leaks
• 00:35:17 — Replication, failover & “emergent behavior”
• 00:43:28 — What’s Next: upstreaming to Rails and future plans

Links & Resources

• Mike Dalessio’s Rails World 2025 talk: Multi-Tenant Rails – Everybody Gets a Database
• Active Record Tenanted
• Fizzy code base
• Apartment
• Writebook Multi-tenant code branch

Transcript

Episode Highlights (00:00:00): That’s the kind of thing that multi-tenancy can give you as well. It’s not necessarily between customers, but it’s just keeping your data separate and having that database connection handled really gracefully and very seamlessly by Rails. Rails does not make your life easy when you want to work in this way, when you have multiple tenants and all sitting in one database. I wanted to make it as hard as possible for a developer to accidentally shoot themselves in the foot.

Kimberly (00:28): Welcome back to another episode of Recordables. This is where we’re sitting down with the 37signals technical team to dive into some of the products that we make, including Basecamp, HEY, Fizzy and open source products. I’m Kimberly, your host, along with my co-host from the mobile team, Fernando. Hello, Fernando.

Fernando (00:46): Hello, hello.

Kimberly (00:47): And this week we are talking a little bit about multi-tenant Rails. We’re diving deep into a Rails topic, and to do that, we have one of our programmers, Mike Dalessio, with us to do that. Mike, welcome to Recordables. Thanks for being here.

Mike (00:59): Really good to be here. Thank you for inviting me.

Kimberly (01:01): Before we dive into this Rails specific topic, Mike, tell us a little bit about you, what you do here at 37signals. Oh, and how long you’ve been here.

Mike (01:08): Sure. I’ve been with 37signals for about a year and a half, about a year and a half, and I work on the SIP team that stands for Security Infrastructure and Performance. I do a little bit less of the product work and a little bit more of the backend infrastructure and keeping things running around here work, which I love. It’s what I did before I came to 37signals at Shopify where I was on the team that worked on Ruby Performance there as well.

Kimberly (01:35): Amazing. Also another huge Ruby on Rails shop, so that’s very cool. Okay, so I know you’ve talked about this multi-tenant Rails a bit at Rails Worlds last year, but we’re going to have time to dive into it deep. I know you’re going to do some screen shares with us, so why don’t you just tell us a little bit about this topic and obviously why you’re so passionate about it.

Mike (01:55): Oh, sure. So the concept of multi-tenancy really just means keeping your customer’s data separate and private from each other because ideally you don’t want one customer to be able to access another customer’s data, but when it’s all together co-mingled in one big database, that is a risk. A developer could introduce a bug that lets that data go. And so multi-tenancy is a little bit about just keeping the boundaries between customer data present. And so Rails doesn’t do a great job of making this easy. It forces you to think about it as an application level concern. In all of my models, I have to keep in mind as a developer that, oh, this model actually contains data for all of my customers and I need to remember to select data that only belongs to account ID 1234. And if I forget to do that, it’ll work in tests.

(02:56): And as soon as I push into production, people will see each other’s. So when we were building Fizzy here at 37signals, we wanted it to be multi-tenant and we wanted one application to work for all the customers similar to what Basecamp does. And David cooked up this hair brained scheme of what if we used SQLite, which is a file-based database. So instead of it being a server in the cloud that you connect to, it’s a file that sits on your local machine and that’s where the database is. And so you don’t have to go over the network so it’s faster. It’s also a little more self-contained where you can just delete the database or move it around or copy it or whatever you need to do. And he really wanted to push the boundaries of what we could do in Rails with SQLite. And so his idea was what if we made Fizzy use the multi-tenant with SQLite where there’s a separate database file on disk for each of our customers in Fizzy, if you think about it, that could be tens of thousands, hundreds of thousands of customers, we’re going to have hundreds of thousands of SQLite files sitting on disk. Wouldn’t that be really neat if we could pull this off and kind of pointed me at that problem and said go.

Kimberly (04:11): Okay, wait, can I ask a quick question?

Mike (04:12): Sure.

Kimberly (04:13): This commingled database, I would assume that’s common, like what most people are doing. Yeah?

Mike (04:19): Yes. That is the dominant mode of deploying web applications today in the world, and it works fine, but it also means that when you scale up, you have to scale your database machine way up. If you look at what we have to do to run Basecamp and HEY, we have large machines running the database. That database is replicated across multiple machines for redundancy and it takes a lot of operational money and effort to keep that running.

Kimberly (04:52): And we don’t currently have any of our products that are not commingled databases where each user has their own database. This would’ve been the first iteration of that. Is that a true statement?

Mike (05:03): Correct, yes. That’s what we were hoping for was to really kind of change the paradigm about how you deploy SaaS apps. Can you do it without having to have a big humongo database machine that everything reads and writes its data to? And we almost got there. I guess maybe the punchline to this story is we didn’t launch Fizzy with SQLite. Unfortunately, it was a really complicated problem and we ran out of time. It doesn’t mean it’s not solvable and it doesn’t mean we won’t solve it, it just means it wasn’t solved when we had to go live with Fizzy.

Fernando (05:36): So I sort of asked this question last time with Jeremy when we were talking about the S3 migration, and I was like, isn’t it just copying files? Well, with multi-tenancy, isn’t this just sort of like you have the file there, what’s the complicated aspect of it?

Mike (05:53): Yeah, that’s a really good question. I know there’s a plan to have Kevin on to talk about SQLite replication, and so he can probably talk about this a little more deeply than I can, but I’ll try. If you only have one machine, you only have one computer running your website, that machine takes all the connections, all the requests that come in, it can go to its local files for all of its data and return the response and life is good, and this is fine. If that was what we wanted to launch with Fizzy, we totally could have flipped the switch and we would be live on it today. But the reality is what you want to do to have a world-class product is you want to have it geographically distributed. You want to have a data center in Europe, you want to have a data center in the PacRim, you want to have a data center on the East coast and on the west coast, and you want customers to be able to go to the closest machine when they do a read. And you want them to be able to all write to the correct machine.

(06:57): So if you think about SQLite, if I had four machines, one in each of those regions, if I do a write, how does the network know where to send my request to? Because that database can only be written to on one of those machines. That’s the thing about SQLite is there’s only one master copy that can be written to at any point in time. So if I’m in Europe and I try to do a write, how does the network know that it needs to go to the US and do the write there because that’s where my database is? And that’s where the complications come in because we can replicate the database so it’s readable everywhere. But then when I do the request, all of a sudden this integration between the top and the bottom of your stack, the network stack at the top, which is what carries the request, and then you have Rails and everything down to the database and the database level needs to be coupled to the network level in a way that’s very unnatural. And so that’s what Kevin and I had been trying to build, mostly Kevin, is an integrated stack where the database routing and the request routing all worked the same way. I’m not doing it justice there, but Kevin can talk about it in way more detail than I can.

Fernando (08:11): No, no, no, that makes sense.

Mike (08:12): And that was actually the bit that caused us to say this isn’t quite ready yet. And we launched Fizzy with MySQL instead of with SQLite.

Fernando (08:23): That makes perfect sense. When you have a stack of things, you usually don’t want to jump between more. One, you don’t want to jump between more than one level, so you want to go straight down or straight up. You don’t want go like, oh, the database needs the network and then that’s where things get fizzy. No, I’m kidding. My question is, if I’m understanding this correctly, there is a single source of truth. Let’s talk about this for database example. There’s a single source of truth. So is that by design due to how SQ Lite? Oh my god, I’ve only read that. I’ve never heard it spoken.

Mike (09:03): I think most people say SQLite. I think the maintainers say S Q L Lite.

Fernando (09:09): That’s how I would say it in Spanish. I’ll say SQLite. Is that a SQLite design decision or is that like a 37signals decision when you’re trying to explore?

Mike (09:20): That’s how the database works. So SQLite is intended to be, it’s just a file that’s on disk, which makes it great for things like mobile applications, which you can talk about right now, right? SQLite is the most widely deployed database in the universe because it’s on every Android and iOS device in the world. And it’s great because on a phone you only have one writer, which is the phone. Using SQLite for a web app is a little bit different. It’s not necessarily what SQLite was designed for, but again, so because it’s a flat file, you can have multiple processes on the local machine open up that file and write to it or read to it. So on a single machine I might have a dozen web server processes running, I can all read and write to the file and it’s fine, but as soon as I go to another machine that doesn’t have access to that hard drive, that machine can no longer write to the file. So that’s really the boundary we’re talking about, about the file system boundary.

Fernando (10:18): And if you say that these are database level design decisions, I assume that there’s some sort of conflict resolution or something that can be done at the file level so that oh, okay, there’s synchronization between two SQLite databases?

Mike (10:39): This is a great question. So if I have multiple processes that are reading and writing to the database, they can all read at the same time, but only one can write at a time. And for this, I think the default is to use POSIX file locking. So it’s just using the file system lock system call to prevent multiple processes from writing to it at the same time. So this also means that SQLite has an interesting bottleneck that RDBMs like MySQL or Postgres don’t have, which is you can only have one writer at a time, but on balance, because the writes are so fast, right? They’re like, I don’t know, nanoseconds, right? It’s you’re writing to the local disk and not having to go over the network, which is maybe milliseconds Most of the time it’s not a big deal that you can only have one writer at a time.

Fernando (11:29): That is really interesting.

Mike (11:31): So we’ve drifted a little bit, so I want to just tie what we’re talking about back to the multi-tenant conversation real quick. So we wanted to use SQLite and we wanted it to be multi-tenant, but those two things don’t necessarily need to be coupled together. So the Active Record Tenanted gem, which we open sourced a few months ago, we have had a couple of contributors who’ve started working on MySQL and Postgres support for this gem. So the same kind of thing applies where I might have for, I’m going to imagine a scenario here, regulatory reasons I need to have a separate database for Europeans than I do for North Americans, GDPR, what have you, contractual regions, I don’t know. But if you have that problem, Rails does not make your life very easy and a lot of people end up deploying two versions of their app and one for Europe, one for North America, and you don’t necessarily need to. If you think about tenant, not necessarily as a customer-specific thing, but maybe it’s a region-specific thing. You could have all of your US customers route their MySQL database connection to your US East to Amazon region, and you can have all of your European customers routed to a Hetzner box that you have sitting in Germany. That’s the kind of thing that multi-tenancy can give you as well. It’s not necessarily between customers, but it’s just keeping your data separate and having that database connection handled really gracefully and very seamlessly by Rails where today it’s very difficult to do that.

Fernando (13:12): Exactly. Yeah. I was going to eventually get to that point. When you explained it like, oh, the database layer is down here and the network layer up there, and you were trying to make it work, I’m sure both you and Kevin plenty of times where were like, we’re trying to make a square peg fin into, right? So my follow-up question was like, well, this sounds like sort of an application concern. And then you mentioned the decoupling, like, oh, okay, we were trying to make SQLite work with multi-tenancy, but they’re not necessarily tightly coupled. You could just split them apart and then use MySQL or Postgres or whatever. That is really interesting. My follow up question then is what is actually implemented in Fizzy? Yeah, what did we go live with? Yeah, did we just cut it all up? Open source the gem and Fizzy works like regular and there’s some open source work that’s happening?

Mike (14:16): Yeah, so right now, Fizzy doesn’t rely on the multi-tenant thing at all. The version, that version that we open source, but Fizzy itself is an open source project. And so you can actually go back through history and you can see the point in time that we had the multi-tenant libraries all working and we ripped it out and replaced it with MySQL.

Fernando (14:35): What is that like on a technical level?

Mike (14:37): Yeah, sure. Okay. Do you remember I mentioned that Rails does not make your life easy if you want to do multi-tenancy. So there was a long tail of bugs that were introduced by switching from SQLite to MySQL or from multi-tenant SQLite to using normal Rails and MySQL. Because if you think about if you are in multi-tenant mode, you have your database, your Rails processes connected to the database, and so you can do something like, I want to find the user named Fernando, so select star from users where name equals Fernando and it’ll bring you back. But now if we are in this MySQL database where all of the customer data is co-mingled, right? I can no longer say select star where name is Fernando, I need to say, and where account ID equals 1234. And so there were a bunch of places where we were relying on this implicit assumption that we’re connected to a database where all of the data belongs to the same customer.

(15:44): And now we switched over to MySQL and now all of a sudden we had this assumption sprinkled throughout our code where we weren’t specifying the account ID the way that we should. And so we had to go in and find that. So you can actually go through the Fizzy source code history and see us fixing these bugs one by one as we went through it. I think it was all one big bang, one big pull request, but there were a lot of very small changes we had to make in order to do that. And that’s the sign I’m talking about here, which is that Rails does not make your life easy when you want to work in this way when you have multiple tenants and all sitting in one database.

Fernando (16:19): I’m not a Rails expert, but the example that you gave sounds very low level when it comes to SQL instructions. Like Rails wouldn’t… do something like, no, no, no. My question is specifically what part makes it difficult from Rails to actually build multi-tenant?

Mike (16:42): It, it’s just the usability. There’s nothing technical preventing. So there’s a really good mature, well-tested gem called Apartment because tenants and Apartments and everything.

Fernando (16:55): That’s a great name.

Mike (16:56): So a word play and Apartment actually does make this easy. Where Apartment will you have to decorate your Rails models with like, oh, I’m tenanted by account id, and you have to put that in all of your models.

(17:10): But once you do that, then Apartment will do some of the heavy lifting and say that you have to have an account ID specified or else it will raise an exception to make it a little safer. And the drawback, I talked about this a little bit in my Rails World talk, Apartment was built a long time ago. I think it’s like 12 years old and it’s still, it’s showing signs of its age and it’s not totally thread safe in all circumstances. So it’s not taking advantage of a lot of modern Rails. That connection handling got completely rewritten a few years ago. So Apartment is still, it’s a little inefficient. So Apartment I think does this thing where it closes the database connection and it opens a new one, and then if you want to switch back, it has to close this one and open the new one as opposed to just keeping a pool of connections open and maybe doing something where you like, oh, there’s a max on a number of connections you can have open at a time. It will reap them if they’re unused. And I built all of that into the Active Record Tenanted gem. It’s all there. So Apartment does handle the usability issues that you’re talking about though. Why is it hard? It’s simply like this implicit where account ID equals 1234 has to be in all of your database queries and Apartment will do a little bit of the heavy lifting there. It does a good job,.

Kimberly (18:32): But it didn’t do what we needed it to do for Fizzy.

Mike (18:34): It did not at all. No, no. It was a little slow. It wasn’t thread safe, like I said. And also, oh, this is the other thing. It’s Apartment only deals with the Active Record bit of multi-tenancy and it doesn’t make the rest of Rails work in multi-tenant mode.

Fernando (18:55): What is the rest of Rails?

Mike (18:57): Let me give you an example. So Rails is a really big framework. So I wrote a list, okay, if you think about, there’s a lot of things that Rails does that need to be aware that we’re working in this multi-tenant space. So one is fragment caching. If we generate a view in Rails. Rails is a really great job of caching that view so that we don’t have to regenerate it the next time you ask for it. But the cache is based on the record ID. And so you need to make sure that if you have two records with the same ID that belong to different accounts, so account A, record one and account B, record one by default will both try to write to the same cache record. So this is another way that you can accidentally get data from one customer being shown to a different customer because you’re not hitting the database, but you’re hitting the view fragment cache.

Fernando (19:58): Oh.

Mike (19:59): If that makes sense.

Fernando (20:01): Yeah.

Mike (20:02): So you’ve got to be careful about that. You want to maybe include your tenant ID when you are uploading blobs, attachments, pictures, whatever. If you have a customer who deletes their account, you want to be able to easily delete all of those attachments or maybe they want an export. There’s no good way in Rails by default to say, oh, these are all the files that belong to customer A because they’re all dumped into one big directory or one big S3 bucket with no differentiation. So having the tenant ID be in there would be really great so that you can say, oh, this is the first tenant account ID slash the blob ID is just a good way to keep all of files organized on disk. That’s something else that the Active Record Tenanted gem does.

Fernando (20:53): It sounds like a lot of work.

Mike (20:55): It was a long tail. It was a long tail and it took about six months to get it all working well and Fizzy.

Fernando (21:00): Wow.

Mike (21:01): But as a result though, it’s super easy to make an existing application multi-tenant. And I can walk you through that if you want. I can share my screen, actually walk you through what that looks like.

Kimberly (21:14): Yeah, let’s do it. Yeah. Mike, it kind of sounds like some of this solution is to eliminate the human factor. As you’re describing this, you’re like there’s all these ways that you can mess this up, and it sounds like it’s trying to prevent some of that human error potential.

Mike (21:33): Yes. I think that’s a big part of it is making sure that it is… well, I mean you can turn that on its head a little bit and be like, oh, well, I’m trying to make it usable because people will invariably, if you design something that’s not very usable, then invariably people are going to hold it the wrong way. If you don’t make the hammer have a nice long handle, they’re going to hold it by the head and try to pound something with it. So usability and also safety or for me, at least the same dimension, right? It’s kind of the same topic.

Kimberly (22:05): Okay, what are we looking at here?

Mike (22:06): So here’s what I’m going to show you. We have an application called Writebook that is open source and that you can get through the ONCE program and you can run it yourself, and it’s just for handling documentation. So we got a bunch of books here that are our internal documentation at 37signals. For example, if I go into the programmer’s handbook, then I can get a nice list of topics and each of these is page in our documentation. It’s just Writebook. This is a SQLite-backed application. The idea is you can run Writebook anywhere you want Digital Ocean or on your local machine. It’ll be writing to a local file that’s just a SQLite database. And so I thought this would be a really great way to show how easy it is to make something multi-tenant. Why don’t I convert Writebook into a multi-tenant application.

(22:55): So instead of having one global, everybody sees the same documents, it’ll actually be multiple instances. You can actually have multiple instances of Writebook. So it’ll be like fu.writebook.com or bar.writebook.com for individual customers. So this is what I did and it turned out to be super, super easy. So I’m going to show you the diff. And I’m not lying. This is the entire set of changes that I had to make to run in code to make Writebook multi-tenant, that is right to multiple SQLite databases. So I’ll walk through the changes real quick. You have to say that your main database connection is tenanted. It’s just… if I wanted to connect to a specific database, I could add some arguments here, but by default I’m just saying, hey, listen, this is the class that I want tenanted and all of our models inherit from this class, so they are all also tenanted.

Fernando (23:55): Can I ask tenanted based on what?

Mike (23:58): Oh, so the tenant, the tenant is just a string. It’s just a name.

Fernando (24:05): Oh my god, of course.

Mike (24:06): And the name is used by default. The name is used, I’ll tie it together in a minute, the name is used by default for the subdomain that you’re going to go to. So fu.writebook.com, the tenant is going to be fu.

Fernando (24:19): Yeah.

Mike (24:20): And then the database name on disk is also going to be fu.

Fernando (24:24): Of course.

Mike (24:24): So that’s how, again, the network layer and the database layer get tied together because we’ve named them the same thing here. And so at the database level, this is our database config. What we’ve had to do is change the name of the database, the path where it used to be storage/db/development. It’s now storage/… oh, I’ve got this little percent tenant in here. So, if you’re familiar with printf() this is just a format specifier that at runtime is going to be replaced with the name of the tenant. And I have to do that for all of our environments, so development, test and production is the same change. And then I have to make sure that when I actually hit in development, I hit a host that it has a wild card where I can hit any host I want. So lemme show you how this actually works.

(25:20): It might be a little bit easier. So I’m going to start off from scratch. I’ve deleted all the databases. I’m going to start up the server and it’s going to say, if I try to go to writebook.localhost, it’s going to give me an error and it’s going to say, oh, well, I can’t connect to a tenanted database while you’re untenanted. So it’s telling me that I’m untenanted because I haven’t provided that subdomain. So if I go and I do mike.writebook, it’s not going to say, ah, tenant not found Mike. Very cool.

(25:58): So if I go to the console, the Rails console, and I say application record, and that was the class that had tenanted added to it. It’s our base class. Create tenant, fubar. So what it just did was it just created the database and it applied all of the schema migrations. And now that database is live on disk. If I go and I look in find storage, here’s now a database called fubar db.development.sqlite. If I go here instead of Mike I now go to fubar, it’s going to kick me into the first run and prompt me to enter everything. So I could be like, oh, my name is Mike, my fake password, right? What it’s going to do now is it’s booting up the database, loading up the initial, oh, it didn’t do it. Hang on, because my password wasn’t long enough. There we go.

(27:04): Now it’s going to work. Ah. So now I’ve got the Writebook manual, which is by default this gets added. And if I look at what’s in storage now, you’ll see this is not great. Tree…. right? You can see that now I’ve also got all my active storage files are being stored in a directory called fubar as well, like automatically, because I mentioned I think that all of the blob keys get the tenant slash added to the front of them, which turns into on disk, I’ve now got a directory that has all of the fubar tenants images from this document.

Fernando (27:44): This is super cool.

Mike (27:46): Thank you. That’s great to hear.

Fernando (27:47): Is this a new default for, should this be the new default for Rails? It feels like a straight up improvement.

Mike (27:53): I would love for this to land in Rails. I feel like it’s got to actually be running somewhere in production first.

Fernando (28:00): No, no, no, of course, of course, of course. But the concept…

Mike (28:02): My plan, my dream is this would get upstream into Rails because it’s additional, if you don’t want it…

Fernando (28:08): Who cares?

Mike (28:09): You can continue to use Rails the way you always did. But if you want to work in this way, then yeah, turn on a bunch of these configs and you’ll get it for free. Yeah, I would love to see this in Rails at some point.

Fernando (28:20): Now, one question I have is, in this hypothetical scenario, if you were to have multi-tenant as a default, what would change in the configuration so that you didn’t need to have subdomains tied to tenants? I’m sure it’s possible, right?

Mike (28:45): Yes. So the gem by default ships with what I will call a, it’s actually it’s Rack Middleware, which maybe isn’t a helpful phrase, but the web server framework is called Rack, and that handles the request. And what we can do, and the way it works is the request gets handed to each stage before it finally arrives at the app. And you can insert a stage in there, and that’s what the gem does or inserts the stage that looks at the subdomain. It says, ah, there’s a subdomain in here. I will now look on disc to see if that is a valid tenant. And if it is, I will connect to it. And if it is not, I will raise an error. So by default, that looks at subdomain, but you can override that and you can use whatever logic you want it’s provided. I will see if I can get the gem…

Fernando (29:39): You could just consider… sorry.

Mike (29:39): No, go ahead.

Fernando (29:42): Oh, I was going to say, you could basically consider the base, like the default domain as a tenant, right?

Mike (29:53): Oh, you could. Yes, you could.

Fernando (29:55): And then that way you could get both the subdomains, each as individually named tenants and the top level domain as a tenant itself.

Mike (30:02): Yes, you could.

Fernando (30:03): Wow.

Mike (30:04): You can do whatever you want.

Fernando (30:05): That is super cool.

Mike (30:06): It’s literally the way this case is configured is you actually just pass in a lambda, an anonymous function…

Fernando (30:11): Right, right.

Mike (30:13): That takes the request and hands back the tenant name. So by default it’s just going to use the subdomain. And so if you look in Fizzy, in the source code, what we’re actually doing there is we look at the path and the first section of the path is a big number, and that’s the account ID. And we do that just like Basecamp does. So it’s that you can take it out of the path, you can take it out of the domain, you can take it out of the host name if you want to. Some of these are operationally easier than others. Having a separate domain for every customer means you have to worry about regenerating SSL keys, and how is your web server going to do that? And can you dynamic, can you automatically regenerate those SSL keys every 90 days? So some are harder or easier, but you have the flexibility to do whatever you want.

Kimberly (31:03): Mike, tell me this, I know in your Rails World talk, you mentioned something about safety checks. Is there something that you can show us on how that exactly works or what you’re checking for?

Fernando (31:13): Safety clearly.

Mike (31:15): Yes. Yes, absolutely. So one of the things that I think is, we talked about usability a little bit earlier and usability and safety kind of being different points on the same spectrum. I wanted to make it as hard as possible for a developer to accidentally shoot themselves in the foot. And so if I….

Kimberly (31:36): Eliminating that human error of possibilities again.

Mike (31:39): Yes, exactly. So if I do user.first, this is on the Rails console, so I’m running Ruby code, I get an exception saying, oh, well there’s there, there’s a default tenant here, which is like development tenant when I’m in development mode, and that tenant doesn’t exist. So if I do our application record with tenants and then I can pass a block and whatever I do in this block is going to be in the context of this tenant. So I can just say fubar, which is the tenant we just created, I can do user.first and it’ll return me a user from that database. And I know it’s from that database because I can see in here it has tenant fubar as an attribute on that model, which is great. Tells me that that object belongs to that tenant. So I can pull this out.

(32:33): I could say user equals blah. Now what happens if I try to update that user or write to that user while I’m in the context of a different tenant? It should raise an error for that too. So I have to create a second tenant, which I will just call second. And again, it migrates the whole database. So now what if I am in the context of the second database and I tried to do user.update, name is… I’m going to try to change the name on that user. It’s going to give me a safety exception saying with the user model belongs to tenant fubar, but you’re currently connected to tenant second to prevent you from doing this at runtime. So if there’s any kind of a bug that’s introduced where you might be saving one record to a different tenant’s database, you can’t cross the streams anymore because it does the safety checks where it compares the objects tenant string to the connection string, if that makes sense.

Kimberly (33:47): So those safety checks are already built into the gem that you created.

Mike (33:51): That’s correct. They’re built into the gem to make it as hard as possible for people to cross the streams and mix commingle customer data.

Kimberly (33:59): You’re making this sound very straightforward and very simple, but we also didn’t put this in Fizzy. I’m curious if we just needed more time because we’re working against a deadline or there was something else that made it not appropriate for what we were doing. Why did we not? How come it didn’t happen?

Mike (34:17): Yeah, the main reason is that we ran out of time. We had everything working and there were some edge cases. If you think about going live with a global product, you want it to have automatic failover where if a machine goes down, you want everything to immediately cut over to the backup. And with the SQLite database, that becomes really interesting to do because you need to make sure that you’ve got the database file being replicated in real time to a second machine where it’s available in read only mode. And then as soon as this machine goes down, you need to have something outside cut requests over to the second machine and it go into writeable mode. So this all of a sudden becomes the primary, where it used to be a secondary or a backup machine, and that was where things, there’s, there’s a long list of edge cases there.

(35:17): There’s a phrase people use, which is emergent behavior. Complex systems have interesting emergent behavior, that is behavior that you only see in when certain weird edge cases happen or when things happen in a certain order. And what we were going through was about once a week finding a new edge case in this complicated system we had built where there’s SQLite and there’s tenanting in Rails, and that was pretty solid. That was working well, but then we needed to replicate it globally and we needed to have network routing route requests to the right place. And then we also needed failover to work properly. And making all of that work in time for the release date just didn’t happen. We just didn’t have enough time to work out all of the bugs.

Fernando (36:05): I’d be remiss if I didn’t ask this, but this work was like last year, right? Middle of last year more or less?

Mike (36:13): It was. So the multi-tenant gem was worked on mostly the first half of 2025, and then the replication and failover work was like the middle of 2025, like the summer and fall of 2025.

Fernando (36:32): So given that this is 2026, the obvious question I have to ask is do you think the amount of time would’ve been reduced had you been using AI?

Mike (36:47): Oh the AI question. I don’t, I don’t know. I don’t know. If we could get an AI, an agent that was able to do things like deploy to our staging environment, actively monitor all of the systems like simulate, it might’ve been helpful to have an agent help us with a lot of those testing scenarios.

(37:16): But we actually, we did use AI for quite a bit of the replication stuff to if you go look at it. So I’m hopeful that Kevin will, open source Beamer. Beamer was the code name of the replication stack that he wrote. And a lot of that is Go code, which LLMs are great at writing Go code because it’s very simple language, syntactically anyway, it’s simple. So we did have a lot of AI help on some of this. The multi-tenant gem was I think a little bit tougher task for agents because it was something that was novel and it was trying to shoehorn something new into the existing framework. And so there were a lot of design decisions that had to be done. So the version of the gem that is open source now I think is version four if you believe that. I think it’s version four, where the first version was just spike threw it away, but then I wrote two more versions and I was unhappy with the API and threw them away before I landed on an API that I was really happy with.

(38:28): That felt Railsy and that felt like it wasn’t getting in the way. And also that was extensible in ways that would lead to creative problem solving. Let me give you an example of that. So I always think that you can tell that something is designed well, if people can use it for purposes that it was not originally designed for. And I came across one of these late in the Fizzy development work where we realized that we were already tenanting by customer. So every customer has their own SQLite database, but we wanted to replicate by region. So we wanted to have, for example, all of the North American customers in our Chicago data center and all the European customers and our Amsterdam data center. And then that meant that when we were running background jobs, we would have to have one database for North America and one database for Amsterdam.

(39:34): Because the job workers are local, they have to write to the local SQLite disks, and they have to keep their state in their own database too. So it’s like you’d have a job worker in the US that connects to any of the US customer databases, but then had its own database as well to keep state in. And they had to have the same thing in Amsterdam. And if we had multiple data centers, we’d do that in multiple places. And so what I ended up doing was we had the customer databases tended by account ID, and then we had the Solid Queue databases tenanted by region. So within the same application, we had two different dimensions of tenanting going on. And the Apartment gem would not be able to handle this because the Apartment gem acts as a, it’s a global, it’s a singleton Apartment, and you’d be like apartment.createtenant.

(40:31): And you’ll notice that when I was just showing you in the Rails console that I was actually using the application record. Application record was annotated with tenanted and I do applicationrecord.createtenant. And so all of the tenanting methods live on your application models. They don’t live on some other class that the gem is bringing. They live on your application models. And because I did that, I mostly did because it just felt a little bit more natural to me. But because I did that than late in the Fizzy development process, I was like, aha, I can use tenanting for Solid Queue as well. And then I would do Solid Queue record base.create tenant, and I’d pass in the region. And that was how you create the Solid Cache database because that API just ended up working much better for that use case. Anyway, where was I going with this? I was going to the point where it took me three generations of API design before I landed on one that I thought was Railsy and flexible enough to do some interesting things. And I don’t think an AI could have necessarily helped me get there better, get there better or faster.

Fernando (41:45): I agree. Sorry, I’m trying to wrap my mind around this. Why are you tenanting the Solid Queue by region?

Mike (41:54): The current version of Fizzy does not do this, but when we were still using SQLite, again, the idea was that our Solid Queue database is still SQLite and needs to live on a machine and a process needs to know that if it’s a running in the US, it should connect to the US database. And if it’s running in Amsterdam, it should connect to the Amsterdam database, and we needed to replicate that also. So we were also replicating the Solid Queue databases so that if the US went down, it wouldn’t be a complete outage. We would reroute everything over to Europe and North American customers would just get routed to Europe until we were able to get the US back up and running. So for failover and replication purposes, we’re still replicating those databases. It’s just that then in that case in Europe, we would have two Solid Queue clusters running. One would connect to the European database and one would connect to the US database. They’re both still local though. So that’s why we were using, we could have done something more complicated, but the fact that tenanting was right there fits and it just automatically multiplexed to the databases based on whatever region they were running for. You look at the customer database and you say that customer’s in the US region, so I should connect to the US Solid Queue database. It just ended up working really well.

Fernando (43:20): That’s awesome. That’s really interesting.

Mike (43:21): Well, if it had gone live in production, it would’ve been even more interesting, but maybe someday we’ll get there.

Fernando (43:25): Do you see a path forward?

Mike (43:28): I do. So I showed you, I just made Writebook with, honestly, that was five minutes of work. I was able to make it multi-tenant. Now there’s a little bit more you need to do around like, oh, I need to document how this works for people and I need to give them a script so they can create a new account when they want to, and they may need to create a wild card SSL cert if they want to run this themselves on their own machine. So there’s a long tail of little things, but the bulk of the code change was very quick, five minutes. So one possible path forward that I’ve been talking to David about is can we take all of our ONCE products that are all running on SQLite and make them all multi-tenant, demonstrate that the multi-tenant gem, Active Record Tenanted does work, get some miles on it. I feel like I really want to run it in production somewhere so that then it’s a more compelling argument about upstreaming it to Rails. I would love to try to get a future product that we’re working on on this as well. But again, I feel like the Active Record Multi-tenant gem is pretty solid. It had a year almost where we were running it internally on Fizzy. We were using Fizzy most of last year, and it was solid, it was work, it worked well.

(44:50): We didn’t really have any problems with it for the last six months or so. The complication was around replication and failover, and really we need to solve that. And so I want to keep working with Kevin to hopefully finish up Beamer and really get that working. Because there’s another missing concern in Rails that I would love to fix at some point, which is failover. There’s no concept in Rails of what host am I running on and should I fail over? Solid Queue has started to scratch at this with how it manages its connection handling. And I actually built, I have a fork of Solid, Solid Queue that knows whether it is in active or passive mode based on something that’s in the database. You can have a flag in the database that says, you have your primary in North America and your backup is in Europe.

(45:41): You have the Solid Queue job cluster running in both places, but nobody’s doing anything in Europe because the US is the primary. So this is all code that we already have in HEY, that it basically lifted, I jammed into Solid Queue. It totally worked in Solid Queue. And I was like, all right, we’re using this in one app, we’re using it in a library. Why isn’t this primitive in Rails? So why shouldn’t there be a class in Rails that tells you, are you in fail… are you in primary mode or are you in passive backup mode? And if you have that, then a lot of the Beamer replication code becomes much simpler as well. So there’s a whole path here, but I really want Rails to become a little bit more capable around failover and replication than it is today.

Kimberly (46:28): Mike, question for you, for anyone who’s listening and is like, okay, I totally get this and I can use this gem and this doesn’t seem hard, do you have any pieces of advice or things you should make sure to look out for that you can share?

Mike (46:42): Yeah, that’s a really good question. So the one thing I think that I would love some more feedback on is how the gem approaches Action Cable. So brief summary of what Action Cable is, is it’s the ability to do, to push data to a web client. Under the hood it’s using web sockets and maintaining an open connection. And so the most common use for this these days in Rails is Turbo, Hotwired and Turbo, where you can essentially broadcast data. So we use this everywhere in our apps to push notifications, to push chats, to push card updates if someone changes the status on it. So you’re pushing data to the web client. That connection also has to be tenanted because you don’t want to push data for one customer to a different customer. And a lot of that is reusing the same middleware I referenced earlier where request comes in. How do you find out what tenant that request is for? I’ve only solved one problem with that, which was Fizzy’s problem. And so I would ask people if they’re going to kick the tires on the gem to make sure that Action cable is wired up correctly for your use case. And let me know if it’s not.

Fernando (48:01): So Mike, I’m on the mobile side. So when it comes to rich text, where’s that stored? How do you handle that with the multi-tenant?

Mike (48:11): Thanks for asking. So Rails has a relatively complicated system for how it stores its own data. And by that I mean there’s metadata around Active Storage uploads. There’s some, the actual action, the rich text content is stored in a separate record. So Rails needs its own tables, basically. So if you’re going to use any of the rich text stuff, you need Action Text and Active Storage. And that means that those need to be in separate tables. And so the gem actually jumps through some hoops to make sure that the Rails models are using your tenanted database too. So the idea is if in my tenanted database I have my card class, and the card has some rich text and the rich text has some attachments. You want to be able to do a join across all of those tables. So they all need to be in the same database.

(49:08): And so there’s actually a concept in the gem of being a subtenant of your database. So your application models are the official tenants, they are the tenants, but then you can have these Rails records that normally would be, I don’t know, in some other, in your primary database. If your primary database is tenanted, then all of a sudden these Rails records are subtenants and we can’t do it through class inheritance. In your Rails application, all of your classes inherit from application records. So you make the change there and everything else gets it because the subclasses, but the Rails records are in a completely separate hierarchy of classes. And so we actually, we have to actually inject some behavior into those classes. But it all just works at the end of the day where then you can do that join across all of those classes. And it also means that those Rails data records also don’t get commingled. And we have that multi-tenant boundary between all that data.

Kimberly (50:11): Mike, so is there a place where people can find this gem? And if so, we’ll add it to our show notes.

Mike (50:16): Yes, there is a GitHub repository for the gem. It is open source. It’s been open source for a few months now. And also you can go to the Fizzy source code if you want, go back through history and actually see how it was being used.

Kimberly (50:26): That’s perfect. Well, thank you for joining us. This has been an episode of Recordables a production by 37signals. To hear more from our technical team, like Mike and Fernando, check out our developer’s blog. That is at blog. No, that’s at dev.37signals.com. I like don’t even know what the address is.

The conversation gets into the human side of a project like this, including verification, anxiety, and the moment you finally hit delete. You’ll also hear how long it actually takes to move that much data and the tools we used to make it happen seamlessly.

Watch the full video episode on YouTube.

Timestamps

• 00:00:00 — Introduction
• 00:02:05 — Why S3 was the last (and scariest) piece
• 00:08:34 — The volume of data to move
• 00:11:11 — Bandwidth limits and AWS constraints
• 00:13:12 — The custom-built Rails tool for copying and reconciliation
• 00:21:25 — The logistics of hard drives, write speeds, and network connections
• 00:28:05 — The intentional order of moving data
• 00:49:55 — Anxiety, verification, and the fear with deleting data you can’t get back
• 00:54:13 — Was there any downtime?
• 00:58:56 — Essential tools that made the migration possible
• 01:07:03 — What happens next

Links & Resources

• Rclone
• DuckDB
• S3 Fast List

Transcript

Episode Highlights (00:00:00): Well yeah, there’s no real difficulty. It’s just copying some stuff, right? Among those five petabytes of data was spread across hundreds of buckets and on the order of about five billion objects. The concept is straightforward, but you got to be careful and do it right. And there are easy ways to do it right, but there are a lot of easy ways to go wrong too.

Kimberly (00:00:22): This is for Recordables, a place where the team at 37signals shares their behind the scenes work, building Basecamp, HEY and open source projects. We’re diving deep into what we’ve done and how we’ve done it so you can learn a thing or two and learn from some of our mistakes. I’m your host, Kimberly. I’m joined to help with the technical side of discussion by Fernando from our engineering team. Hello, Fernando.

Fernando (00:00:45): Hello, hello.

Kimberly (00:00:46): Well, we are talking this week a little bit more about our move out of the cloud. If he’s been following 37signals at all, you know that we started a move out of the cloud back in 2022. We have wrapped up the big portion of moving data out of S3. To join us, we have Jeremy Daer, principal programmer here at 37signals who did much of that final work to talk about it. Jeremy, thanks for being here. Before we dive into our topic today, tell us a little bit about you and how long you’ve been at 37signals.

Jeremy (00:01:17): Yeah, I’m a programmer here. I’ve been around for some ages and this storage migration is the most recent of probably half a dozen over 15, 20 years. So coming to it feeling like it’s time. We need to move all the bits across the continent again.

Kimberly (00:01:40): Okay. Well, I’m sure there’s a lot to dive into. Fernando, you’re going to help me with all the technical aspects of it, but let’s just start kind of where we ended, if you will. We know we’ve started this move back in 2022. S3, moving that piece of data was the last part of the project as I understand it. Kind of tell me why that was the last piece and then we’ll dive into the technical side.

Jeremy (00:02:05): Yeah, there are two things. It’s expensive and it’s scary. We’ve got a lot of things to move and we had built a lot of trust in S3. It’s super durable, super reliable. You can’t go wrong with S3. It’s like the old school IBM thing. Nobody’s going to get fired for using IBM. Whereas storing your own stuff, it’s all on you. And so if you haven’t been shouldering that risk for a long time, you kind of forget what it feels like. And so a bunch of planning, a bunch of mitigation work, risk assessment can help prepare for that. But kind of getting over that hump and just choosing to do it, we’re going to do it solves that problem. The other is cost. And what are we doing here? It’s easy to open up our wallet and pay AWS. And it causes us to cry a little bit or a lotta bit.

(00:03:05): But when we want to replace it, we’re looking at spending hundreds of thousands, millions of dollars on something. And we’ve got to be pretty sure it’s going to work. Well, we’ve got to be certain it’s going to work, but also not spin our wheels for weeks, months, trying to figure out and prove it for sure. So we need to come up with some kind of framework for understanding what we need and being able to prove it before spending a bunch of money. Third thing really, S3 also has this tricky bit which motivated some of the cloud exit of you got to pay. You got to pay for everything. And one of the things you got to pay for is a bandwidth coming out of S3. So if you want to move your data, you’ve got a little bit of, again, a lot bit of a handcuff situation.

(00:03:49): You got to pay to get your data out. So the EU came up with some kind of regulations around this and all the major cloud providers got out ahead of the regulations saying that you got to be able to exit and you can’t kind of keep people’s data for ransom. And it was like this before, people didn’t necessarily want to leave S3 because the alternatives were not great. And you’re on the upswing of cloud adoption of look at all the things we’re not doing. You just get to send it out to this abstracted thing and pay some monthly fee rather than doing it all yourself and paying hundreds of thousands upfront for storage hardware that you’ve got to run yourself and maintain yourself. So anyway, AWS came up with this waiver program wherein you could get your data out of S3 under certain conditions. And the conditions run like this.

(00:04:51): You’ve got 60 days, 90 days, whatever. You’ve got a ticking clock and you got to get all your data out. You can’t half ass it. You got to get it all out. And if by the end of that time you can prove that it’s all out, then you’ll get AWS credits for the bandwidth cost. So you got to estimate how much stuff you’re storing and then how much bandwidth it should take to get out. If you do it all perfectly, which of course everybody would do, then that would be your credit. So the public messaging is like, “We’re chill with this. We’ll let you get all your data out.” The reality is you’ve got this kind of tight binding contract and you got to do it perfectly and then we’ll give you a refund.

Kimberly (00:05:42): Oh, wow. Do we pick the timeframe?

Jeremy (00:05:45): We do not pick the timeframe.

Kimberly (00:05:46): Do we pick the 30, 60 or 90 days?

Jeremy (00:05:49): It’s negotiable. So depending on … Well, the internal messaging toward customers is a limited timeframe, like 60 days or something. The external messaging is, well, we’ll work with you. We’ll be reasonable. So the reality is kind of in between. We do have great account reps, so I can’t fault that end of things. It’s been wonderful interacting with AWS, but you see both sides of things of we’re going to look to the world like we’re ready for you to leave, but in fact, we’re going to make you jump through a bunch of very tight hoops.

Fernando (00:06:26): Well, I was going to joke like, “Oh, it doesn’t sound that difficult. You just, I don’t know, copy a bunch of files over to your hardest drive somewhere.” What is the real difficulty there?

Jeremy (00:06:38): Well, yeah, there’s no real difficulty. It’s just copying some stuff, right? It’s just that if you want to copy things, well, there’s a lot of things you got to do, especially if you’ve got a lot of files stored. For most people, this copy is not technically hard. The basic process would be clone your S3 bucket. A bucket of objects is like a folder full of files. You list the bucket to see what’s in there. You copy it over to your new place and typically be doing this with a live system. So you need to have some kind of awareness of your system would need to know that you’re moving to a new system and there’s a live old system. So if you want to do it without downtime, you need to be able to store files to both places. You need to mirror your files to both storage systems.

(00:07:23): And then you need to do a copy. And after that first copy, maybe you’ve got some stuff that’s in one storage system, but not the other and vice versa. So you need to bring them into sync. You need to reconcile them.

Fernando (00:07:34): And none of this is offered by AWS. It’s all on you.

Jeremy (00:07:37): AWS does offer something that does something like this, but you got to pay.

Kimberly (00:07:44): I feel like that’s a theme.

Jeremy (00:07:46): Yeah. So if there’s a data transfer service that can do something like this, and in fact, it can do a lot more. It can do incremental syncing between disparate systems, but it is very expensive. You’re looking at tens of thousands of dollars to move large scale buckets of stuff.

Fernando (00:08:06): Was it ever an option for us?

Jeremy (00:08:07): We did evaluate it. It would be very nice not to have to do this job ourselves and to have somebody else do it. And there are vendors that do stuff like this, but again, you’re looking on the order of like tens to hundreds of thousands of dollars, depending on the size of your buckets. And most of them will scale with your storage. So they’re looking to take a percentage cut because the more you’ve got, the more you can pay probably.

Kimberly (00:08:34): Jeremy, when you’re saying the size of our buckets, how much data are we talking about that was being moved?

Jeremy (00:08:41): So in aggregate, we had about 10 petabytes of data across a bunch of buckets. Some of our applications were responsible for a lot more than others. Our average object size was about 1.1 megabytes. Of all that data, we’re also geographically distributed. So deduplicated, we had probably about five petabytes of unique objects. And among those five petabytes of data was spread across hundreds of buckets and on the order of about five billion objects. It’s a lot of stuff. You get into the realm where you can’t … If this is a folder on your computer and you tried to open the folder, your computer would crash. And you can’t list a bucket of that size without taking literally days.

(00:09:36): So there are a bunch of interesting constraints that come into play when you try to do this conceptual process of you copy, then you stop your application or whatever, storing things for a little while. Then you do a catch-up copy to reconcile, to make sure your destination has all the stuff that was in the source. And then you cut over, you start using the new system and you got to be sure that you actually got everything, that everything was copied correctly and there were no mistakes and you didn’t miss anything and nothing showed up while you were copying, et cetera, et cetera. So even at the small scale, the concept is straightforward, but you got to be careful and do it right. And there are easy ways to do it right, but there are a lot of easy ways to go wrong too. And once you get to a larger scale, there are a lot of easy ways to go wrong and a lot more easy ways to go wrong and a lot of ways to go slow.

Kimberly (00:10:32): Well, I feel like we should talk about all of those things. Not only the things that are easy to do well, but what are all the things people should avoid if they’re trying to do this?

Jeremy (00:10:43): Well, what would be wonderful is if egress were free and we weren’t stuck in the situation of needing to move quickly because when we look at the kind of bigger picture of what the job to be done is we got to move five petabytes of stuff within … We negotiated 90 days and that was based on our back of the napkin math of what size of network connection we have available and how quickly we can move stuff.

Fernando (00:11:11): Is there a limit on the AWS side on the connection?

Jeremy (00:11:15): There are plenty of limits, yes. And there are some hidden limits and some stated limits. So that comes into play when we try to figure out what our limiting factors are.

Fernando (00:11:25): Wait, sorry.

Jeremy (00:11:25): So if we do a bounds analysis of…

Fernando (00:11:27): Yeah, let me backtrack just a little bit. You said like, okay, we negotiated 90 days. What was that like? Oh, we have five petabytes if we have access to, I don’t know, 10 gigabytes per second and we do this twice.

Jeremy (00:11:40): I was chatting with our rep saying, “Here’s how fast we can possibly do it. “ So that’s what we need. And they said, “Okay.”

Fernando (00:11:48): Okay.

Jeremy (00:11:50): We built in plenty of buffer because part of this process is not just the copy. And something that anybody doing a process like this will quickly discover is that copying is, it seems like the central point and the purpose of the job, but it’s really reconciliation and verification, making sure that you did what you thought you did. And the doing what you thought you did is as costly as the copying. And it has different kinds of limitations because you need to go look and see what you’ve got. And that means listing all your files, listing all the objects in a bucket. And when this process can take days to perform, you need faster ways to do it and you need cheap ways to do it because again, you got to pay. If you want to list a bucket, you got to pay for API requests.

(00:12:43): And so you’ve got to be careful that you don’t get in a situation where you’re repeating work or doing unexpectedly expensive work. And in many cases, these like API requests are cheap. So it turns out we’re totally fine, but you got to do the due diligence first, so you don’t end up discovering that in order to save whatever tens of thousands of dollars on a paid service, we don’t end up spending tens of thousands of dollars on errant API requests ourselves.

Fernando (00:13:12): Yeah, that makes sense. How do you even … I’m trying to wrap my mind around this, where … this process must be automated somehow, right? Is it in the app? Where does this reconciliation process happen?

Jeremy (00:13:25): So that gets in a little bit deeper into the approach that we arrived at. So there’s more than one way to do reconciliation and something I’d recommend to people who are moving just a single application, which ideally it’s a situation you’d be in. So a little digression. Looking back, the ideal way to set up AWS accounts in the first place is around these kind of scopes of kind of responsibility radius, blast radius of if you ever wanted to leave, you got to take your whole account out. So don’t go putting a bunch of buckets from different places, don’t share an account. Like make accounts early and often so that accounts are aligned with your application. So that gives you the flexibility to move one application at a time. We were in that situation to some extent with some of our newer stuff, but our older applications, we had a shared account, so we needed to do everything all at once.

(00:14:30): And when you have multiple storage systems in a single account and you need to move them all, you can’t tailor the solution for the system that’s being moved. Instead, you need something that’s going to work for all of them. So that led us to, it’s got to be something that’s at a lower level and is mostly transparent to the applications we were running. So if I were doing this with a modern rails app, this is something I’d build into Active Storage at the application level. I’d build in some kind of modeling for where things are stored. So you get metadata within the application itself, knowing that I’ve got this object stored in AWS and I’ve also have it stored in my, whatever my new destination storage is. And so the process would be that I’ve got some rule that says I need to copy storage and they need to be in each of these locations.

(00:15:23): And so the application could do that with just a bunch of active jobs, copying things on its own, could do it lazily in the background and just let it trickle through, if you didn’t have these other constraints of you got to get it all done, you got to blast it at maximum speed and you’ve got to make it work with, and you got to make it work with older apps. So not all of our stuff is on latest Rails and we extracted active storage from our own apps. And so most of our older apps are using older abstractions that were kind of the source of the extraction. So we needed to be compatible with all these possible systems. So we needed something lower level.

Fernando (00:16:04): So we built…?

Jeremy (00:16:05): So we built something. We didn’t want to build something, but we also didn’t want to pay a lot of money. So kind of rock and hard place thing. And we are in the position of having expertise. We are a technical organization. We have programming and operations teams. We do all this stuff. So it’s natural for us to do our jobs. Maybe this isn’t the thing that we’d outsource. If there was a natural fit, we would. And in fact, we did to the greatest extent we could by picking purpose-built tools to do each of the jobs. And in fact, we went through multiple iterations. As we got closer and closer to the kickoff of we want to begin the copy, we reinvented the system that we built to do copying several times as we discovered kind of quicker, simpler, straighter paths to do that job. So we started out with something that would be dead simple, of distributing jobs over a bucket itself, would store objects and each object would be a file manifest.

(00:17:21): And then that file manifest would be available to a bunch of workers. We’d spin up as many workers as we needed to saturate whatever bottleneck we had. So the bottleneck could be our network connection, how fast we could pull things from S3. It could be our write rate to our destination storage. How quickly can we write objects and are we bandwith limited? Is there a metadata limit on the IO operations per second, or is it going to be the S3 read rate? So we can discover all these things.

Fernando (00:17:54): Read rate. Why does it have a read rate? They’re just throttling you.

Jeremy (00:17:59): Yeah, because S3 doesn’t technically have a cap on read rate, but they do have a cap on what they call bucket partition read rate. And so a normal bucket will only have a single partition and that’s kind of … Well, the way S3 is laid out, it looks like it’s kind of hierarchical folders, but it’s not. It’s actually key names that are separated typically with slashes. So it looks like a file path. And the natural way to partition a bucket would be, let’s take all the first file paths up to the first slash and we’re going to turn that into an internal S3 partition, which is like their own way of kind of sharding the bucket so that they can scale out. So that if you want to do things like write a ton of things to S3, it can be fanned out to multiple partitions.

(00:18:49): And as you write more, it’s smart about noticing that write rates are hot on certain partitions and it’ll split them automatically for you all behind the scenes. So you can’t even tell what your partitions are unless you give some kind of cues or use like a traditional slash character, which can help S3 figure out, but also you don’t need to. It’ll just do it for you, but it’ll do it based on your usage. So if you have a moderate usage app, the first time you’re going to have high usage is when you try to copy things out of it. So you’re going to hit the rate limits pretty quickly, especially if you have a lot of small objects. If you’ve got a lot of big objects, you’re going to be bandwidth constrained, probably on your own network connection. If you’ve got a lot of small objects, you can fill that pipe with tons of connections and you’re going to hit the rate limit quickly.

(00:19:46): The rate limit is, for the record, 5,500 GET requests per second. So that’s fetching…

Fernando (00:19:56): And we hit that.

Jeremy (00:19:57): Oh yeah. Yeah.

Fernando (00:19:58): Oh my God. That’s insane.

Jeremy (00:20:01): So out of the box tools can hit that pretty quickly, especially if you have small objects. We got a little bit lucky because when we did our migration into S3, we worked with S3 to pre-partition our buckets knowing that we’d be writing at high rates. So on the way in, we contacted S3 ahead of time saying, “Hey, here’s about how much stuff we’ve got. Here’s what the key layout looks like for the objects.” And we generally use random hash keys, so it’s like completely uniform. So on S3’s end, the job is easy because when you have a uniform key distribution, they can just say, “Let’s take whatever the first two or three characters of the key in, we’ll use that as partition keys.” So our big buckets were already partitioned, so who knows what the limit could be because when we did a load test, we didn’t hit a rate limit on those larger buckets. On smaller ones, we quickly hit a rate limit in our load testing.

Fernando (00:21:06): I know this is mostly about S3, but I’m also curious, did you really hit a write limit? Because David has spoken at length about this, about the hardware, about the amount of money that we spent on brand new hardware that’s blazing fast. Even that couldn’t keep up. We just went full throttle.

Jeremy (00:21:25): We certainly hit its limits. So it became a question of where are the limits and which one is … So we’ve got a system with a bunch of components and each component has a kind of maximum, has a cap, and which one is going to be the weakest link, which one’s going to slow us down. And it turns out that we ended up pretty similarly bound by our network connection. We got a hundred gigabit network connection dedicated to just this copying process that thanks to our data center pros at Summit, they set this up just for this job and we set up a separate VLAN for the machines that would be doing the work. So it was essentially their dedicated little network universe of, you can saturate this pipe. Turns out to not have been completely true. We were actually sharing it for a couple other things, which we discovered because as we tuned our system to eke out the maximum possible performance, we actually overshot a little bit and started interfering with other traffic, but that was what we thought would end up being our bottleneck.

(00:22:33): And to our wonder, it was not our destination storage. We had initially considered using MinIO and using hard drives and our read and write rate for normal application usage can easily be satisfied by spinning disk hard drives. We didn’t exactly relish the idea of maintaining a bunch of spinning hard drives because the failure rate can be notoriously poor depending on which batch you get and whether you have kind of a hot rack in your data center. And it’s just, we were not looking forward to it. And Eron, our head of ops team, had a line on a new storage system from Pure Storage Flashblade. They’ve got this fancy, super duper proprietary flashy, bashy setup where they’ve got, rather than using off the shelf flash stuff, they mounted their own flash on their own boards and did something a lot cheaper, kind of bringing the kind of flash of two years in the future back a couple of years, which made it kind of cost competitive with hard drives.

(00:23:46): So that was a surprise and ended up being a huge blessing, not because we need the performance for steady state usage, but because we needed the performance for the copy. So if we had been on hard drives, we definitely would have been limited by our write rate into that storage cluster.

Fernando (00:24:06): By physics, just how fast you can spin those.

Jeremy (00:24:09): Yeah. Yeah. And so, I mean, it would depend on the number of drives. And I mean, actually we probably … Yeah, thinking back, I think we probably would be able to satisfy 100 gigabit traffic on hard drives because we’d have so many of them. In any case, it was a blessing, not quite in disguise, but we’re all happy to take the other path. I’m digressing a little bit, but if you’re choosing storage systems, if you’re choosing storage systems, it can seem like make the choice based on what you can afford now, but it’s also what you’re paying over the course of five years or 10 years. And our total cost of ownership analysis was based on five years, seven year, 10 year. What it would be like to keep the system around for a long time. And the power savings alone from flash are significant. It’s a lot cheaper when your power is expensive in a costly data center that uses up less rack space.

Fernando (00:25:13): I hadn’t even thought of that. When you go from an actual … I’m just picturing a bunch of machines in a place, but if you’re going to clone five petabytes, it has to be a lot, right? A lot of m achines, a lot of power, a lot of network, a lot of everything.

Jeremy (00:25:29): Yeah. Yeah. I mean, it’s a crazy time in the storage world. There’s a bunch of new form factors for solid stage drives coming out right now. So it gave me a little bit of FOMO because I see these things coming out just as we purchase this giant system and we still made the right decision for the time, but in about a year or two, there’s going to be kind of a new generation of solid state drives coming out at 256 terabyte size per module. So you can fit … Shoot, what was it? 40 petabytes in two units of rack space.

Fernando (00:26:07): Wow.

Jeremy (00:26:07): So you could fit all of our storage into just a tiny little bit of a rack. And in the hard drive era, we’d be looking at like two full racks just for that storage. And so the shrinkage and power savings are dramatic and it’s all happening now.

(00:26:23): And a lot of this is driven by AI stuff of… people need a ton of data stored and super high bandwidth to it. And so new vendors are cropping up daily trying to do this kind of job. So they’re also driving the flash hardware side. So hopefully this will become just kind of a commodity storage problem. And you’ll be able to go to Super Micro or Dell or whatever and order up some servers that are packed full of these drives and you won’t need a special setup. You’ll just go to Newegg and…

Fernando (00:26:57): And buy one.

Jeremy (00:26:59): Yeah.

Fernando (00:26:59): Yeah.

Jeremy (00:27:00): The limiting factor by far is network bandwidth. So in almost any copy, that’s going to be the cap you’re going to hit. So having a good data center partner is essential there. We were able to get 100 gigabit connections set up within just a span of days. And we had plenty of lead time, et cetera, but still it’s just wonderful to be able to bring in a big pipe like that. And of all things, we have AWS direct connects already, but not allowed. You cannot use your special direct, fast connection to AWS to do the data egress. You’ve got to use the public internet for-

Kimberly (00:27:45): Even if you pay money, pay more?

Fernando (00:27:47): Oh, wow.

Jeremy (00:27:48): Yeah. So you pay money, but to get this other bandwidth covered, you got to use the egress cost. And I guess maybe it’s just tied up in some kind of red tape in the direct … Would they cover the cost of the direct connect? Maybe they can’t account for it. Who knows?

Kimberly (00:28:05): Jeremy, I do have a question because obviously we were moving multiple applications or working with multiple applications to make this move. Was there a specific order that you were moving them or was it just you just kind of picked an application and did that? What was the reasoning behind the order for the move?

Jeremy (00:28:24): There were two phases. So we chose some smaller applications with less storage, but that would be representative of our applications with larger storage needs that we would migrate before the egress window opened. So essentially we wanted to test the process. We wanted to be prepared that when the window opened, we’d be able to blast. Didn’t turn out to be that way. We had some delays as we optimized and restarted things and whatever, but it was crucial to do that, identify some systems that we could do real life copies and not just kind of test runs. So we chose a couple of representative systems and did those first to prove kind of a blueprint for how we do it for our other applications. And each application ended up being not quite cookie cutter, but once you’ve kind of improvised a recipe a few times, you kind of know what you need to do.

(00:29:26): So you come into it with a plan of attack of you need to do dual rights to multiple places, you need to do a reconciliation step, you need to have a well-defined cutover process. And so you come up with a checklist, validate the checklist by doing a live migration and iterate, fine tune, and then you’re ready to go. Then when it comes to the actual copy, we go criticality first. The things that we want to be absolutely sure and the business critical stuff, Basecamp, HEY, all our primary revenue generating apps with the big data. We want to get those started as soon as possible because there’s the most a copy and we want the greatest assurance that we did it all properly. So give us the most headroom for unknown unknowns. We’ve got our known unknowns of things that might crop up, but we also have room for just we don’t know. Who knows what would happen?

(00:30:27): And we came up with a bunch of those. So we’re grateful for having started them early. But yeah, definitely dive into the trickiest, biggest thing first.

Fernando (00:30:42): That makes sense. I’m still trying to wrap my mind around this. You go to a small app and you’re like, okay, you know what? We need to migrate this. You start building the program that you mentioned, the reconciliation program. Is that program a Rails server on its own? Is it like the modifications that you mentioned to Active Storage?

Jeremy (00:31:08): No. So we built a new thing that kind of scales up the basic idea I started with of I need to list things to know what’s in the source. I need to copy everything in that list over to the destination. And at a small scale, I can use a single program. We used one called Rclone. There’s one called RSync. And most folks, technical folks have used one of these before. And you just fire it up, you give it a source and a destination and it churns and it does the job. At most scales, that’ll work fine. And in fact, for AWS, their free bandwidth egress limit is pretty generous, so most people would fit within it. And you could just Rsync and do it yourself or Rclone and call it done. And it does all the bookkeeping for you. At our scale, we needed to fan out to a bunch of workers doing this job.

(00:32:00): So we needed to do it in parallel. That means splitting up, batching it up. And so it becomes kind of a classic like map reduce problem. You’ve got a big input, you need to spread out to a bunch of jobs, and then they’ve got all their individual outputs. In this case, it’s take a big list of files, split it up into batches, send those to workers that are going to do the copies. Have those workers have some kind of supervision that’s tracking what they’re doing, their progress, whether there’s an error, retries, all that kind of stuff. That’s where we used a Rails app for command and control, for wrapping up the jobs and the work. We arrived at a Rails app after trying what I thought might be some kind of simpler lo-fi ways of doing things. And of all things, I kind of backed my way into doing, to using Rails because I was missing some of the conveniences of home.

(00:32:59): One of them was secrets management, credentials. Here we’re doing something that’s copying between a bunch of AWS accounts to a bunch of destination buckets. You’ve got a ton of sensitive credentials in one application. And so as I was building this kind of script-based simple system, I realized I’m rebuilding a credentials manager. This is not the life I want to be leading right now. And how about I do something that’s already built for me? And so going essentially to Vanilla Rails because of this mildly auxiliary concern. I had a bunch of other kind of pressures swirling in my mind that were resolved by this, so who knows whether my unconscious was also kind of like, uhhhh… I can kind of feel that change is going to need to be made and here’s the thing that just triggered it. And the trigger then led to a bunch of nice outcomes like being able to use ActiveJob and Solid Queue. And a lot of things we’re familiar with for basically how do you distribute this work?

Fernando (00:34:03): So in the end, you have a single Rails app that you’re constantly looking doing this window, you’re constantly monitoring like, okay, did it complete everything or are there any retries errors? And then you go and fix them like you say, right? Oh, we hit this limit. Let’s try and work around it.

Jeremy (00:34:21): So that’s part of the approach based on the initial migrations as our kind of test runs. As we discovered in the test runs, you’re going to have new errors, kind of surprises crop up. And so we prioritized failing fast and not trying to be resilient and automated way. So not building in things like exponential back off too early because sometimes things were not things we wanted to retry. There were actual errors. So being able to identify something that was truly like a transient failure and then automate it late rather than early. So we treated it kind of like an Andon cord of like, here we’ve got a production line, we notice something’s failing, we pull it, we stop everything, we fix it, and then we proceed from there. So that drove a lot of other decisions of when you break things up into chunks, they need to be observable, they need to be retryable, they need to be supervised.

(00:35:19): And especially for a diagnostics and troubleshooting, you need to be able to see what’s going on. So in these cases, you’ve got a active job process that invokes another tool and you’ve got hundreds of these running. And so you’ve got this standard output and standard error from a bunch of tools. You’ve got the exit status. How do you see them? And so this, for me, this was kind of a crucial stumbling block of like, if I can’t see exactly what’s going on, I don’t know what’s going on. And so I don’t want to spend a bunch of time guessing at and troubleshooting. I want to just look at the output and I want to be able to figure it out as if I were running it on my own console. So that was a critical step early on too, of making something that was easy to supervise and just witness.

(00:36:12): If something’s failing, I could try it myself, I could invoke it myself, or I could pull up a transcript of what that process had done. So this was the job of the Rails app of coordinate, pulling an inventory or a catalog from the source, split that catalog or inventory up into a bunch of pieces, which was its own whole thing. If you’ve got something that’s huge and you need … Anyway, you can get into tooling in a little bit here, and then make a bunch of jobs for all that stuff. And each job, well, it has its responsibility is its chunk of files and the output for that job is a bunch of things like status and transcripts and whatnot. And those themselves are actually stored in a storage bucket as well. So every job, there’s a unique idea associated with it and you can go inspect the whole process.

(00:37:11): And in fact, we did have a live tail. So since these things can take a while, you can visually see what’s going on. You could just kind of snoop on any transfer.

Fernando (00:37:25): So you were basically Neo for 90 days?

Jeremy (00:37:28): Not 90 days, thank goodness. We ended up getting it down thanks to Pure Storage and the very fat pipe of bandwidth to less than 10 days of transfer.

Kimberly (00:37:42): What?

Fernando (00:37:42): Wow.

Jeremy (00:37:44): Which turned out to be pretty critical because that gave us lots of time to do reconciliation, verification. Yeah, we didn’t know that. And so we had built in a lot more buffer than we needed, but I’m sure I’m glad we had the buffer because I was also going to … I was headed on vacation on sabbatical right after this was going to wrap up, which was not the wisest of career choices, but …

Kimberly (00:38:11): Jeremy, tell us a little bit about those 10 days. Is it 10 days just nonstop? Are you breaking … Kind of walk us through…

Jeremy (00:38:18): Oh, it’s nonstop. Yes.

Kimberly (00:38:19): Yeah. Okay. Yeah.

Jeremy (00:38:20): So the setup is I didn’t want to trickle things into a pipe and have to be carefully tending things. I wanted to feed a pipe and have a backlog. And so I can go through some of the technology stuff just briefly. If you’re copying a bucket with billions of objects from S3, use S3 inventory reports. It’s something you can turn on S3 console. It’s easy to do. You got to pay. You got to pay.

Kimberly (00:38:49): It’s the theme. It’s the theme.

Jeremy (00:38:51): But it is the most efficient, effective way to get a large scale bucket listing without doing the work yourself. It is delayed. The most frequent you can do it is daily, and so they drop on a schedule. And so for this kind of process where you want to do a big bulk transfer, daily’s fine, and particularly if the system that you’re migrating is doing dual writes, you’re writing to both the old source and your new destination, you know that you’re already in sync, so there’s not going to be missed writes. So the thing you do here is you turn on dual writes to both places, and then you take the inventory from the day before you turned on writes. So you know all new objects are being written to both places. So the old inventory is sufficient for knowing that you’re going to get a bulk copy of, and it’s going to bring you into accord.

(00:39:50): Everything’s going to be the same. So you start from that snapshot of the bucket. So we used S3 inventory reports. We didn’t have that turned on everywhere. We developed … We looked for another tool that could do something like this. And there are a bunch that take a similar approach. If you try to list an S3 bucket, it’ll take literally days because you need to sequentially list files and it lists whatever, 1,000 at a time or some 10,000 at a time, something like that. Anyway, order magnitude-wise, it’s ages, but there’s some tricky, very clever ways of doing this where you can do it in parallel by estimating what the prefixes of a bucket are. And you can ask S3 for bucket listing starting from a certain prefix. So if you know your key distribution, then rather than doing a single sequential listing, you can instead do thousands of parallel listings for every prefix you’ve got.

(00:40:50): So you can turn a multi-day bucket listing into something that takes like 30 minutes. So I had that in my back pocket in case we needed live listings. If we discovered that we were going to be in a situation where we needed to do a downtime or we weren’t able to do dual writes in a system, so you’d need a downtime to be able to stop writes from the old system, but also not write to the new system so you wouldn’t get out of sync. And also to list the objects in the destination buckets because Pure Storage and the Flash Blade product does not have an equivalent to inventory reports. So you’ve got to do the listing yourself. So on that other side, you want to take the inventory report from S3 and then you want to develop your own report of the destination and compare them and any discrepancies need to be accounted for.

(00:41:45): So you need a fast way to do that, so there’s a tool called S3 Fast List that’s on our GitHub. We forked it from AWS samples and we adapted it so that it would support non-S3 storage systems so that we can use it to list flashblade buckets. Works great. Very clever approach. Really pleased to find that. We didn’t end up needing it that much, but it was a wonderful diversion and felt like kind of an insurance policy of a special built tool in the toolbox. The next thing was, how do we split these things up? And we started with, gosh, how are we going to take this S3 inventory report comes in either CSV format or Parquet format and parquet being like flooring, it’s like kind of split up and actually I don’t know how far the metaphor goes, but it’s a very efficient format for doing columnner data storage.

(00:42:51): So it’s great for analytical processing and whatnot where you know which columns you want to work with and you need to do some transformations on them. And it’s particularly nice for something like this because there’s tooling that can ingest it, that can stream it from a remote destination and operate on it and then emit it again. So took us a while to discover this because I was looking initially for just something like polars. There are different tools that can ingest parquet and operate on it, split it up, whatever, like using a windowing function to … We wanted to do something like split this not just into number of objects, but in total batch size, so that we would evenly distribute batches across machines so that we wouldn’t end up with uneven bandwidth demands. So you wouldn’t have one machine that is working on a batch of a bunch of small objects and it can’t fill the pipe.

(00:43:45): What you want is to have an even distribution size wise so that you’re maxing out the pipe on each of the worker machines. So to do that, you need a windowing function that goes through the inventory report and does a cumulative sum on the bite size of the objects. And each time it reaches 10 gigabytes, it says, “Oh, I’m going to do a split right there and I’m going to turn that into a chunk.” And this turned out to be hard, use a lot of memory and kind of work, but ended up maxing out the memory on a machine I was using. And it’s like, “Well, okay, this is probably, might be feasible, might not be feasible.” And then I discovered DuckDB, frigging awesome. DuckDB is amazing. I cannot sing its praises enough. It’s like somebody discovering SQLite for the first time, although DuckDB is like SQLite on whatever next generation steroids, because it can even do the stuff SQLite does maybe better because I’m just glowing with its capabilities, but it can do SQL, it can like connect to remote databases, it can work with a local database end process just like SQLight.

(00:45:03): It can work with CSV files and parquet files on the local file system. So most kind of big data, data science stuff, DuckDB can do locally on a single machine and super efficiently. It’s really smart about spreading out IOs to do things as smartly as possible to try to avoid doing things like bringing everything into memory. So I’ll sing this praises a little bit more.

(00:45:34): Not only can connect to everything like a Swiss Army knife of data analysis, it can also connect to remote URLs and to S3. So I had this whole system built of ingesting data from S3, downloading it, and then staging it on local file system, and then doing the splitting myself using my own tooling, and then storing those split files in another bucket as a staging area for jobs that would then be dispatched to work each of those chunks. Turns out I could skip all of that. With one DuckDB invocation, I can point it at a glob that’s referencing multiple S3 files. So the inventory report is split up into hundreds of files. It can reference all those files, stream them all in, partition them the way I like, and then write them to a remote S3 compatible file store, all streaming. So there’s no local file system, there’s no other code I need to write.

(00:46:36): It’s just you got to configure it properly, you got to know what you’re doing. But when you get it working, it’s like, “Oh God, yes, this is sweet.” You’re working with the remote thing, streaming it all through, not using a bunch of memory, and then writing it out to remote storage. And then I was able to take the files that had been written and I kind of wrapped Active Storage records around them. I said, I’m going to make Active Storage records that point to where those batch files had been stored. And then I distribute those active storage records out to the jobs to work.

Fernando (00:47:13): That’s a benefit that you were in the Rails app, right?

Jeremy (00:47:15): That is a benefit of the Rails app, yes. Yeah. And I had that abstraction to work with. And thankfully, Active Storage was able to accommodate this where I was kind of going behind its back because I was using DuckDB to write the files rather than using Active Storage. And I said, haha, Active Storage, I’ve got these files. Can you make use of them? And it’s like, “Yeah, of course I can do that. “ So you just feed it the key of where it’s stored and Active Storage will be happy to work with it as you bring it.

Fernando (00:47:39): That is so cool.

Jeremy (00:47:39): So any case, DuckDB, amazing, able to partition the problem and kind of eliminate a whole step of what would otherwise need to be custom code.

Fernando (00:47:51): It just comes on to show you that nothing is new. How did the DuckDB guys know like, “Oh, you know what? What process would be really nice if you take this, partition it and then put it this way?” That’s so, so cool. I love that.

Jeremy (00:48:07): Yeah. The thing in common, people’s resource constraints, everybody’s constrained in similar ways and they all have different problems, but they’re all stuck in similar ways. And here somebody comes along and solves it elegantly and does it with open source.

Fernando (00:48:18): Very cool. And this process, this had to happen once a day because of the limitations of the inventory. Am I right?

Jeremy (00:48:27): So I did end up automating it.

Fernando (00:48:31): Of course.

Jeremy (00:48:32): But it can happen at most once a day. So I made a scheduled scan for new inventory reports and I automatically process them so they’d be ready if I did choose to use them. And I only did this because it turned out that partitioning was so easy and cheap now. I had anticipated it being like a pretty slow process. And so I wouldn’t want to just be firing it off all the time. I’d want to choose which specific inventory report I used and it would take hours or who knows how long to do the partitioning, but now it’s a matter of minutes. So I’m like, well, I’m just going to do it, make it easy to choose which inventory report I want to use as a sync, as a copy source. So from the app dashboard, I could have a list for every app and every AWS account in every bucket, which things were transferred using which source manifest. I’ve got a bunch of inventory reports. For each one, I can see that I’ve kicked off a copy. I have state tracking for every part of the process of partitioning to copying to errors to reconciliation.

Fernando (00:49:44): Wow.

Kimberly (00:49:44): As a non-technical person here, I do have a question because all of this sounds very hard and scary. I’m curious, Jeremy, what was the most nerve-wracking part of this process?

Jeremy (00:49:55): Deletion. Yeah, the final deletion. I mean, you know, I mean, it’s like anything that’s high stakes, your brain’s got a lot of things going on. My brain’s got a lot of things going on. I’ve got feelings, sensations, I’ve got some kind of cognitive whatnot that’s blinking on and off sometimes. And some parts are telling me like, “I know that things are fine, but anxiety’s telling me,

(00:50:17): Maybe you should discover why it’s not. “ And those things need to work together. And I can use my anxiety as a guide that maybe I haven’t figured everything out, but I could then use my cognitive process of here’s the things I’ve worked out, I’ve ruled all these things out and I have some standards of proof. I can demonstrate conclusively in a way that is not dependent on my anxiety, that can be externally verified, that it worked and that I’m done. Nonetheless, deletion is still dicey, but when you do press delete, then it’s, I mean, oh… Somebody else take the wheel. It’s just all happening now. Yeah, right. There’s no going back. We’re now doing deletions. Yeah. Coming up with the biggest unlock feeling was adding kind of a belt and suspenders step, you know, keeping your pants up you want more than one way. When you’re verifying and reconciling, you want to be a little bit more than sure.

(00:51:23): And so thanks to Pure Flashblade’s extraordinary metadata read/write rates, we can do hundreds of thousands of metadata operations per second just without breaking a sweat. It made it easy to do rather than doing reconciliation against inventory reports, to do reconciliation by doing a live sync. And so with an inventory report, you’ve got lag time between the listing and kind of the live state of things. Whereas with a live synchronization, you can see exactly how many objects were needed to be copied and the size of them. And so you could essentially do repeat copies fairly cheaply until you can see that everything is done. And it really helped just psychologically and as a matter of certifiable proof that you get the final copy that says nothing needed to be copied. Everything was up to date and that typically came after, there’s another key step of when you’re doing dual writes to a previous source and the new destination, you turn off dual writes and you go to single write just to your new storage destination.

(00:52:49): And then at that point you know only your new one is new. And if you did need to roll back, you’re kind of like, ah, if I did need to roll back, now you’re out of sync. Now you need to copy the new stuff back to the old place. So that happens after you get your green, everything was cool, nothing new needed to be copied. Turn off the dual writes and a final sync and double verify. Nothing is changing. Nothing is accidentally writing. So I did a bunch of other, more than just belt and suspenders also, whatever else, contraptions you couldn’t imagine keeping your pants up. One of them was just changing the permissions on the S3 buckets. Some of these, we had old systems that had multiple things writing to them and you just, you kind of know, but do you really know? And one of the ways to be sure is to just turn off writes.

(00:53:42): So if something was writing to it, it would error. So that’s kind of the final straw of assurance. When I go to delete this or I go to turn things off, I don’t have some straggler that’s going to surprise me. And it was only writing some hours of the day or it was on a cron job or something, so I wouldn’t have caught it in the initial sync.

Fernando (00:54:04): My hands are sweating and I had nothing to do with this. The amount of … Okay, the million dollar question, was there any downtime?

Jeremy (00:54:13): There was no downtime. Everything worked and it was really quite wonderful. There were some things we broke, so I suppose-

Fernando (00:54:24): We don’t need to talk about that. No, I’m kidding.

Jeremy (00:54:25): So the 100 gigabit link was actually on a shared link with some other things that we’re using a portion of the bandwidth. And so when we pushed over about 80 gigabits, we started kind of impinging on some other stuff that needed to not be impinged on. And so we did cause some errors elsewhere, but the copy was fine.

Fernando (00:54:53): Suspenders worked, right?

Jeremy (00:54:54): Yes.

Kimberly (00:54:58): Jeremy, question for you. Now that this is done, like it’s tied up with a bow, looking back, are there things that you’re like, “Oh, I wish I’d done this differently?” You can say no.

Jeremy (00:55:08): No.

Kimberly (00:55:09): Okay.

Jeremy (00:55:09): No. Yeah, I’m pretty happy with how things worked out. I appreciated the incremental approach to starting simple and focusing on the epicenter of, what I thought was the epicenter of the problem because I didn’t build too much as I discovered that the true epicenter was in verification, reconciliation, inventory management. It’s about how do I track what’s going on and the copy itself was fairly simple. I did rabbit hole a couple of times. I built out a live view system that I didn’t actually end up using much, but in key times I did. So it’s a kind of like hindsight bias of like I didn’t need to do all that, but I kind of did to discover that I didn’t need it. It’s a little circular, but it allowed me to diagnose and troubleshoot things that were blockers that would’ve been really hard to work out otherwise because it’s things that turn into like little Heisenbugs of, if I run this myself outside of supervision, it works.

(00:56:11): But then in my supervision framework, something breaks. And it’s stuff like I’m opening a pipe and I’m feeding things to standard input and I got another pipe that’s reading and like a pipe could get wedged because somebody hasn’t read from it frequently enough for something like this and it can manifest as some other kind of error. Anyway, having that kind of visibility was really helpful, but looking back, I could probably delete it from the app we built.

Fernando (00:56:43): My question would be, I mean, you are an eminence within the Ruby on Rails community. Let’s assume I’m just an average Rails developer. What is the complexity of the project I can take on from moving out of history to our own hardware?

Jeremy (00:57:04): You could take on this whole project. And this is one of the … And the magic of this is that a lot of the feeling of criticality is the business criticality. It’s not technical difficulty. It’s a modeling problem. And there’s some tricky things with process supervision that Ruby doesn’t make super easy, but it’s not bad. And there’s plenty of other kind of worked examples you could start with. And otherwise, the degree to which Vanilla Rails just works is quite gratifying. And in particular, well, Kamal also, in fact, leading into this as part of figuring out our upper bounds was doing S3 load testing against our flashblade. So we got this new S3 service. What can it actually do? Well, there are load testing tools out here that can do that. And I used Kamal to deploy one of these tools out to a bunch of nodes and hammer it as much as I could. Worked great.

(00:58:06): And I used Kamal to deploy our copying application called Nostos and worked fantastically too. And I was able to use accessories, Kamal Accessories to stand up the database that did all the state tracking, do open telemetry observability, do logging. It was all just single system, like a single developer pushing to some VMs somewhere. So it’s all kind of bog standard stuff, but it’s being used in the employment of a kind of critical operation. But again, the criticality is all in our heads. The actual app is fairly simple.

Fernando (00:58:50): You mentioned DuckDB as being like, wow, an incredible tool in your arsenal.

Jeremy (00:58:56): I cannot sing as praises enough.

Fernando (00:58:59): Are there any other tools that were completely necessary in this process?

Jeremy (00:59:03): Yeah, I sung Rails praises a little bit. Vanilla Rails turns out to be the way to go. And I don’t mind tooting that horn a little bit. This other tool, S3 Fastlist, was like a little pleasant discovery, diamond in the rough. In our clone itself, what we did to do the heavy lifting copies was incredible. It did all the stuff we needed. It’s open source. It was easy to contribute to. In fact, as part of this, we added a flashblade, an official flashblade destination to our clone. So when you go do an R-clone of your own, and it’s got this nice kind of interactive thing where it asks about where you’re coming from, where you’re going to, and now Pure Storage Flashblade is one of the places you’re going to. And what that is, it’s essentially like a list of characteristics of the system so that Rclone knows how to best do its transfers.

(00:59:57): There are certain quirks and asynchronicities with different S3 compatible file stores and Pure Storage has some of them, and now it’s just all set up out of the box where you don’t need to go figure out the command line flags yourself.

Fernando (01:00:14): That change was upstreamed.

Jeremy (01:00:15): Yes, that’s part of Rclone now.

(01:00:19): The other key thing is that Rclone is bandwidth and metadata operation efficient and it’s oriented around resilience first. So you can operate it in a bunch of different ways. You can kind of back off on the kinds of checks it does, but it does nice things like pull the checksum from the source and check it against the destination. And it can even do a kind of extreme check where normally you would write the file to the destination, you get a checksum back and you compare that the checksum that it says was written is what you had. And you say, “Okay, cool. It looks good.” But you can also do it in kind of really be careful mode, which is write to the destination, get the checksum, then download it from the destination and actual checksum of the bits. So it’s got you covered for every degree of risk mitigation you want to have at play.

(01:01:16): So if you don’t trust your destination yet, because funny things can happen where like a bit gets flipped on a hard drive or a gamma ray hits something and messes something up where it says that you got the checksum you expect, but it turns out on disk, there was a modification. So depending on the level criticality of your data, how many infinite nines you’re going for after that decimal point, Rclone’’s got your back. I’m just going to sing his praises a little more. Sorry.

Fernando (01:01:42): Yes, go for it.

Jeremy (01:01:43): So there are certain things you can be very efficient with Rclone where you can skip operations that don’t matter. And if I don’t want to do a bunch of metadata operations against S3, like checking last modified time, whatever, things that are not present in the S3 file listing or the inventory report that would normally need to make a head request against S3, which you got to pay.

(01:02:08): I don’t want to go do that and I don’t want to bottleneck on making those calls to S3, because again, there’s just adds to the rate limit bucket. And on the flip side, if you don’t want to do excess metadata operations on the destination, you can tune Rclone to your heart’s content.

Fernando (01:02:26): I was going to ask, did we at any point consider doing random statistical analysis with like a full, give me the bytes back, do the checksum here.

Jeremy (01:02:36): We did.

Fernando (01:02:37): Okay.

Jeremy (01:02:37): We did. So we did some of the copies with the full pull and we didn’t run into any issues. We just did kind of a grab bag sample. I just manually ran some of the batches just to satisfy myself.

Fernando (01:02:51): Yeah, because I mean, you could do it for everything and we completely quote unquote completely sure, but it would be insanely expensive, insanely time consuming, but that makes sense.

Jeremy (01:03:00): Yeah. And it would suck a lot of our bandwidth. So all the bandwidth we want to be using exclusively for writes, we don’t want to be eating it up on reads, verification reads. Whereas metadata operations, they don’t use much bandwidth. They just eat up CPU time on the flashblades. So we were able to calculate all that out too, depending on the number of, kind of, compute cores in the flashblade cluster, how many metadata operations you can possibly do concurrently. So essentially we wouldn’t flood it. We would edge it just a little bit over what it ought to be possible, what ought to be able to do to keep it full utilized.

Fernando (01:03:43): That makes sense. I want to go back to the open sourcing. I feel it’s such a great part of this, of 37signals that we’re both contributing to open source in the patch that went out. But I’m curious if we are going to open source our tool, like this tool that you built.

Jeremy (01:04:05): I did build it with a mind toward open sourcing, and part of that was to … It’s almost like a kind of a design discipline that I’m not going to make something like too bespoke. And there were some decision points along the way. At one point, it’s easy for us to do an NFS mount, for example, and to use an NFS mount to share files, but it’s not terribly different to use an object store and to use Active Storage. So in a case like that, I erred on the side of using Vanilla Rails of using object storage rather than developing a different kind of file storage backend. And if I was just doing batch scripts, I would probably do an NFS mount, but as soon as I moved into Rails, it’s like, let’s just do it all the Rails way. And once you’re doing things the Rails way, it becomes almost hard to do it in a 37signals specific way.

(01:04:58): Of the things you hard code in, it’s easy not to. And things like where your credentials go, well, goes into a separate credentials area and things like AWS accounts and credentials are modeled in the database. I used Active Record encryption to store credentials in the database so they’re not part of the repo. It’s part of your onboarding process. You start with a blank app, you add AWS accounts, it takes your credentials and goes and scans for all the buckets you have and imports them into the app. And on a regular basis, it can go pull them and then you can start copies from that stuff. And similarly, you set up destinations and just the same way. So you can support different kinds of sources, different kinds of destinations, perfect setup for being a general purpose tool. As we got closer to the end, I did start specializing some of my design decisions based on the phase of the copying that we were in.

(01:06:00): So I started tuning the dashboard to reflect what I needed to know. And that sacrificed a little bit the things that are important earlier on. So as it got into like, “Okay, I just need this from me, I’m going to totally revamp things right here.” Now the UI is, it’s a little bit narrower and purpose built for my needs and not everybody else could necessarily understand it.

(01:06:26): So in any case, yeah, it’s very open sourceable. We’d like to do it. I would like to do it. The investment in doing it is like, okay, got to tease apart some things. And so one possibility is sharing it kind of like an artifact of here’s what this look like frozen in time at the end. And if you wish to take it and adapt it and turn it into something, because this is certainly not our line of business. We’re not going to make a product out of doing this.

Fernando (01:06:52): That’s right.

Jeremy (01:06:52): So I’m not going to spend a six-week cycle of work on polishing up something that then we end up being open source maintainers for. It’s like, no, I’m not going to … So I’ll share it, but I’m not going to maintain it.

Fernando (01:07:03): That’s awesome. And what do you feel are the next steps now that the full transfer is done, you deleted everything? Where do we go from here? What about backups? What about …

Jeremy (01:07:18): Yeah. So it’s all the other stuff. This is all the programming side of … I drove the copies and the transfers from the application and the software looking out. Our awesome ops team and Matt and in particular did all the operations side of how do we stand up these storage systems on the back end and feed them and keep them well fed. And now that we’ve got all the bits on a disk somewhere, how do we make sure we don’t lose them? It’s all the standard kind of data reliability stuff. And so we run through the cases of like, what can happen, what are the risks we need to mitigate? And things like data loss on a drive or losing a system, losing a power supply. Each of these things has its own kind of redundancy and that’s all on the system side. And there are other kinds of redundancy concerns like what happens when a truck backs into the power transformers at a data center and takes the whole thing down.

(01:08:20): Well, we’re not going to lose our data, but it’s going to be unavailable. So then we got the availability problem. Well, we’ve got a second site, which is our backup of the first. It’s kept in sync live and it lags in a sense that we don’t write to it directly. So we’re insulated from things like software bugs in our applications. If we accidentally delete something like crap, we wrote a bug that deleted some stuff. It’s like you need a backup. It’s not sufficient just to have like high durability. You also need to be insulated from other kinds of mistakes. And our second site is essentially that. We’ve got another storage system that similar size, similar class, similar write rate, and we replicate from the first to the second. So if there is an issue availability wise or durability wise, we’ve got a place to go and we’ll just flip the applications over to use the second one.

(01:09:17): Now, that’s not the whole story. For backups, there’s the kind of age old, three, two, one rule where you’ve got three copies of your data on two different kinds of media and one offsite. It’s a little bit rusty around the corners and the age of cloud stuff and particularly with modern flash because the old school thing was typically hard drives and hard drive failures and your other media would often be tape.

(01:09:47): And these days, that picture looks a little bit different. What is different media really? Is it other kinds of flash? Is all flash one kind of media. You need sufficiently different characteristics that if something catastrophic were to happen to one mode of file storage, your other mode would not be affected by the same risk factor. So we’re looking at doing a third site with hard drive storage or different kind of flash and acting as our insurance policy. And the other factor that we’re pulling in here is that our two sites are using the same vendor. So we’ve been very happy with Pure’s Flashblade product, and we had similar resource constraints, both in power and cooling and rack space in both data centers. So using Pure for both made sense and we’re able to smush them together into one contract, just sensible step, but that’s a single vendor and we run the same operating system.

(01:10:47): We do stagger upgrades, but if there’s a bug in the system and we deploy to both systems, well, we essentially have a single point of failure. We don’t have two systems anymore and we’ve got one operating system and the same bug affecting both. So what we’d like is a third system with a different vendor. So we’re looking at using MinIO, the kind of open core storage S3 combatible storage system. They’ve got a dual license where you can use MinIO free or you can get a support plan with them. And it’s what we had been considering using before we discovered Pure. So we’ll be going back a little bit and having not just a second vendor, but also going to have an open source fallback. That’s our insurance policy against things like storage vendors being acquired. So that is often the final destination that’s pushed us off of other vendors in the past of you got bought by some big company and now what do we do?

Fernando (01:11:46): Wow, that is a lot of things to consider.

Jeremy (01:11:48): I will say the other storage medium that was kind of a romantic diversion for quite some time was using tape for real. Back in the day, I operated tape libraries and there’s something that’s just kind of a retrofuture tech satisfying about seeing a literal robot going and grabbing a tape from a library and putting in a little drive and like, “Oh, I’m going to copy my stuff onto there.” And it’s super durable. You don’t need power for tapes just sitting there and I mean, their lifetime is like 30 years and you can go put them in whatever salt mine somewhere and you’re good. And you could pack up your tapes and you could carry them in luggage if you wanted to. There’s just a lot of aesthetically pleasing characteristics about it. So we really tried to make this work because you could get a whole multi-petabyte storage system going for like less than a hundred grand, which is as far as like capital outlay, it’s like fairly cheap per terabyte, but devil’s in the detail.

(01:12:57): So like these tape systems are built for older school systems that have a directory, like a file system that you can scroll through to see what needs to be backed up. Object storage is a little bit different. To figure out what needs to be backed up with object storage you need some kind of gateway, which pulls new objects into a scratch space and then backs those up to tape and keeps a catalog. And there’s just a whole finicky kind of additional system you need. And then you’d have the troubles of like, how do actually restore from tape? Well, the only case it would be in is like a truly end of the company kind of scenario where some kind of nuke has hit and you want to recover things and it’s going to take weeks to recover from tape. So yeah, it’s truly an insurance policy at that point.

(01:13:49): And like, well, it’d be an aesthetically pleasing insurance policy and one that’s fun to consider, but it’s not actually going to work. Dang it.

Kimberly (01:13:57): Jeremy, thank you for joining us. This has been Recordables, a production of 37signals. To learn more from our technical team, check out the developer’s blog at dev.37signals.com.

We couldn’t think of a better way to kickoff RECORDABLES than exploring recordables! We’ve had many requests over the years to explain how and why we use the delegated types pattern from Rails in our products and we’re finally explaining it in-depth.

Kimberly and Fernando sit down with Principal Programmer Jeffrey Hardy to unpack the recordables architecture that powers Basecamp and HEY. Jeffrey explains how delegated types made it possible to scale Basecamp for over a decade without constant rewrites. The discussion explores why this approach makes copying, version history, timelines, and mobile support dramatically easier, along with the tradeoffs and learning curve. We’re sharing a behind-the-scenes look at how 37signals builds and evolves complex products with a small team and the Rails pattern that helps make that possible.

Watch the full video episode on YouTube. And stay tuned for another episode of Recordables with Jeremy Daer in the new year!

Timestamps

• 00:00 — Introduction
• 01:20 — What “recordables” are and why they matter
• 02:15 — The delegated type hierarchy
• 04:15 — The challenge with single-table inheritance
• 09:05 — Organizing recordables and recordings
• 11:45 — Tracking change history with recordables and events
• 22:40 — Copying and moving content efficiently
• 27:54 — Pagination and querying across content types
• 34:00 — The learning curve with delegated types
• 39:54 — Building new features faster with reusable behavior
• 43:03 — Long-term scalability and performance

Links & Resources

• Ruby on Rails API: ActiveRecord::DelegatedType
• Ruby on Rails Guides: Active Record Associations

Transcript

Episode Highlights (00:00:00): You got to think of the recordables as being pretty dumb. They’re just like, in the case of a message, it’s literally just a title and content. That’s it. They have no connection to the outside world. You can build entirely new features that should take months in like a week, two weeks. We’ve been running this architecture for 10 years.

Kimberly (00:00:19): This is Recordables, a place where the 37signals team share some of the technical work behind Basecamp, HEY, and some of our other open source projects. I’m Kimberly from the product team joined by Fernando who’s helping out with some of the technical aspects of today’s discussion. And this one, we’re excited to kick off this series with an episode about recordables. To do that, we have principal programmer Jeffrey Hardy with us today. Jeffrey, thanks for being here.

Jeff (00:00:46): Hey, thank you. This is great.

Kimberly (00:00:47): Before we get started, I know this is a topic people really want to hear about, but tell us a little bit about you and what you do here at 37signals and how long you’ve been here. You’ve been here for a while.

Jeff (00:00:56): Yeah, I’ve been at 37signals for 18 years, so I’ve worked on all our products, all versions of Basecamp, including the most recent version, and HEY, and everything we’ve done. It’s been a really great place to spend 18 years and have the advantage of seeing the evolution of Basecamp through all these different versions. So yeah, I work on the product team.

Kimberly (00:01:20): Amazing. Well, we know people want to hear about recordables. We’re excited to kick off this new series where we’re sharing some of the behind the scenes stuff with this episode. And before we dive in, I think I have to ask kind of the obvious question as the non-technical person here. I always hear people talking about recordings and recordables. Will you just kind of break that down? What does that even mean before we deep dive into the topic?

Jeff (00:01:43): Right, it’s a little bit abstract. Recordings and recordables and buckets and… what is this? So in Basecamp, pretty much all content, things like documents, messages, comments, uploads, these are modeled as a pattern we use called a delegated type. And so in the delegated type pattern, which comes from Rails, you have a primary table. In our case it’s recordings. And recordings reference recordables, and the recordables are like the concrete types. A recordable would be a message, a comment, a document, or an upload. And so with this pattern, we’re able to model all the various content in Basecamp, which is essentially all the same. There’s a bunch of advantages to treating all that content in the same way. And Basecamp’s architecture hinges on this concept, and it’s our third iteration of Basecamp. We call the version of Basecamp, the internal version is three. We call that the chassis, like the architecture on which it’s built.

(00:02:51): It’s our third attempt. And so when building the third version, we came up with this pattern that we use and it’s called a delegated type, which I know it doesn’t make a lot of sense. What does that mean? But I can describe the sort of hierarchy. So you have recordings, and this is a table of records that stores all the metadata and common information between recordable types like messages and documents and uploads, but it doesn’t have any specific information. It doesn’t know the title of a message or the content of a message or the location of a file or other things. It delegates that knowledge to other records. Those records have their own tables in the database, messages, documents, uploads comments, all separate tables. And the advantage here is that you have your single recordings table that delegates its types can be lean and tight.

(00:03:53): It only has metadata. It has references to recordables. So it has a recordable ID and a recordable type. It has timestamps. It has a creator id. It has tiny little columns, no text columns, nothing big and heavy. And so all of the specific information is contained in the recordable tables, which can vary. So to back up a little bit, in Rails and in software architecture patterns in general, there are a few ways to handle this idea of things are mostly the same, but they’re kind of different. And how do you account for the differences without repeating yourself? And one common way is a pattern is called single table inheritance. You have a database table and you have subtypes that also use that same table. That’s where you would have your shared metadata, the things you have in common. Then comments might have a body and messages also have a body.

(00:04:50): And so you’re like, great, I have a body column. I can use the same table to store these. I just have to keep the type in the table. Is this a message row? Is this a comment row that works good for things that are super similar, like maybe messages and comments. But what about things like events? They have a starts at and an ends at column. Now you’ve got to add those to the main single table. And so the table keeps getting bigger widthwise. It has to have all the different kinds of columns that you might have for any type. And adding a new type means you have to modify that single table and that table gets really, really big. It’s just sort of intrinsic to having a single table that does that. The bigger the table is, the harder and slower it is to migrate.

(00:05:38): So that’s one way to do it. Another way is to have just different tables for everything. And then you take the things that are the same and instead of belonging to a specific kind of record, lots of things, for example can be commented on in a system like Basecamp. So you might have a documents table and you might have an uploads table and a messages table and you want comments and all these things. So the comments table could instead of belonging to, having a reference to a specific thing, like having a document ID or an upload ID or a message ID, it can belong to a polymorphic type. It can store a commentable ID and a commentable type. And so now you get this is called polymorphic relationships. That’s another way to model this sort of, everything is the same but slightly different. We need to differentiate. And the delegated type pattern is neither of these, but it’s similar and it’s something that I think we pioneered it at Basecamp. Maybe some other really smart people have come up with this, but it’s possible they haven’t. But that’s what I can talk about today is how this system works. It’s a really neat system and it’s enabled us to scale Basecamp and to use the same version of the chassis of Basecamp version 3 for multiple subsequent versions without having to do a rewrite.

Kimberly (00:07:12): So just to clarify, we’re currently working on Basecamp 5, we said that, but we’re building it on the same platform as 3 was built and 4 was built. And that is because of this recordable situation?

Jeff (00:07:27): Yeah, I mean the main reason we made other versions of Basecamp was because they become too difficult to work with. They’re kind of rigid and we also want to introduce new features or have a different take on how we do something. And that’s hard to change when existing customers are using it. But it’s also been good to sort have a blank slate and start over because you don’t have all this baggage, this big ship that’s hard to turn around. While using this delegated type recording recordables pattern has made the ship less big.

(00:08:03): I was talking about single table inheritance and how you’ve constantly got to modify that single table and migrate it every time you want to add a new kind of thing. You don’t have that with the delegated type recordable pattern. Your recordings table basically never changes. You just add new types, which are their own fresh tables. And so there’s no penalty to adding a new type, deciding that you want to add comments or messages or documents or any other thing that we can imagine. You just make a new table and you’ll have new recording rows that reference it, but you don’t have to modify recordings. So it’s been much easier to scale than other versions of Basecamp. And consequently, we don’t start hating the effort of doing a small thing. You want to be able to iterate on your product easily. And I think that in the past, making a new version of Basecamp was the best way to do that brand new, and we just haven’t felt that pain with the version 3 architecture.

(00:09:05): And it goes entirely to this recordings recordable pattern. Now, there are a few things about the, it’s not just the delegated type. So for those that don’t know delegated type, it’s available in Rails. The documentation in the Rails docs is pretty good, but I think a lot of people don’t quite understand when they would use it or why to use it. Hopefully we can clarify that today. But there are a few other patterns that we use in addition to the delegated type with recordings that make this content system possible. One is that the recordables themselves, we treat them as immutable, so we don’t modify them in place. We create new ones. And another thing that we do is the recordings are organized in a tree, like a parent child relationship. So I’ll go back to the messages example because it’s the most intuitive. We have a message board, and a message board’s children are messages. And a messages children are comments. And a comments children are like attachments. If you’ve uploaded a file to your comment.

Kimberly (00:10:17): What about a boost? Would a boost be a child?

Jeff (00:10:20): Ah. No, boosts are different. Boost actually used that other pattern I was describing called polymorphic relationships where boosts belongs to a boostable thing. And that’s because we want to boost things that aren’t necessarily recordings. You might want to boost a status notification or an event. Events are another key part of the pattern. So that helps me tie into events. So we have the delegated type recording recordable pattern. We have the arrangement of recordings as a tree, we have immutable recordables, and then we have an event system. And the event system can tie a recording to a recordable at a particular moment in time. And so this is why said the recordables themselves are immutable. So we never change them, but we can move the pointers to them around. And at any given time, a recording only points to one recordable. But if you want to have a history of what are all the recordables that it ever pointed to, we track that in an event model. The event has a reference to the recording and it has a reference to a recordable. This means we can look at the history of a recording and see all of its changes and look at that recordable that is immutable at any moment in time to see how it looked.

Kimberly (00:11:43): Yeah, here’s a good example here.

Jeff (00:11:44): Yes, yes, I know what you’re going to show here. This is great.

Kimberly (00:11:49): Do you want to walk us through this change log?

Jeff (00:11:53): Yes, this is a document in Basecamp and what enables this, what we have the change log here, and what enables this is this combination of immutable recordables and events. So what you’re seeing here in this history of changes is really a listing of all the events for this particular recording. And so where you see save to change to this document and you can say, see what changed, what we’re able to do is find the document recordable instance at that moment in time. This is how it looked, this is what its content was, and compare it to the previous version. And we can only do that because we didn’t update the document in place. We created a new one. Now we can compare version A to version B, and you’ll see there’s a button there. Make this the current version. When you click that button, all we’re going to do is update the recording record to point to this version of the document instead of the current version.

Fernando (00:12:49): So recordings are not immutable,

Jeff (00:12:51): No recordings are fully mutable. We change them. It’s the recordables, the type that we delegate to,

Fernando (00:12:57): But what are we changing in a recording? We’re only changing the point of reference to the recordable, right? Yes.

Jeff (00:13:05): There are a few other things that can change. One is its timestamp, like it’s updated at timestamp.

Fernando (00:13:09): Oh, of course.

Jeff (00:13:11): Or if… most other things won’t change. We store a color value on the recording so you can change its color, presentational, things like that, but…

Fernando (00:13:24): Oh, that makes sense.

Jeff (00:13:26): Yeah, you’re right. The recording is mutable, but many things about it don’t change. All of the meat, like all of the action is in the recordables themselves.

Fernando (00:13:36): And you mentioned, for example, a document can have comments. And so the logic I have in my mind here is I want to get to this document. I have the recording. The recording has a reference to the recordable document. When I fetch that recordable, that document, which is a recordable, are the reference to its children… are the references to its children recordings.

Jeff (00:14:08): Yeah, so it’s only the recordings that have children. So you look up a recording by primary key, that would be the ID that you see in the URL. And then you ask for its recordable. And the thing is you don’t care often. You don’t care what kind of recordable it is, if it’s a document or a message. When you do care, the pattern that Rails exposes lets you ask for that type of recordable by name. So if you know want a document, you’re expecting a document to be the recordable, you wouldn’t just ask for the recordable, the generic recordable. You would say recording.document or recording.comment. Another convenience that Rails is delegated, type provides is the ability to query and filter for the type of recordable, the type of delegated type. So you would say recordings.messages would only return recordings that have a type of message. Or recordings.comments would only return recordings that have a type of comment.

Fernando (00:15:09): And if I do like recordings.messages.children, I would get back an array of recordings.

Jeff (00:15:16): Yes. That were messages, right?

Fernando (00:15:20): No, no, no. The children of messages. Let’s say comments.

Jeff (00:15:24): Yes.

Fernando (00:15:25): Okay.

Jeff (00:15:25): Yeah, you’re always going to get recordings. So the children and the parent, they’re always recordings.

Fernando (00:15:28): Always?

Jeff (00:15:30): Yeah, this is a good point to clarify.

Kimberly (00:15:33): Is this something we can show? Or no?

Jeff (00:15:34): I kind of can. Yeah, I can show this. It might help to see how the code is organized. Let me share my screen. So here I’m on my Mac. I have been using Omarchy on a Framework 13 laptop, but my camera is not working great right now. Now it’s probably configuration, I don’t want to say anything bad about Framework. It’s an excellent device, but I didn’t want to risk it for this presentation yet. But ok, so this is the recording class in Basecamp. So what you’re seeing is a whole bunch of concerns. Basically, you see there’s very little, the main model in Basecamp is 42 lines long. It’s kind of incredible. But you can see a few things immediately that are important about a recording. It belongs to a bucket. And I’ll talk about, I can just introduce the bucket briefly. The bucket is just a container for recordings. Buckets are how we control access in Basecamp. So buckets have accesses. You add people to a bucket. If you can have access to the bucket, you can see all the recordings that are in the bucket. Buckets are also a delegated type. A kind of bucket is a project, a template, a ping. Anything that can contain a distinct set of recordings is a bucket.

Fernando (00:16:55): So is a bucket a recordable?

Jeff (00:16:58): No. A bucket is a delegated type, which creates a bucketable. So have the recordable is just the name we’ve given to the target of the delegated type. So in HEY we use the same pattern except in HEY we’ve called them entries. Entry has a delegated type, which we’ve named entryable and in HEY those are things like messages, replies, notes on a message, and these represent emails. So the naming can be whatever you want, but the pattern is the same. So buckets have bucketables. Bucket is the thing that delegates to a bucketable.

Fernando (00:17:36): Oh, okay, perfect, yes.

Jeff (00:17:37): A recording is a thing that delegates to a recordable. And in HEY, an entry delegates to an entryable.

Fernando (00:17:44): And so a bucketable can be a recordable depending on the context, right?

Jeff (00:17:48): I mean technically, but no, you wouldn’t do that. Bucketable would be a different thing. But there’s no reason, they’re just classes. They can reference any kind of thing they want, but we want all of our recordables to behave the same or have a similar interface. And you would want that of all the types you intended to delegate to. So you want all bucket things to quack like buckets.

Fernando (00:18:14): To bucket buckets.

Jeff (00:18:16): Right? As containers of recordings in this case. Or in HEY, as things that are like email messages. Or in Basecamp, recordables, things that are like our chunks of content, documents, messages, uploads. So mixing the types while it would be technically possible would be kind of confusing. So the thing I want to point out here in the recording class is right here, this little inclusion of a Ruby module called recordables, and I have that up here. This is what the recordables concern looks like. And right here on line 5, this is what does all this work. So we say, imagine that this is mixed into recording. So we say that recording has a delegated type called recordable. All the types that it can have are named in this constant, which is in another file, which I’ll show you later. And this does all of the work. And so these are by mixing this into recording, these are all the things that a recording can do.

(00:19:24): But you see this is also a really small file. It’s like a hundred lines long. It doesn’t do a whole lot. It doesn’t even include a lot of other behavior. So where does all this come from? And so this is sort of the key. So this is the recordable class. This is what every individual recordable class, like a message, a comment or whatever includes, and this is where all the types are defined. So all of these, an attachment, an auto link image, a bulletin message, a bulletin link, all the chat things, like a chat, everything from a chat line to a chat transcript, a cloud file, a comment, a doc, a document, a door. All of these things are recordables. So every one of these content types, which is tons are all modeled in the same way as far as recordings and recordables go. And then all of the things that recordables have in common are listed here.

(00:20:23): So I’ll scroll down a bit. These are things that they can do. Are they auto subscribable? Are they subscribable in general? So this is where the type that we delegate to defines its capabilities. So by default, no recordables are subscribable, but if we go to a document and we look like, oh, it enables subscribable. Here it includes the recordable class, the recordable mix in, and now it overrides the subscribable method to say, yes, I can be subscribed to. I can be exported. I can be commented on. I’m auto positioned when I’m placed inside a list. So all of these things you can kind of like, respondable, backlinking, like when one recording references another, we can create a link to that so you can navigate it easy in Basecamp. If something’s copyable or movable, if it’s recurrable or recurring, if it repeats. These are things that you can ask of any kind of recordable.

Kimberly (00:21:25): Jeff, is there something in here we have public links that we can make, is that part of this?

Jeff (00:21:30): Yeah, it probably would be as publishable.

Kimberly (00:21:34): Publishable.

Jeff (00:21:36): Yeah.

Kimberly (00:21:37): Publicly linkable.

Jeff (00:21:37): I’m looking for it. Right? Is it memorizable? But I think… do we have it? I think all recordings we’ve defined as are publishable, so we tend to list only the things that might need to vary. So if some things are commentable, but some things aren’t, then we define a commentable capability and let other recordable types define it, override it. You can see some of our comments here too. There’s lifecycle events that happen. A recordable is recorded, meaning we’ve created a recording that references this recordable. And often you have specific behavior that isn’t going to apply unless you… it only applies to specific types. You might want to do something with a comment being recorded that you wouldn’t want to do when a document is recorded. And so this is one of the challenges of the delegated type and type pattern in general is that where do you stick that specific behavior?

(00:22:41): You’re mostly working with your type that does the delegating, the recording, but you’ve got specific behavior that should only apply to certain recordable types. And in that case, we pass in the context of the recording because recordables themselves, they don’t know they don’t belong to any particular recording. They can be referenced by many recordings. And in fact, that’s how copying works, and that’s how we make copying really efficient. Instead of actually copying the content of a message, we just create a new recording row and point to the same message recordable that already exists. We don’t need to copy it at all, and it’s super fast and storage efficient for that reason. But it means that a message doesn’t know, it doesn’t have any particular recording that owns it, so it can be owned by many or none. And so sometimes when you need to work with a recording, we pass it in. So here we can say the comment reads overwrite to perform recordable specific actions after making a recording. So comment uses this to compose its title based on the parent of its recording. So…

Fernando (00:23:56): That’s the whole purpose of buckets then. Recordables have no understanding of access control. So if you’re in a private project, you have your own document there and you want to copy it over to a public, it’s just referencing the same recordable. The recordable, there’s no change in access to that recordable because access goes through the bucket.

Jeff (00:24:18): Yes, right. They’re sort of orthogonal in that sense though. It’s more that the recordable is, it doesn’t even need to, we don’t even keep things like timestamps on our recordable records because they’re just chunks of content.

Fernando (00:24:34): Yes.

Jeff (00:24:36): Know what I mean? Some recordables don’t even have any row, any columns. They’re just a table that has an ID, their rows just have an ID. They sort of act as a placeholder, if they don’t have any distinct content to store, or they might just have a title, you know?

Fernando (00:24:56): Right. That makes sense. So they can be as small or as large as they need to be.

Jeff (00:25:02): They can be, yeah, they could be as smaller as large as they need to be, and you get to modify them without having to modify all of the recordings, which becomes the big table. The table with billions and billions of rows. I know it’s a little abstract even when I’m showing the code, it’s kind of like…

Fernando (00:25:20): No, no, no, I think it makes perfect sense. You have the recordings, which themselves are just fixed point. They’re the latest pointer to a recordable.

Jeff (00:25:30): Yes, they reference usually the latest version.

Fernando (00:25:34): Yeah, usually. And then, however, if you needed to go through the history of how that recording got to a certain recordable, you go to the events table, right?

Jeff (00:25:44): Yes. You ask the recording for its events. And you could go over each event and say, what was your recordable when this event was created?

Fernando (00:25:53): Now for example, if I’m in a project, what is the algorithm? What are the steps? If I’m opening a tool and I want to see the messages.

Jeff (00:26:06): Right. So the bucket has recordings. The bucket is the project. A bucket delegates its type. Buckets are just things that have recordings. And so if you were in a project, we would be looking for buckets that have a type of project. Then from the bucket record, you can ask a bucket for all of its recordings. You would want to filter those recordings to just those that are messages. Your call would look like bucket.recordings.messages. Now you’re going to get a list of recordings that all point to message recordables, right? So in Basecamp, the messages themselves because of the tree structure, are owned by a message board. So every bucket has one or more message… every project bucket has one or more message boards. You would find a message board — bucket.recordings.messageboards.first. This would give you a recording of type message board, and you would ask for its children.

(00:27:04): Now in Basecamp, the only children of a message board are going to be messages, but you could have different types. So you would want to filter that down and say, just show me the messages. So you would say message board, your message board recording. You would say children.messages. Now you’ll only get its children that have a type of message, still recordings, it’s recordings all the way down. When you finally get to the message recording that you want, you would ask for the recordable. That’s where the title is and where the content is and where other message specific things are.

Fernando (00:27:40): So how do you enforce that? I am coming from a very compiler part of the programming world, and all I’m hearing is let’s make a message board hold pings and chats.

Jeff (00:27:54): Right? You don’t enforce it, right? There’s no method to enforce that. It’s just how you do it. I guess you could say it’s convention, and that’s why the conveniences the delegated type provides like the ability to filter on type are useful. You probably wouldn’t want to just ask for any child because it could contain mixed types. Let’s think about in Basecamp, you have documents, but you can have Google documents and uploads, and all of these go into a container, a parent that we call a vault. A vault is a thing that holds document like things. So you wouldn’t want to just, if you ask a vault for its documents, you’re only going to get documents. So there’s an instance where you do just want to ask for children. All of the things that this might contain, you want the mixed result. So this actually brings me to another cool thing.

(00:28:50): This is one of the reasons we have this pattern is this mixed use case. Let’s say you have a timeline. We have a timeline in Basecamp. We want to see everything that’s happened at a global level. How do you do that and paginate it if all of your records are in different tables? You have to select from the comments table and the documents table and the uploads table and the messages table. Like everything, it’s super hard. But with recordings, you’re just querying for recordings. You can say recordings where recordable type is message, document, comment. You’re going to get all the ones that are that type in one query that you can paginate with a limit and an offset. Super handy. And that query itself is cheap because you’ve only asked for recordings rows that are not big. They don’t have any text columns.

(00:29:50): It’s sort of n plus one by design. But as you iterate over those, then you go and fetch the recordable records. And to improve performance, you can preload them, get them all in one batch for each type. But the key thing that you’re able to do is work with just recordings as if they’re all the same. So I’ll give you another example, unless this is too much, but another example where this uniformity matters. Copying, right? Or any, actually just think not just copying, but any generic process that you’d like to do with a recording. We can export recordings, right? We have an HTML export feature in Basecamp. It’s nice to be able to write the exporter service that doesn’t care about its individual types. All of these, the stuff that would apply to exporting a recording is common to any recordable type, you know? It knows how to write itself… it knows that it has to write something, some output, and then it can delegate the what to do to its recordable.

(00:30:53): So we basically say recordable, write out your export format. Thank you. It means that you don’t need to change the exporter when you add new recordable types, right? Each recordable type just has to define the format that it exports as. Same thing, the copying and moving. You write a copier that works with recordings. That’s it. It doesn’t need to care about what kinds of recordings it’s copying as it iterates over each recording that it’s copying, that recording sends a message to its recordable that says, I would like to, you’ve been requested. A copy has been requested of you, and it just does its thing. So again, adding new types, there’s no penalty. You just add them and the system can work with them. So any of those, what we have in Basecamp, like a bunch of controllers, these are what field requests that come in over HTTP.

(00:31:50): So instead of having to have, like if you want to trash a document or archive a document, we don’t need a separate documents controller for this. We have a recordings controller that knows how to trash things and archive things and restore things. We just need one. It works with any kind of recording. Same thing, that’s how copying and moving works too. We don’t need a document copier and a message copier and an upload copier. We have a recording copier, one controller. It takes the recording that you want to copy and a destination and the copier itself and the controller that orchestrates it doesn’t care what kinds of recordables it’s working with. It just works with recordings. So I think that’s another one of the reasons why Basecamp 3, that architecture has been so easy to maintain and scale. When you decide like, oh, we’re going to add Hill Charts. Oh, we’re going to add, I don’t know, templates. You don’t have to change or migrate the existing system. It adapts well to change. You just insert new types and the system knows how to deal with them.

Fernando (00:32:52): Are you at any point worried that the recording stable is massive? Billions of entries?

Jeff (00:33:00): I mean, it is billions of entries. Not really because its size on disc is small, so it doesn’t take much to index it. But I mean, and if it did, let’s say it did get too big and we had to shard it or break it into pieces, it would be much easier to do because it’s an inherently lightweight table. It really just has foreign key references in it. That’s it. By contrast, the messages table can have many megabytes of text, and these are big on disc. And so the way copying or making an index on disc works is that you’ve got to copy the table. It’s still an entity on disc that you need to copy, and the bigger it is, the slower. So yeah, there is a danger that recordings gets too big, but if it does, it’s much easier to deal with than if it was storing content.

Kimberly (00:33:50): Jeffrey, you’ve made it sound like this recordables pattern that we’re using is much simpler, makes things a lot easier. Are there any downsides?

Jeff (00:34:02): Yeah, I think probably the main downside is familiarity and with the way you would quote unquote normally do this in Rails. Normally you would have a message class, a document class, and everything it knows about being a message or a document would be encoded in that class. I would call that a rich class. It has all the functionality in it. This is what the active record pattern is about. It’s backed by a database table, but you never interact with the database table directly. You interact with your message model. And so if there’s specific things that a message would do that are not generic, you would write those methods right inside the message class or the comment class. And with the delegated type pattern, you kind of want all of your types to be generic and you’re not working with them directly, ever. We never look up just a message because the message on its own is it’s not linked to anything.

(00:35:01): It’s not useful. So in order for a message to implement its specific work, it often needs a reference to a particular recording. If a comment wants to know, you saw in that one comment when I was showing the code where it said comments can set their title based on the parent that created them. Well, a comment doesn’t have a parent. A recording can have a parent. So in order for the comment to be able to do this, it needs to be given a recording to work with. Now it can ask for, it’s that recording’s parent and be like, oh, okay, great. Now I can change myself based on some context. You got to think of the recordables as being pretty dumb. They’re just like in the case of a message, it’s literally just a title and content. That’s it. They have no connection to the outside world.

(00:35:53): They don’t have any associations and nothing points to them directly except for recordings and events. So I think some of that richness gets a little bit lost. You don’t do things quite in the Rails standard way, where you tend to think more abstractly, like how is this a generic concern? How if I want something to be commentable, do I make this work with any kind of recording? So it means that you would have to add some facility to the recording class that helps it stay generic. That’s where more delegation comes in. So you would never just add a comment to a recording. You would ask, is this recording commentable? The recording can’t answer this itself, but it can ask it’s recordable and pass the message along. So it can say like, document, what do you say when I ask are you commentable? And document says yes. But something like, I don’t know, message board. The message board container itself says, no, I’m not commentable. You can’t put comments on me directly. So in the generic comments controller, one of the preconditions is is that the recording has to be commentable. Now you can have one generic controller that handles comments and you can pass it any kind of recording, and it just asks that recording, are you commentable? Can I work with you? And if the recording says, yes, you’re good. And if it says no, it’s an error.

Fernando (00:37:21): Speaking of drawbacks, is that, I mean, I think I know the answer to this, but is that slower? Are you trading speed for clarity?

Jeff (00:37:31): I mean like development speed?

Fernando (00:37:33): Yes.

Jeff (00:37:34): Maybe a little upfront. There’s a higher learning curve cost, right? So when we onboard new developers, it can be non-intuitive. It’s like, ah, I want to add some behavior to the messages. I just opened the message file and there’s nothing in it. Where’s all this behavior? And we’re like, oh, yes, it wouldn’t go here because…

Fernando (00:37:57): Rails magic.

Jeff (00:37:59): But it’s like, no, you wouldn’t put it here. You sort of have to define it in a more abstract, generic way. And then I think that then it becomes a huge net benefit. Once you’ve learned the pattern, now you start to get, because then you’re like, well, I want my message to be copyable. And it’s like, oh, that already works. I want it to be commentable. Oh, just add a def commentable method and return true, done. All the commenting works. I want it to be exportable. Done. Just add an exportable method. So you start to realize, oh, what I’ve sacrificed in being able to make changes to message directly and add some richness to that… I’ll give you an example. Messages can be categorized in Basecamp. You can make categories. It’s useful, but it’s not really useful for other types, maybe documents, but we don’t have that feature enabled.

(00:38:55): So I think in a standard Rails app, you would say that a message belongs to a category and categories have many messages, but in the delegated type recordable pattern, you’ve got to make it so all recordings have the potential to be categorizable and only message opts in. So that cost is there upfront, but then a month later when someone’s like, hey, can we have comments or categories on documents? It’s like, yes, that’s easy, no problem. And so we’ve gotten a ton of leverage out of that. It allows you to build a lot of things that should be hard, just are not hard. They come down to a matter of configuration. You have the system behaves in a uniform way. And where I think this works best, ‘cause this won’t work in all application domains, and we don’t use a delegated type for every kind of model.

(00:39:54): But in Basecamp where you have basically the same chunks of content, like you would have in a content management system or a Wiki or something Notion, what these chunks of content have in common is the operations you can perform on them and with them. And so their ability to just participate in that system just by being defined is incredible. We added the Card Table, our Kanban board system, we call the Card Table, and it was super easy to do because we’re just like, all right, we have a board. It’s got children, you can watch a column. Well, that’s just a subscription. So we’ll make those columns are a kind of recording. Their children are the cards themselves. Each of the cards can have comments. The column itself can be watched, the board itself can also be watched. You’re sort of mixing and matching behavior that already exists.

(00:40:58): And then when it’s like, well, how would I move a column to another column? It’s just a move operation. All recordings are movable. They can move to different containers. So that just works. How do I get a reminder about a due date I’ve given myself on the card? Oh, that just works because all recordings are remindable. And so you’re like, you can build entirely new features that should take months in a week, two weeks, I mean excluding the design, but the software modeling part. And it’s all uniform. So once you’ve figured out how this works, you understand the whole system. It doesn’t scale up with the number of types. It behaves the same whether you have one kind of recordable or 200.

Kimberly (00:41:46): It kind of sounds like, Jeff, I’m imagining as you’re describing this, a whole bunch of Legos or puzzle pieces that are just being redone in a different way. You’re taking elements that you guys already have and reconfiguring them to make a new feature.

Jeff (00:42:02): And if you think about Legos in particular….

Kimberly (00:42:03): I’m probably grossly, oversimplifying that.

Jeff (00:42:06): No, no, but Legos are a good example. If the Legos didn’t fit together, if they didn’t have the same hole and bump pattern, it wouldn’t work. What you get is being able to say, okay, here’s all the Legos, and they’re all interchangeable. Every Lego works with every other Lego and that’s why it works. So there were all these things that motivated the pattern, and what I don’t think we could have anticipated was how well it worked out in the long term, right? From being able to query efficiently to query across types with pagination, to be able to copy and move things really efficiently because we are not having to recreate the message content every time. If a message gets copied 100 times, there’s still only one message recordable. That’s it. Whereas in earlier versions of Basecamp, we would have 100 different rows that each had a copy of that content.

(00:43:03): So things like this adds up. Whenever you start a new Basecamp account, you get a demo project. So with thousands and thousands of Basecamp accounts that all have the same starting content, and we would need thousands and thousands, tens of thousands of rows to represent this. And so you get a really, really big database. Basecamp is the majestic monolith. We don’t use microservices. It’s one app with one big database, and there’s a lot of efficiency in terms of workflow that you get from just having one database. And this is one of the patterns. This delegated immutable types in particular make that possible. Stay in the majestic monolith as long as you can. We’ve been running this architecture for 10 years. Other versions of Basecamp didn’t make it quite that long before we created a new version, and then therefore they’re still growing, but it really reduced the rate that the content is growing in those systems.

(00:44:05): But Basecamp has just been going up and up and up for over 10 years, version 3. So this architecture scales really well. We’re totally confident to build Basecamp 5 on it. It’s just a little bit different maybe than what you’re used to. So once you, and I don’t know how widely used it is because of that. So David is going to do a talk on and a code demonstration of the mechanics, but that’s useful. But we sort of already have that too in documentation and still people miss the big picture. So I’m hoping that that’s what I am able to convey here is why would you do this? Why not just have messages and documents and separate records like the universe intended? And that’s why. I mean, it is fine for a lot of purposes. There is a cost to this level of abstraction, but it’s not as great as you might think and it pays dividends once you’ve made the investment.

Kimberly (00:45:05): I mean, I’m fascinated by this.

Fernando (00:45:08): Oh, I agree. I’ve been trying to think…

Jeff (00:45:10): Well, Fernando, you work on the mobile team and think about the API. We’re able to expose a single recordings, JSON API. Pretty great.

(00:45:19): You can just ask for recordings and you can provide filters between this time and that time. You’re going to get all different types, but this is really easy to implement. And you can do things like your native code can do things and not… It means that things just work. So we add a new type and we don’t need to issue a new, create a new build of the mobile apps. It’s going to get a new kind of recording and it can deal with it in a generic way. It knows that all recordings have titles, all recordings might have a position. All recordings can be archived or active or trashed. It’s not like, oh, everything breaks because we introduced a new type. And that’s one of the other ways because we also are super small team. I mean super small at times. There’s only a handful of people working on any of this stuff, and you would need way more people if you had to coordinate these sorts of changes.

(00:46:15): It means that we just deploy Basecamp whenever and we’re confident that nothing is going to break even as we had new things. You can’t do that for everything. But as far as recordings go, which is the bread and butter of Basecamp, these different kinds of content, these content blocks that are fundamentally content oriented, all behave in the same way. And it means that we ship new stuff and we don’t have to wake Fernando up at night or anybody on his team to say, you need to do a release because we busted something. That’s the kind of efficiency I think you get, and that’s the kinds of things we’re looking for so that we can stay a small team.

Kimberly (00:46:54): If someone is like, yes, this is a better way to work or this is a better way to do this programming with the delegated types, all of this, is it something that you can incorporate into something you’re already doing or do you have to start from scratch?

Jeff (00:47:09): You can incorporate it into what you’re doing. It is a little hard though. It’s definitely better to start from scratch. The migration path would look like it would be a little bit complicated. You would introduce your main table, we’ll call it recordings that delegates its types, and you would either create new types, so if you already had comments, you would need to name it something else, like recordable comments and migrate your content into that. Create recordings as you go. Because the way you create a recording is you create a recording record and you pass it a new recordable, and then when we save the recording, that’s when we update the reference and persist the recordable. So you would sort of do that process and move your content into it. So we’ve done this before when we worked on the HEY calendar. The HEY calendar also uses the delegated type pattern for all of its calendary things. Think of all the things that can appear on a calendar day. You’ve got the event itself. You might want to circle the event. You can have a countdown.

Kimberly (00:48:11): You might want to name it.

Jeff (00:48:12): Name it, it has a title, right.

Kimberly (00:48:14): You can add a picture.

Jeff (00:48:15): Yes, all of these things are implemented as… a calendar is the bucket, and the bucket has recordings. And the recordings are things that are eventlike. Now I talked about how the recordings table, the main table, stores like the metadata, the stuff that all of the types have in common. So in an application like HEY calendar, they have… everything that can appear on a calendar has to have a starts at and an ends at timestamp. You can’t put it on a calendar unless it has knowledge of when it exists.

(00:48:49): So in that case starts at and ends at aren’t delegated to an event type, they’re on the recordings table. That means we can query for any kind of thing that might be on the calendar and any moment in time from one place. That’s what allows us to in one query, fetch all the things that we need to display on a given day or in a given week or month. So part of it is, it’s that you find the stuff that never changes, the things that all of the types would need, and that becomes your main class, your main table. And then you’ve got it when you can just query that one table to get everything you need. Our first iteration of HEY had us going through the different tables, so we had to query for events and we had to query when they’re like, oh, we’re going to add other things to the calendar. I forget, even just being able to circle a certain day or something. And they were all slightly different.

Kimberly (00:49:51): And it’s like tasks. Sometimes this week tasks.

Jeff (00:49:54): That’s another great example. And so we did this and then we had to migrate into that. This was still while we were developing, but we didn’t do it by deleting everything. We were able to transition to it. But that’s a great question. It is the kind of pattern you want to start with. The single table inheritance pattern and the polymorphic pattern are similar. Yes, you can migrate into them, but it’s a pain. It’s better to identify upfront that this is something you might want to use. And similarly, you could go the other way. You could start a delegated type and turn these all into concrete types that have their own tables. You would iterate over the recordings. For each recording you would create a new record, you know. Take its recordable content, take the recording metadata, stick them together and insert that in a new table. So there’s migration paths to and from. But you’re right, you want to start with something like this.

Fernando (00:50:53): Jeff. I’m skeptical. I have to say, I work at 37signals. I work with Basecamp all the time. It is very fast. Why is this not more popular? Is it like obscurity? Is it like, what? Because like, come on, we’ve been talking about this for however long the episode has been going on and it’s like, I keep trying to, I’ve been thinking about poking holes at it. There’s like very few…

Jeff (00:51:25): I don’t know. I think it’s just familiarity, which is why I’m glad we’re doing this and sort of educating people. I think it would be a lot more used if people understood its strengths. The Rails documentation on it is good, but it’s kind of terse. It’s kind of like describes a shallow version of the benefits. But you talked about fast, speed. That’s another great example. Caching. We don’t cache at the recordable level.

Fernando (00:51:50): Oh my. That’s right.

Jeff (00:51:52): We cache at the recording level. It means that the way caching works for any kind of content type is identical across content types, across recordable types. We cache on the recording. When you want to expire the cache, you touch the recording. In the tree pattern, if the tree needs to touch its children, it can, but it’s more common for a child to touch up its tree. But then we have like, we call it the Russian doll caching pattern, where it doesn’t matter, you can invalidate a child way down the tree and not need to invalidate the parent. The parent can still render itself, and then when it gets to this child, it’s like, oh, this part I need to change. But yeah, you don’t get to use the same caching scheme for everything when you don’t have a uniform thing like a recording. So what you’ll see in our views is like “cache (recording) do…end”. Everything is wrapped in a cache block keyed only on the recording. So you get to share a ton of stuff in terms of, it’s not just the, I’ve been mostly talking about the domain model, but this also this cascades to the controllers and the views. It’s all the same. It’s recordings all the way down.

Fernando (00:53:05): You mentioned something about Rails. Is this pattern in Rails somehow? Or part of Rails?

Jeff (00:53:14): Yeah, it’s part of Rails. So it started life in Basecamp as, you know, just in our application code. And then once we realized how good it was, David packaged it up and wrote documentation for it and tests and extracted it to Rails. That’s another reason we know it works. It sounds very abstract, but it’s not ivory tower sort of theory thing. This is what we’re actually using and we’ve proven it out over a decade. So it really, really works. And it’s replaced our use of single table inheritance and even polymorphic relationships. We almost don’t… when you’re going to reach for one of those, you should give delegated type a look and be like, could I do the same thing with the delegated type? Because you get the same sort of sharing, but I think that it’s sort of inverted, right? It’s particularly an inversion of the polymorphic type. In a polymorphic pattern, you have a record that can belong to any kind of record right?

(00:54:16): And in the delegated type pattern, it’s the other way. It’s like, so it’s not your child record that can belong to any kind of thing. It’s the parent record that can have any kind of thing, but it means you get to work with those parents. You’re starting at the top because if you think about something like a comment can have any kind of commentable, when are you ever working from the comment? When do you want to start with the comment table and find all the things? No, it’s at the wrong end of the relationship. You want to start at the top or that’s why the tree pattern works really well too. So I think that it just needs more people once they try it, would realize this is really cool and this can change a lot.

Kimberly (00:55:02): Okay. Jeff, I have a question for you. So you’re clearly an expert at all this recordings, recordables, all of the things. What would you say, if you had to give someone advice, like someone who is using this, what do you think are some of the biggest mistakes people have? I know I didn’t prep you for that question.

Jeff (00:55:22): No, it’s okay. I don’t know if there’s any mistakes so much as… maybe the mistake is not taking it far enough. You know what I mean? You can model all the things that you have in common in this way and you can go pretty far with it. We did Hill Charts in Basecamp, like how do you model those as recordings and recordables? But there’s a way, and you need other patterns. Or how do you take, lilke you want document history or event history. The event stream is another thing. Not only do we want to show you events, but we want the events that we’re listing to be accurate. So you have a timeline and it’s like, Kimberly saved a new document called this. Well, then you went and renamed it, but when you created it, it had the original name. And so when I’m scrolling back, I’m like, Kimberly didn’t name this…

(00:56:16): It says new features March, but I’m reading this, I’ve scrolled back to February. That doesn’t make sense. Does she have a time machine? No. You want accurate history and that’s another way to get it is with this. So I think it’s combining the delegate pattern, the delegate type pattern with other patterns and they really can build off of each other. So once you have this concept of the type doesn’t matter, you can do interesting things with the type. Like, don’t change them, keep a log of them, right, so that you can easily move between them. Move pointers around. It gives you ideas for how you could copy and move things efficiently. So I think that people don’t realize where, it’s not that they use it wrong, it’s that they don’t realize in how many places it’s actually applicable. Not nearly as rigid as STI or polymorphism. It’s super, super flexible.

Fernando (00:57:11): Were you a part of the…. or was this David’s invention?

Jeff (00:57:17): It was a big part of David… David came up with the original idea and it was out of the frustration of how expensive and slow it had gotten to copy things in Basecamp 2, and how every time you introduced a new thing, you also had to introduce changes to the copier. And so yeah, David had the first idea for this. Typical David thing, like how can I invent a brand new pattern that nobody is using? That’s why I think it originated here. And it’s sort of like it has a lot in common with STI, but it’s flipped on its head. And I think also a thing that only David can do is that it was unproven when he introduced it. It was like, eh, this might work. It’s harder to get away with the speculative, let’s give it a shot, unless you’re David, but it’s David and he can just do that.

(00:58:11): And then it was like, let’s see how this works. Because at the time, we hadn’t been designing Basecamp like that. It was very Basecamp 2 like, and then we just, from there, it became a shared thing we could iterate on. And we didn’t extract it for several years after it was in use, maybe a year or two years after it was in use in Basecamp before it proved itself to be a viable pattern. It went through a bunch of iterations. But yeah, the kernel of that idea that you have one row of one table that holds most of the information about the records, but none of the specifics came from David motivated by copying efficiency and the need to paginate, the need to have a timeline of different types that we could paginate with one query. Because the other way to do something like that is to sort of create a table of copies that you insert different records in the table that you’re going to paginate, but now you have a copying system. And we do have a version of that in Basecamp, but it’s a performance optimization. It’s not the main way that we query for recordings. Think of it like caching. We can relay events into another table so that we can very efficiently be like, here’s all the events that apply to just this person. Everybody can see different things in Basecamp. My timeline doesn’t look exactly like your timeline because we’re on different projects.

Fernando (00:59:42): No, I could see how that would be a nightmare to implement if you had different types. I can imagine it in my head like a PM going like, okay, I want you to give me a timeline of things that are happening and the programmers going, oof! That’s going to be rough.

Jeff (01:00:01): And you pay that cost every time you add a new thing. And even when we’re doing work in beta, that’s the one time… so we do work in beta. We’ll modify the database, we’ll add new types and production needs to know how to deal with these new types. In most cases, it just can, because we’ve programmed it in a way that if they don’t know how to deal with a particular recordable, it just deals with it in a generic way. It displays the generic icon. It asks for things that it knows that all recordables have — a title, possibly content, subscribers. And so nothing breaks.

Fernando (01:00:41): Nothing breaks

Jeff (01:00:42): Yeah.

(01:00:43): And so that was another key element of the design. How can we have something that’s flexible? What we found with Basecamp 2 is that, yeah, it works, but it’s slow. And every time we’re like, hey, we want to add a new thing, we’re like, eh, that’s expensive. That’s going to be a pain in the ass. How do you make it not a pain in the ass? And I think good patterns emerge when that’s your criteria. We come up with something to make it easier. What would make this easier? And then so you find the specific problems we had and then correct for those. And David is basically an expert at that.

Fernando (01:01:17): And he hates when things break.

Jeff (01:01:19): And he hates friction…

Fernando (01:01:21): He does, yeah.

Jeff (01:01:22): When the common thing is hard, the common thing should be easy. The common thing we do is introduce new features, new kinds of content. The common thing we do is, and with those content, that content, we need it to work on our mobile apps with no extra effort without having to hire new mobile developers. This is why we have Hotwire and why we have Hotwire native, why we have Rails. Make these super common things easy. The uncommon things just need to be possible. You don’t do them that often. It’s okay to have friction there. Look at deploying apps, right? Everyone was deploying apps with Docker before using Docker and production before we were, but David looks at it and is like, this is a pain. Why is it like this? And so he even invents Kamal. It’s that response. He has a low tolerance for friction on the critical path, and I think people just, they become oblivious to that friction. They sort of ramp up slowly with it and they’re like, this is just the way it is. David never accepts that this is just the way it is. He’s got a real eye for like, this could be different. And he’s not just going to complain about it. He’ll do something. He’ll disappear for a week or two and then be like, here. I fixed it. It’s great.

Kimberly (01:02:47): Jeff, you mentioned some documentation, some Rails documentation over delegated types. We’ll link to that in the notes for this episode. Are there any other places people should go for information about all of this?

Jeff (01:02:58): I mean, right now there’s not a ton of places. There are a few people who have written articles that are pretty good, and they’re sort of the exploratory kind of article. They’re like, hey, you know, I haven’t found a lot of documentation about this cool feature of Rails. And so they’ve written about it, so that’s really good. The Rails documentation, while terse, is also very good, and I think that what we’re doing right now is adding to the growing body of documentation on this pattern. And what David’s going to do is also going to help that. But yeah, if I can find any, I’ll send you the links and we can add that to the show notes because yeah, I think that’s the main thing we’re trying to correct by doing an episode like this is to share the benefits of this underrepresented pattern.

Kimberly (01:03:45): Awesome. Well, with that, we’re going to wrap it up. This has been a production of Recordables by the 37signals team. To hear more from our technical team about their behind the scenes work, visit our developer’s blog at dev.37signals.com.

• Good HTML semantics. Paragraphs are real <p> tags, as they should be.
• Markdown support: shortcuts, auto-formatting on paste.
• Real-time code syntax highlighting.
• Create links by pasting URLs on selected text.
• Configurable prompts. Support for mentions and other interactive prompts with multiple loading and filtering strategies.
• Preview attachments like PDFs and Videos in the editor.
• Works seamlessly with Action Text and Active Storage.

We created Lexxy because Trix was falling short of expectations in certain areas, and we encountered technical barriers when attempting to offer the experience we wanted. Lexxy comes with a bunch of juicy improvements, but more than that, we now have a fantastic foundation to build on top of.

Lexxy will also bring a great improvement to Action Text: we will let you configure the editor in Action Text just like you configure the database in Active Record. This will open the door to integrating other editors in Rails.

Text editing is central to our products. We believe Lexxy will let us deliver the editing experience we want. We are launching an early beta today, give it a try!

Like many organizations, we’d been using external container registries for years. Our ecosystem was tightly coupled to both Dockerhub and Amazon’s Elastic Container Registry.

However, as part of our cloud exit and kamalization journey, several issues started emerging:

• Cost: Not only does the paid license for Dockerhub produce a considerable invoice — pulling and pushing our images over the internet dozens of times a day caused us to hit the contracted bandwidth limit with our datacenter provider Deft repeatedly. We tried working around this by running pull-through caches, but this still locked us to Dockerhub.
• Performance: Migrating HEY to Kamal and expanding the deployment to another continent caused deploy time penalties — up to 45 seconds on uncached pulls per host. This was exacerbated once our largest application Basecamp 4 was moved to Kamal — suddenly deployments took minutes longer simply because of push/pull speeds out of our control.
• Security and Governance: We all hope to never leak credentials in our images, and yet it still happens — the scale ranging from easily mitigated to catastrophic. We wanted to eliminate that threat surface once and for all by keeping our artifacts where they belong — with us.
• Independence: Despite being on a paid account, we fell into the crunch of API limitations for arbitrary reasons a couple of times. In addition, we’d been keeping all of the images used in our Chef CI/CD infrastructure still on AWS.

Our criteria for the solution to pick were fairly simple: reliable, performant, easy to set up, open-source.

We evaluated running the default distribution implementation as our registry, but quickly set our eyes on Harbor. Harbor provided us with a more expandable and rich feature set right out of the box, and required minimal extra tooling to make it robust and scalable.

Setting up Harbor

Harbor’s deployment is optimized for using it within Kubernetes environments, but the single-server setup using the pre-packaged docker-compose configuration proved to be exactly what we were looking for.

We had three key points to cover in our plan for the v1 of our on-premise registry:

• Use our own S3 storage.
• Make sure we have at least two replicating sites that can be easily failed over.
• Keep the storage footprint as small as possible by enabling retention policies.

Configuring S3 storage

At 37signals, we’re running our own Pure FlashBlade storage cluster providing us with S3 object storage right out of the box, but for Harbor, any S3-compatible backend will do.

The configuration in Harbor was easy, but it was crucial to get the permissions set right on the Pure backend. You can obviously run an s3:* policy, but let’s be real, we want to do better! After some trial and error with broken image pushes, these are the minimal permissions needed on the bucket to operate Harbor with a custom S3 backend:

s3:AbortMultipartUpload
s3:DeleteObject
s3:GetBucketLocation
s3:GetObject
s3:ListBucket
s3:ListBucketMultipartUploads
s3:ListMultipartUploadParts
s3:PutObject

Configuring multiple instances

For the v1 of the Harbor deployment, we opted to run two stand-alone instances at first: one in our Ashburn and one in our Chicago location.

Harbor comes with several components, such as PostgreSQL and Redis services, handling manifest/user management and job scheduling. We explored an elaborate HA per datacenter with colocated instances of those services, but decided to wait for the first results of the all-in-one stand-alone deployment before making it more complicated than it has to be.

This is an excerpt of our harbor.yml in use, which gives you a functional instance, including the s3 configuration and enabled monitoring:

hostname: "#{node['fqdn']}"
http:
  port: 80
data_volume: /data
harbor_admin_password: "#{admin_password}"
storage_service:
  s3:
    bucket: docker-registry-bucket
    accesskey: "#{bucket_credentials['access_key']}"
    secretkey: "#{bucket_credentials['secret_key']}"
    regionendpoint: "https://purestorage.#{node["domain"]}"
    region: us-east-1
    encrypt: false
    secure: true
    v4auth: true
    chunksize: 5242880
    loglevel: debug
metric:
  enabled: true
  port: 9090
  path: /metrics
database:
  password: "#{db_password}"
  max_idle_conns: 50
  max_open_conns: 100
clair:
  updaters_interval: 12
jobservice:
  max_job_workers: 20
  job_loggers:
    - FILE
  logger_sweeper_duration: 3600
log:
  level: info
  local:
    rotate_count: 50
    rotate_size: 200M
    location: /var/log/harbor
notification:
  webhook_job_max_retry: 3
  webhook_job_http_client_timeout: 10
_version: 2.10.0

As you can see, it is a fairly default config. Encapsulated in a Chef recipe, this will be executed on the respective nodes in each DC, setting the correct FQDN and pointing to the correct storage endpoint. These nodes are then fronted by our F5 loadbalancers for SSL termination and region-specific domains.

Each Harbor node is currently a virtual machine equipped with 64GB of RAM, 32 vCPU and 320GB of storage.

Configuring replication

The initial Chef setup only needs to run once for bootstrapping. For further configuration we decided to rely on the terraform provider for Harbor. In addition to the initial user management setup, here we could also configure replication between the endpoints easily. We decided on a two-way replication scheme to keep it all in sync, inspired by this setup.

• Images are pushed to a registry endpoint.
• The endpoint pulls data from the opposite registry every 10 minutes.

resource "harbor_replication" "replication_push_sc_chi" {
  provider               = harbor.sc-chi
  name                   = "Replicate images on push to df-iad"
  action                 = "push"
  registry_id            = harbor_registry.df_iad.registry_id
  schedule               = "event_based"
  dest_namespace_replace = -1
  filters {
    name = "**"
  }
  filters {
    tag = "**"
  }
}

resource "harbor_replication" "replication_pull_sc_chi" {
  provider               = harbor.sc-chi
  name                   = "Replicate missing images/artifacts from df-iad"
  action                 = "pull"
  registry_id            = harbor_registry.df_iad.registry_id
  schedule               = "0 0/10 * * * *"
  enabled                = false
  dest_namespace_replace = -1
  filters {
    name = "**"
  }
  filters {
    tag = "**"
  }
}

In addition, we’re replicating the underlying S3 buckets directly on our Pure cluster for an extra failsafe backup mechanism. It’s important to note however that this is not enough to make Harbor aware of the data in the other location — explicit replication on the Harbor level like the setup above must be configured.

Syncing the catalogue

You could of course start with an empty registry and fill it as you go, but this isn’t very feasible if you want a drop-in replacement for your current registry. In our case, we had to make sure that the entire image catalog from Dockerhub gets copied into Harbor — the challenge being that this meant dealing with 80+ individual image repositories. Thankfully, Harbor offers replication directly from Dockerhub, so we opted for that.

Sounds straightforward? Here’s a funny caveat — depending on the amount of repositories you want to fetch and replicate, Dockerhub is likely going to throttle you on the API level if you want to do this all at once. You could totally write a functional replication rule that just targets **/**, only to be showered with 429 — even on a paid account.

Thus, the replication has to happen in batches. For this, we chose to create individual replication rules per repository with a “manual” (also scripted) trigger to avoid overloading the API.

The definition for the replication rules in Terraform:

variable "repositories" {
  type = map(string)
}

resource "harbor_registry" "dockerhub" {
  provider      = harbor.sc-chi
  provider_name = "docker-hub"
  name          = "DockerHub"
  endpoint_url  = "https://registry-1.docker.io"
  description   = "Endpoint for replicating the existing catalogue"
  access_id     = var.dockerhub_username
  access_secret = var.dockerhub_password
}

resource "harbor_replication" "dockerhub_mirror" {
  for_each = var.repositories

  provider               = harbor.sc-chi
  name                   = "mirror-dockerhub-${each.key}"
  description            = "Replicate and mirror images from DockerHub"
  registry_id            = harbor_registry.dockerhub.registry_id
  dest_namespace         = "yourorg"
  override               = true
  dest_namespace_replace = 1
  copy_by_chunk          = true

  filters {
    name = "yourorg/${each.key}"
  }

  filters {
    tag = "**"
  }

  action = "pull"
}

The script to pull out a list of repositories from Dockerhub, make it accessible to Terraform and create the individual replication rules per repository:

#!/usr/bin/env bash

set -euo pipefail

# -----------------------------
# Config
# -----------------------------
DOCKERHUB_USER="${DOCKERHUB_USER:-your-dockerhub-username}"
DOCKERHUB_PASSWORD="${DOCKERHUB_PASSWORD:-your-dockerhub-password}"
DOCKERHUB_ORG="yourorg"
TFVARS_DIR="./generated_tfvars"
TFVARS_FILE="$TFVARS_DIR/all_repos.tfvars.json"

mkdir -p "$TFVARS_DIR"

# -----------------------------
# Authentication
# -----------------------------
echo "🔐 Getting Docker Hub token..."
TOKEN=$(curl -s -X POST https://hub.docker.com/v2/users/login/ \
  -H "Content-Type: application/json" \
  -d '{"username": "'"$DOCKERHUB_USER"'", "password": "'"$DOCKERHUB_PASSWORD"'"}' |
  jq -r .token)

if [[ "$TOKEN" == "null" || -z "$TOKEN" ]]; then
  echo "❌ Failed to authenticate. Check Docker Hub credentials."
  exit 1
fi

export AUTH_HEADER="Authorization: Bearer $TOKEN"

# -----------------------------
# Helper Functions
# -----------------------------
fetch_repos_starting_with() {
  local letter="$1"
  local page=1
  local repos=()

  while :; do
    local url="https://hub.docker.com/v2/repositories/${DOCKERHUB_ORG}/?page=$page&page_size=100"
    local response=$(curl -s -H "$AUTH_HEADER" "$url")

    local matched=$(echo "$response" |
      jq -r ".results[] | select(.name | startswith(\"$letter\")) | .name")

    repos+=($matched)

    local next=$(echo "$response" | jq -r ".next")
    [[ "$next" == "null" ]] && break
    ((page++))
  done

  echo "${repos[@]}"
}

generate_tfvars_file() {
  local repos=("$@")
  local tfvars_file="$TFVARS_FILE"

  echo "{ \"repositories\": {" > "$tfvars_file"
  for repo in "${repos[@]}"; do
    echo "  \"$repo\": \"$repo\"," >> "$tfvars_file"
  done
  sed -i '' '$ s/,$//' "$tfvars_file"
  echo "} }" >> "$tfvars_file"

  >&2 echo "💾 Created tfvars file: $tfvars_file"
  >&2 ls -l "$tfvars_file"

  # Only echo the filename to stdout
  echo "$tfvars_file"
}

run_terraform_once() {
  local tfvars="$1"

  local abs_tfvars
  abs_tfvars=$(realpath "$tfvars")

  echo "🔁 Applying Terraform with $abs_tfvars"

  (
    cd harbor-production || exit 1
    local rel_tfvars
    rel_tfvars=$(python3 -c "import os.path; print(os.path.relpath('$abs_tfvars', '.'))")

    terraform apply -var-file="$rel_tfvars" -auto-approve
  )
}

# -----------------------------
# Execution
# -----------------------------
main() {
  echo "🚀 Starting repository sync..."
  letters=(a b c d e f g h i j k l m n o p q r s t u v w x y z)
  all_repos=()

  for letter in "${letters[@]}"; do
    echo "📦 Fetching repos for prefix: $letter"
    repos=($(fetch_repos_starting_with "$letter"))
    all_repos+=("${repos[@]}")
  done

  if [[ "${#all_repos[@]}" -eq 0 ]]; then
    echo "⚠️ No repositories found."
    exit 0
  fi

  tfvars_file=$(generate_tfvars_file "${all_repos[@]}" 2>/dev/null)
  run_terraform_once "$tfvars_file"

  echo "✅ All repositories synced."
}

main "$@"

And the script to enable those rules alphabetically in batches:

#!/usr/bin/env bash
set -euo pipefail

# -----------------------------
# Config
# -----------------------------
HARBOR_URL="${HARBOR_URL:-https://registry.yourdomain.com}"
HARBOR_USER="${HARBOR_USER:-your-harbor-user}"
HARBOR_PASSWORD="${HARBOR_PASSWORD:-your-harbor-password}"
BATCH_DELAY=10  # seconds between batches
PREFIX="mirror-dockerhub-"

# -----------------------------
# CLI Args
# -----------------------------
DRY_RUN=false
RANGE_START="a"
RANGE_END="z"

usage() {
  cat <<EOF
Usage: $0 [--dry-run] [--range <start-end>]

Options:
  --dry-run         Only print the rules that would be triggered, no API calls.
  --range a-d       Trigger only rules whose names start with '${PREFIX}' plus letter in <start-end>.
                    Example: --range a-d
EOF
  exit 1
}

while [[ $# -gt 0 ]]; do
  case "$1" in
    --dry-run)
      DRY_RUN=true
      shift
      ;;
    --range)
      if [[ "$2" =~ ^[a-zA-Z]-[a-zA-Z]$ ]]; then
        RANGE_START=$(echo "$2" | cut -d- -f1 | tr '[:upper:]' '[:lower:]')
        RANGE_END=$(echo "$2" | cut -d- -f2 | tr '[:upper:]' '[:lower:]')
        shift 2
      else
        echo "Invalid range format. Expected like 'a-d'."
        usage
      fi
      ;;
    *)
      echo "Unknown argument: $1"
      usage
      ;;
  esac
done

if [[ "$RANGE_START" > "$RANGE_END" ]]; then
  echo "Invalid range: start ($RANGE_START) > end ($RANGE_END)"
  exit 1
fi

# -----------------------------
# Auth & Token
# -----------------------------
echo "🔐 Authenticating with Harbor..."
AUTH_HEADER="Authorization: Basic $(echo -n "$HARBOR_USER:$HARBOR_PASSWORD" | base64)"

curl -s -H "$AUTH_HEADER" "$HARBOR_URL/api/v2.0/users/current" | jq -e .username > /dev/null || {
  echo "❌ Harbor auth failed"
  exit 1
}

echo "📋 Fetching all replication rules..."
rules=$(curl -s -H "$AUTH_HEADER" "$HARBOR_URL/api/v2.0/replication/policies?page_size=100")

declare -A letter_to_ids

while IFS= read -r rule; do
  id=$(echo "$rule" | jq -r '.id')
  name=$(echo "$rule" | jq -r '.name')

  # Filter by prefix first
  if [[ "$name" == "$PREFIX"* ]]; then
    # Get letter after the prefix
    suffix_letter=$(echo "${name#$PREFIX}" | cut -c1 | tr '[:upper:]' '[:lower:]')
    if [[ "$suffix_letter" < "$RANGE_START" || "$suffix_letter" > "$RANGE_END" ]]; then
      continue
    fi
    letter_to_ids["$suffix_letter"]+="$id "
  fi
done < <(echo "$rules" | jq -c '.[]')

echo "✅ Loaded replication rules matching prefix '$PREFIX'."

increment_letter() {
  local c=$1
  printf "\\$(printf '%03o' "$(( $(printf '%d' "'$c") + 1 ))")"
}

current="$RANGE_START"
while [[ $current < $RANGE_END || $current == $RANGE_END ]]; do
  ids=${letter_to_ids[$current]:-}
  if [[ -n "$ids" ]]; then
    echo "🚀 Processing prefix '$PREFIX$current' with rule IDs: $ids"
    for id in $ids; do
      if $DRY_RUN; then
        echo "   (dry-run) Would trigger rule ID: $id"
      else
        echo "   🔁 Triggering rule ID: $id"
        curl -s -X POST -H "$AUTH_HEADER" \
          -H "Content-Type: application/json" \
          -d "{\"policy_id\": $id}" \
          "$HARBOR_URL/api/v2.0/replication/executions" > /dev/null
      fi
    done
    if ! $DRY_RUN; then
      echo "⏳ Waiting $BATCH_DELAY seconds before next batch..."
      sleep "$BATCH_DELAY"
    fi
  fi
  current=$(increment_letter "$current")
done

echo "✅ Done."

The progress of all replication tasks at the same time is quite hard to monitor within Harbor’s UI (despite excellent logging). Thus, another small script helped summarize this:

#!/usr/bin/env bash

HARBOR_URL="${HARBOR_URL:-https://registry.yourdomain.com}"
HARBOR_USER="${HARBOR_USER:-your-harbor-user}"
HARBOR_PASSWORD="${HARBOR_PASSWORD:-your-harbor-password}"

# Check required env vars
if [[ -z "$HARBOR_USERNAME" || -z "$HARBOR_PASSWORD" || -z "$HARBOR_URL" ]]; then
  echo "❌ Missing HARBOR_USERNAME, HARBOR_PASSWORD or HARBOR_URL. Set them as env vars."
  exit 1
fi

# -----------------------------
# Auth
# -----------------------------
echo "🔐 Authenticating with Harbor..."
pong=$(curl -s -u "$HARBOR_USERNAME:$HARBOR_PASSWORD" "$HARBOR_URL/ping")
if [ "$pong" != "Pong" ]; then
  echo "❌ Authentication failed. Harbor did not return expected 'Pong'."
  echo "Response: $pong"
  exit 1
fi
echo "✅ Auth successful."

# -----------------------------
# Fetch executions
# -----------------------------
echo "📋 Fetching replication executions..."
executions=$(curl -s -u "$HARBOR_USERNAME:$HARBOR_PASSWORD" "$HARBOR_URL/replication/executions?page_size=100")

# Cache for policy ID to name mapping
declare -A POLICY_NAMES
get_policy_name() {
  local policy_id="$1"
  if [[ -n "${POLICY_NAMES[$policy_id]}" ]]; then
    echo "${POLICY_NAMES[$policy_id]}"
  else
    local name=$(curl -s -u "$HARBOR_USERNAME:$HARBOR_PASSWORD" "$HARBOR_URL/replication/policies/$policy_id" | jq -r '.name // "unknown"')
    POLICY_NAMES[$policy_id]="$name"
    echo "$name"
  fi
}

# -----------------------------
# Build table
# -----------------------------
rows=()
while IFS= read -r exec; do
  id=$(echo "$exec" | jq -r '.id')
  policy_id=$(echo "$exec" | jq -r '.policy_id')
  status=$(echo "$exec" | jq -r '.status')
  start_time=$(echo "$exec" | jq -r '.start_time' | sed 's/\.[0-9]*Z$/Z/')

  start_epoch=$(date -u -d "$start_time" +%s 2>/dev/null)
  now_epoch=$(date +%s)
  runtime_min=$(( (now_epoch - start_epoch) / 60 ))

  policy_name=$(get_policy_name "$policy_id")

  row=$(printf "%-8s %-10s %-25s %-12s %-14s %s" "$id" "$policy_id" "$policy_name" "$status" "$runtime_min" "$start_time")
  rows+=("$runtime_min $row")
done < <(echo "$executions" | jq -c '.[] | select(.status == "InProgress")')

echo
echo "🟡 In-progress replication tasks (sorted by runtime):"
printf "%-8s %-10s %-25s %-12s %-14s %s\n" "ID" "Policy_ID" "Rule Name" "Status" "Runtime(min)" "Start Time"

# -----------------------------
# Print
# -----------------------------
for line in "${rows[@]}"; do
  echo "$line"
done | sort -rn | cut -d' ' -f2-

After enabling those rules in batches, it’s also crucial to make sure enough job worker resources are available to sufficiently speed up this process.

Analyzing performance

After migrating all our Kamal-ized apps to push and pull from our new on-premise registry, it was finally time to actually get some numbers on performance in. We grabbed this data directly from the deployment logs printed by Kamal.

You can use these quick one-liners for extracting the pull times from the Kamal log on your terminal:

# Mac
pbpaste | sort | sed 's/  INFO //' | grep -E "Running docker pull" -B1 | grep Finished | awk '{print $4}' | sort | tr '\n' ' '
# Linux
xclip -o | sort | sed 's/  INFO //' | grep -E "Running docker pull" -B1 | grep Finished | awk '{print $4}' | sort | tr '\n' ' '

Or from a logfile:

awk '
/Running .*docker pull/ {
  if (match($0, /\[([a-f0-9]+)\]/, m)) {
    id = m[1]
    running[id] = $0
  }
}
/Finished/ {
  if (match($0, /\[([a-f0-9]+)\]/, m)) {
    id = m[1]
    if (id in running) {
      print running[id]
      print $0
      print ""
      delete running[id]
    }
  }
}
' /path/to/kamal.log

After analyzing the numbers, we were quite happy to see that:

• The overall image pull timings on our fleet decreased by up to 25 seconds for HEY, Basecamp 4 and Basecamp 2 (our three largest apps), with the lion’s share of improvement on our HEY nodes on the Amsterdam outposts.
• Deploy times decreased by 15 seconds for HEY.

In addition, it allowed us to:

• Retire the Dockerhub cache setup, further detangling our infrastructure.
• Implement proper retention policies and garbage collection to decrease the overall storage quota from almost 9 TiB to 1.5 TiB.
• Save roughly $5k/year on subscription fees going forward.

Remember: this is basically a single-node infrastructure, with the primary endpoint being in Chicago, and the Ashburn site providing the backup. We found that this small setup has been reliable for roughly two months now. During this time, Harbor has served more than 32,000 pulls under company-wide use in day-to-day business.

Conclusion

This project proved to us that it’s — again — worth considering a departure from large SaaS offerings and public cloud providers. We’ve been dependent on external registries keeping our app images for years, but the simplicity and benefits of our current setup give little reason to doubt that cutting the cord was the right decision: better performance at less cost with minimal infrastructure.

Note: Shortly after releasing this gem, we renamed it from Action Native Push to Action Push Native, in case you arrived here looking for the gem of the former name. More details here.

We’ve open-sourced Action Push Native, a Rails gem for sending push notifications to mobile platforms. It supports both Apple and Google push notification services.

Why did we build it?

We created it to migrate off Amazon SNS and Pinpoint, as part of our broader cloud exit. We’re using it in Basecamp and HEY to send more than 10 million push notifications per day without a hitch.

Action Push Native relies on HTTP/2 persistent connections to the Apple Push Notification service, which significantly reduced job duration compared to our previous HTTP/1 setup with AWS Pinpoint:

AWS Pinpoint jobs duration

Action Push Native jobs duration

How does it work?

The gem connects directly to the Apple (APNs) and Google (FCM) push notification services. It handles retries, rate-limiting, and deleting dead devices automatically. Configure each platform with your credentials, and you can start sending notifications like this:

device = ApplicationPushDevice.create! \
  name: "iPhone 16",
  token: "6c267f26b173cd9595ae2f6702b1ab560371a60e7c8a9e27419bd0fa4a42e58f",
  platform: "apple"

notification = ApplicationPushNotification.new \
  title: "Hello world!",
  body:  "Welcome to Action Push Native"

notification.deliver_later_to(device)

Version 0.1.0 is available now. You can read more on GitHub.

We hope you find it useful!

A web-first framework for building native mobile apps

Improvements

There are a few significant changes in v1.2 that are worth specifically highlighting.

Route decision handlers

Hotwire Native apps route internal urls to screens in your app, and route external urls to the device’s browser. Historically, though, it wasn’t straightforward to customize the default behavior for unique app needs.

In v1.2, we’ve introduced the RouteDecisionHandler concept to iOS (formerly only on Android). Route decisions handlers offer a flexible way to decide how to route urls in your app. Out-of-the-box, Hotwire Native registers these route decision handlers to control how urls are routed:

• AppNavigationRouteDecisionHandler: Routes all internal urls on your app’s domain through your app.
• SafariViewControllerRouteDecisionHandler: (iOS Only) Routes all external http/https urls to a SFSafariViewController in your app.
• BrowserTabRouteDecisionHandler: (Android Only) Routes all external http/https urls to a Custom Tab in your app.
• SystemNavigationRouteDecisionHandler: Routes all remaining external urls (such as sms: or mailto:) through device’s system navigation.

If you’d like to customize this behavior you can register your own RouteDecisionHandler implementations in your app. See the documentation for details.

Server-driven historical location urls

If you’re using Ruby on Rails, the turbo-rails gem provides the following historical location routes. You can use these to manipulate the navigation stack in Hotwire Native apps.

• recede_or_redirect_to(url, **options) — Pops the visible screen off of the navigation stack.
• refresh_or_redirect_to(url, **options) — Refreshes the visible screen on the navigation stack.
• resume_or_redirect_to(url, **options) — Resumes the visible screen on the navigation stack with no further action.

In v1.2 there is now built-in support to handle these “command” urls with no additional path configuration setup necessary. We’ve also made improvements so they handle dismissing modal screens automatically. See the documentation for details.

Bottom tabs

When starting with Hotwire Native, one of the most common questions developers ask is how to support native bottom tab navigation in their apps. We finally have an official answer! We’ve introduced a HotwireTabBarController for iOS and a HotwireBottomNavigationController for Android. And we’ve updated the demo apps for both platforms to show you exactly how to set them up.

New demo apps

To better show off all the features in Hotwire Native, we’ve created new demo apps for iOS and Android. And there’s a brand new Rails web app for the native apps to leverage.

Hotwire Native demo app

Clone the GitHub repos to build and run the demo apps to try them out:

• iOS repo
• Android repo
• Rails app

Huge thanks to Joe Masilotti for all the demo app improvements. If you’re looking for more resources, Joe even wrote a Hotwire Native for Rails Developers book!

Release notes

v1.2 contains dozens of other improvements and bug fixes across both platforms. See the full release notes to learn about all the additional changes:

• iOS release notes
• Android release notes

Take a look

If you’ve been curious about using Hotwire Native for your mobile apps, now is a great time to take a look. We have documentation and guides available on native.hotwired.dev and we’ve created really great demo apps for iOS and Android to help you get started.

We store different kinds of information on S3, from the attachments customers upload to Basecamp to the Prometheus long-term metrics. On top of that, Pure’s system also provides filesystem-based capabilities, enabling other relevant usages, such as database backup storage. This makes the system a top priority for observability.

Although the system has great reliability, out-of-the-box internal alerting, and autonomous ticket creation, it would also be good to have our metrics and alerts to facilitate problem-solving and ensure any disruptions are prioritized and handled.

For more context on our current Prometheus setup, see how we use Prometheus at 37signals.

Pure OpenMetrics exporter

Pure maintains two OpenMetrics exporters, pure-fb-openmetrics-exporter and pure-fa-openmetrics-exporter. Since we use Pure FlashBlade (fb), this post covers pure-fb-openmetrics-exporter, although overall usage should be similar. The setup is straightforward and requires only binary and basic authentication installation.

Here is a snippet of our Chef recipe that installs it:

pure_api_token = "token" # If you use Chef, your token should come from an encrypted databag. Changed to hardcoded here to simplify
PURE_EXPORTER_VERSION = "1.0.13".freeze # Generally, we use Chef node metadata for version management. Changed to hardcoded to simplify

directory "/opt/pure_exporter/#{PURE_EXPORTER_VERSION}" do
 recursive true
 owner 'pure_exporter'
 group 'pure_exporter'
end

# Avoid recreating under /tmp after reboot if target_binary is already there
target_binary = "/opt/pure_exporter/#{PURE_EXPORTER_VERSION}/pure-fb-openmetrics-exporter"

remote_file "/tmp/pure-fb-openmetrics-exporter-v#{PURE_EXPORTER_VERSION}-linux-amd64.tar.gz" do
 source "https://github.com/PureStorage-OpenConnect/pure-fb-openmetrics-exporter/releases/download/v#{PURE_EXPORTER_VERSION}/pure-fb-openmetrics-exporter-v#{PURE_EXPORTER_VERSION}-linux-amd64.tar.gz"
 not_if { ::File.exist?(target_binary) }
end

archive_file "/tmp/pure-fb-openmetrics-exporter-v#{PURE_EXPORTER_VERSION}-linux-amd64.tar.gz" do
 destination "/tmp/pure-fb-openmetrics-exporter-v#{PURE_EXPORTER_VERSION}"
 action :extract
 not_if { ::File.exist?(target_binary) }
end

execute "copy binary" do
 command "sudo cp /tmp/pure-fb-openmetrics-exporter-v#{PURE_EXPORTER_VERSION}/pure-fb-openmetrics-exporter /opt/pure_exporter/#{PURE_EXPORTER_VERSION}/pure-exporter"
 creates "/opt/pure_exporter/#{PURE_EXPORTER_VERSION}/pure-exporter"
 not_if { ::File.exist?(target_binary) }
end

tokens = <<EOF
main:
 address: purestorage-mgmt.mydomain.com
 api_token: #{pure_api_token['token']}
EOF

file "/opt/pure_exporter/tokens.yml" do
 content tokens
 owner 'pure_exporter'
 group 'pure_exporter'
 sensitive true
end

systemd_unit 'pure-exporter.service' do
 content <<-EOU
 # Caution: Chef managed content. This is a file resource from #{cookbook_name}::#{recipe_name}
 #

 [Unit]
 Description=Pure Exporter
 After=network.target

 [Service]
 Restart=on-failure
 PIDFile=/var/run/pure-exporter.pid
 User=pure_exporter
 Group=pure_exporter
 ExecStart=/opt/pure_exporter/#{PURE_EXPORTER_VERSION}/pure-exporter \
 --tokens=/opt/pure_exporter/tokens.yml

 ExecReload=/bin/kill -HUP $MAINPID
 SyslogIdentifier=pure-exporter

 [Install]
 WantedBy=multi-user.target
 EOU

 action [ :create, :enable, :start ]
 notifies :reload, "service[pure-exporter]"
end

service 'pure-exporter'

Prometheus Job Configuration

The simplest way of ingesting the metrics is to configure a basic Job without any customization:

- job_name: pure_exporter
  metrics_path: /metrics
  static_configs:
  - targets: ['<%= @hostname %>:9491']
    labels:
      environment: 'production'
      job: pure_exporter
  params:
    endpoint: [main] # From the tokens configuration above

For a production-ready setup, we are using a slightly different approach. The exporter supports the usage of specific metric paths to allow for split Prometheus jobs configuration that reduces the overhead of pulling the metrics all at once:

- job_name: pure_exporter_array
  metrics_path: /metrics/array
  static_configs:
  - targets: ['<%= @hostname %>:9491']
    labels:
      environment: 'production'
      job: pure_exporter
  metric_relabel_configs:
  - source_labels: [name]
    target_label: ch
    regex: "([^.]+).*"
    replacement: "$1"
    action: replace
  - source_labels: [name]
    target_label: fb
    regex: "[^.]+\\.([^.]+).*"
    replacement: "$1"
    action: replace
  - source_labels: [name]
    target_label: bay
    regex: "[^.]+\\.[^.]+\\.([^.]+)"
    replacement: "$1"
    action: replace
  params:
    endpoint: [main] # From the tokens configuration above
- job_name: pure_exporter_clients
  metrics_path: /metrics/clients
  static_configs:
  - targets: ['<%= @hostname %>:9491']
    labels:
      environment: 'production'
      job: pure_exporter
  params:
    endpoint: [main] # From the tokens configuration above
- job_name: pure_exporter_usage
  metrics_path: /metrics/usage
  static_configs:
  - targets: ['<%= @hostname %>:9491']
    labels:
      environment: 'production'
      job: pure_exporter
  params:
    endpoint: [main]
- job_name: pure_exporter_policies
  metrics_path: /metrics/policies
  static_configs:
  - targets: ['<%= @hostname %>:9491']
    labels:
      environment: 'production'
      job: pure_exporter
  params:
    endpoint: [main] # From the tokens configuration above

We also configure some metric_relabel_configs to extract labels from name using regex. Those labels help reduce the complexity of queries that aggregate metrics by different components.

Detailed documentation on the available metrics can be found here.

Alerts

Auto Generated Alerts

As I shared earlier, the system has an internal Alerting module that automatically triggers alerts for critical situations and creates tickets. To cover those alerts on the Prometheus side, we added an alerting configuration of our own that relies on the incoming severities:

- alert: PureAlert
  annotations:
    summary: '{{ $labels.summary }}'
    description: '{{ $labels.component_type }} - {{ $labels.component_name }} - {{ $labels.action }} - {{ $labels.kburl }}'
    dashboard: 'https://grafana/your-dashboard'
  expr: purefb_alerts_open{environment="production"} == 1
  for: 1m

We still need to evaluate how the pure-generated alerts will interact with the custom alerts I will cover below, and we might decide to stick to one or the other depending on what we find out.

Hardware

Before I continue, the image below helps visualize how some of the Pure FlashBlade components are physically organized:

Because of Pure’s reliability, most isolated hardware failures do not require the immediate attention of an Ops team member. To cover the most basic hardware failures, we configure an alert that sends a message to the Ops Basecamp 4 project chat:

- alert: PureHardwareFailed
  annotations:
    summary: Hardware {{ $labels.name }} in chassis {{ $labels.ch }} is failed
    description: 'The Pure Storage hardware {{ $labels.name }} in chassis {{ $labels.ch }} is failed'
    dashboard: 'https://grafana/your-dashboard'
  expr: purefb_hardware_health == 0
  for: 1m
  labels:
    severity: chat-notification

We also configure alerts that check for multiple hardware failures of the same type. This doesn’t mean two simultaneous failures will result in a critical state, but it is a fair guardrail for unexpected scenarios. We also expect those situations to be rare, keeping the risk of causing unnecessary noise low.

- alert: PureMultipleHardwareFailed
  annotations:
    summary: Pure chassis {{ $labels.ch }} has {{ $value }} failed {{ $labels.type }}
    description: 'The Pure Storage chassis {{ $labels.ch }} has {{ $value }} failed {{ $labels.type }}, close to the healthy limit of two simultaneous failures. Ensure that the hardware failures are being worked on'
    dashboard: 'https://grafana/your-dashboard'
  expr: count(purefb_hardware_health{type!~"eth|mgmt_port|bay"} == 0) by (ch,type,environment) > 1
  for: 1m
  labels:
    severity: page
 # We are looking for multiple failed bays in the same blade
- alert: PureMultipleBaysFailed
  annotations:
    summary: Pure chassis {{ $labels.ch }} has fb {{ $labels.fb }} with {{ $value }} failed bays
    description: 'The Pure Storage chassis {{ $labels.ch }} has fb {{ $labels.fb }} with {{ $value }} failed bays, close to the healthy limit of two simultaneous failures. Ensure that the hardware failures are being worked on'
    dashboard: 'https://grafana/your-dashboard'
  expr: count(purefb_hardware_health{type="bay"} == 0) by (ch,type,fb,environment) > 1
  for: 1m
  labels:
    severity: page

Finally, we configure high-level alerts for chassis and XFM failures:

- alert: PureChassisFailed
  annotations:
    summary: Chassis {{ $labels.name }} is failed
    description: 'The Pure Storage hardware chassis {{ $labels.name }} is failed'
    dashboard: 'https://grafana/your-dashboard'
  expr: purefb_hardware_health{type="ch"} == 0
  for: 1m
  labels:
    severity: page
- alert: PureXFMFailed
  annotations:
    summary: Xternal Fabric Module {{ $labels.name }} is failed
    description: 'The Pure Storage hardware Xternal fabric module {{ $labels.name }} is failed'
    dashboard: 'https://grafana/your-dashboard'
  expr: purefb_hardware_health{type="xfm"} == 0
  for: 1m
  labels:
  severity: page

Latency

Using the metric purefb_array_performance_latency_usec we can set a threshold for all the different protocols and dimensions (read, write, etc), so we are alerted if any problem causes the latency to go above an expected level.

- alert: PureLatencyHigh
  annotations:
    summary: Pure {{ $labels.dimension }} - {{ $labels.protocol }} latency high
    description: 'Pure {{ $labels.protocol }} latency for dimension {{ $labels.dimension }} is above 100ms'
    dashboard: 'https://grafana/your-dashboard'
  expr: (avg_over_time(purefb_array_performance_latency_usec{protocol="all"}[30m]) * 0.001)
  for: 1m
  labels:
    severity: chat-notification

Saturation

For saturation, we are primarily worried about something unexpected causing excessive use of array space, increasing the risk of hitting the cluster capacity. With that in mind, it’s good to have a simple alert in place, even if we don’t expect it to fire anytime soon:

- alert: PureArraySpace
  annotations:
    summary: Pure Cluster {{ $labels.instance }} available space is expected to be below 10%
    description: 'The array space for pure cluster {{ $labels.instance }} is expected to be below 10% in a month, please investigate and ensure there is no risk of running out of capacity'
    dashboard: 'https://grafana/your-dashboard'
  expr: (predict_linear(purefb_array_space_bytes{space="empty",type="array"}[30d], 730 * 3600)) < (purefb_array_space_bytes{space="capacity",type="array"} * 0.10)
  for: 1m
  labels:
    severity: chat-notification

HTTP

We use BigIp load balancers to front-end the cluster, which means that all the alerts we already had in place for the BigIp HTTP profiles, virtual servers, and pools also cover access to Pure. The solution for each organization on this topic will be different, but it is a good practice to keep an eye on HTTP status codes and throughput.

Grafana Dashboards

The project’s GitHub repository includes JSON files for Grafana dashboards that are based on the metrics generated by the exporter. With simple adjustments to fit each setup, it’s possible to import them quickly.

Wrapping up

On top of the system’s built-in capabilities, Pure also provides options to integrate their system into well-known tools like Prometheus and Grafana, facilitating the process of managing the cluster the same way we manage everything else. I hope this post helps any other team interested in working with them better understand the effort involved.

Thanks for reading!

Reloading the browser automatically on source changes is a problem that has been well-solved for a long time. Here, we wanted to put an accent on smoothness. If the reload operation is very noticeable, the feedback loop is similar to just reloading the page yourself. But if it’s smooth enough — if you only perceive the intended change — the feedback loop becomes terrific.

To use, just install the gem in development:

group :development do
  gem "hotwire-spark"
end

It will update the current page on three types of change: HTML content, CSS, and Stimulus controllers. How do we achieve that desired smoothness with each?

• For HTML content, it morphs the <body> of the page into the new <body>. Also, it disconnects and reconnects all the Stimulus controllers on the page.
• For CSS, it reloads the changed stylesheet.
• For Stimulus controllers, it fetches the changed controller, replaces its module in Stimulus, and reconnects all the controllers.

We designed Hotwire Spark to shine with the nobuild approach we use and recommend. Serving CSS and JS assets as standalone files is ideal when you want to fetch and update only what has changed. There is no need to use bundling or any tooling. Hot Module Replacement for Stimulus controllers without any frontend building tool is pretty cool!

2024 has been a very special year for Rails. We’re thrilled to share Hotwire Spark before the year wraps up.

Wishing you all a joyful holiday season and a fantastic start to 2025.

• Fight hard before adding Ruby dependencies. Keep that Gemfile that Rails generates as close to the original one as possible.
• Fight even harder before adding Javascript dependencies. You don’t need React or any other front-end frameworks, nor a JSON API to feed those.
• Hotwire is a fantastic, pragmatic, and ridiculously productive technology for the front end. Use it.
• The same goes for mobile apps: use Hotwire Native. With a hybrid approach you can combine the very same web app you have built with a wonderful native experience right where you want it. The productivity compared to a purely native approach is night and day.
• Embrace and celebrate rendering things on the server. It has become cool again.
• ERB templates and view helpers will take you as long as you need, and they are a fantastic common ground for designers to collaborate hands-on with the code.
• #nobuild is the simplest way to go; don’t close this door with your choices. Instead of bundling Javascript, use import maps. Don’t bundle CSS, just use modern standard CSS goodies and serve them all with Propshaft. If you have 100 Javascript files and 100 stylesheets, serve 200 standalone requests multiplexed over HTTP2. You will be delighted.
• Don’t add Redis to the mix. Use solid_cache for caching, solid_queue for jobs, and solid_cable for Action Cable. They will all work on your beloved relational database and are battle-tested.
• Test your apps with Minitest. Use fixtures and build a realistic set of those as you cook your app.
• Make your app a PWA, which is fully supported by Rails 8. This may be more than enough before caring about mobile apps at all.
• Deploy your app with Kamal.

If you want heuristics, your importmap.rb should import Turbo, Stimulus, your app controllers, and little else. Your Gemfile should be almost identical to the one that Rails generates. I know it sounds radical, but going vanilla is a radical stance in this convoluted world of endless choices.

This is the Rails 8 stack we have chosen for our new apps at 37signals. We are a tiny crew, so we care a lot about productivity. And we sell products, not stacks, so we care a lot about delighting our users. This is our Omakase stack because it offers the optimal balance for achieving both.

Vanilla means your app stays nimble. Fewer dependencies mean fewer future headaches. You get a tight integration out of the box, so you can focus on building things. It also maximizes the odds of having smoother future upgrades. Vanilla requires determination, though, because new dependencies always look shiny and shinier. It’s always clear what you get when you add them, but never what you lose in the long term.

It is certainly up to you. Rails is a wonderful big tent. These are our opinions.

If it resonates, choose vanilla!

Guess what our advice is for architecting your app internals?

• Support for Solid Queue’s recurring tasks, including running them on-demand.
• Support for API-only apps.
• Allowing immediate dispatching of scheduled and blocked jobs.
• Backtrace cleaning for failed jobs’ backtraces.
• A safer default for authentication, with Basic HTTP authentication enabled and initially closed unless configured or explicitly disabled.

Recurring tasks in Mission Control — Jobs, with a subset of the tasks we run in production

We use Mission Control — Jobs daily to manage jobs HEY and Basecamp 4, with both Solid Queue and Resque, and it’s the dashboard we recommend if you’re using Solid Queue for your jobs. Our plan is to upstream some of the extensions we’ve made to Active Job and continue improving it until it’s ready to be included by default in Rails together with Solid Queue.

If you want to help us with that, are interested in learning more or have any issues or questions, head over to the repo in GitHub. We hope you like it!

Kicking things off

At 37signals we run projects in six-week cycles informed by Shape Up. At the beginning of each cycle, Brian, our Head of Product, posts a kick-off message detailing what we plan to ship. This usually consists of new features and improvements for Basecamp, HEY, or a ONCE product. Each gets its own Basecamp project, and each project includes a pitch. The pitch lays out the problem or need, a proposed solution, and the “appetite” or time budget. The kick-off is also QA’s cue to dive in! We offer early feedback, ask questions or illuminate things that aren’t covered, and give extra consideration to flows and interactions that may require extra work on the accessibility front. We then step back and let the teams focus, design, and build things for a while.

The right time to test

We wait until the feature or product reaches a usable state to start testing in earnest. This helps us keep a fresh perspective, unencumbered by the knowledge of compromises made along the way.

We use a Card Table within our QA Team project to track what’s ready for testing or in progress. Teams add a card to the Ready for QA (Triage) section when the time is right. The table is kept simple with just two columns, In Progress and Pending Input, for when we’ve completed our test run and the team is addressing the feedback. Depending on the breadth and complexity of the work being tested, this flow can take anywhere from a few hours to a few days.

A holistic approach to QA

Once we take on a request, we explore and scrutinize the feature much like an (extremely zealous!) customer would. We want to help teams ship the most polished features they can. We look out for bugs of all kinds: performance issues, visual glitches, unexpected changes, and so on, but perhaps most importantly, we offer feedback on the usability of the feature. We guide our feedback with questions like:

• Is this feature easy to discover and access? Is it in the right spot?
• Does it interact in an unexpected way with another part of the app?
• How does the change play with our mobile apps?
• Does this solve the problem in a way that customers will find obvious?

Critically, what we raise with this type of QA testing are suggestions, not must-haves. The designer and programmer working on the feature make the call on what to address and what to shelve.

We document this feedback in a dedicated Card Table within the feature’s Basecamp project. The designer and programmer will then review the cards we’ve added to Triage and direct them to the In Progress and Not Now columns as appropriate. From In Progress, cards are moved to a column called QA to confirm fixed, then finally to Done.

More focus, less bloat

Our overall approach to testing is guided exploration. We don’t maintain an exhaustive collection of test cases to dogmatically review each time we test a feature. We’ve tried using dedicated test plan tools and comprehensive spreadsheets of test cases upon test cases; the time spent certifying every little thing was considerable, yet it didn’t translate into finding more issues. Worse, it left us with less time to spend sitting with the feature in a more subjective way. We’ve landed on a more pragmatic approach.

We’ve boiled down the test plan to a concise list of considerations that live in Basecamp as to-do list templates, one for each product. Instead of a multitude of test cases, each template contains around 100 items. These act as pointers, touching on overall concepts (like commenting, dark mode, email notifications), specific areas of the app, and platform-specific considerations. We reflect on the work presented and how it ties into these areas. Some examples from recent projects have been:

• Did we update exporting to consider this new addition of time tracking entries?
• Are email notifications properly reflecting the new Steps feature we added to Card Table?
• How about print styles, do they look good?

QA Considerations for Basecamp 4

We create a to-do list via the template directly in the project we are working on, and use that as our reference for reviewing the work. We also ask the feature team if there are areas that deserve extra attention. Being flexible and discerning about how much time and coverage we use in our testing allows us to cover anywhere from 4 to 12+ projects in a very short span of time.

We love working as a team of two and being able to riff on how to approach testing a feature. Sometimes, we divide and conquer; other times, both of us review the work. Fresh eyes provide a good chance of catching something new. Gabriel has a better knack for Android conventions and Michael for iOS, but we actively avoid over-specializing. Keeping up with multiple platforms requires extra effort, but it’s worth it when considering the consistency of the experience across all of them.

Accessibility

As part of our review, we test the accessibility of the changes. We use a combination of keyboard navigation and at least one screen reader on each platform to vet how well the feature will work for someone who relies on accessible technology. We also use browser extensions like axe and Accessibility Insights for Web to validate semantics of the code and Headings Map to make sure heading levels are sequential. At times, we bring in customers who use a screen reader full-time to help us validate whether everything makes sense and learn where things can improve. Our new colleague, Bruno, is a full-time user of the NVDA screen reader and can offer this sort of direct feedback on how a feature or flow works for him.

Explorations in tooling

A recent addition to our toolkit is a visual regression tool built on BackstopJS with the help of our colleague Lewis. Whenever we review work, we can run the suite of tests — mostly a list of URLs for various pages around the app — first pointed to production, then against a beta environment where the new feature is staged. Any visual differences will be flagged in a report we review, then write up bug report cards for the team if needed.

Walking the walk

Part of what enables us to keep our process minimal is that we use our products daily, both on the job and in our everyday lives. This affords us an intimate understanding of how they work and how they can be improved. We’re passionate about what we do. We find ourselves fortunate to work with each other and with so many talented colleagues.

We hope this post has given you some helpful insight into the way we do things! If you have questions or if there are topics you’d like us to cover in future posts, drop us an email at qa@37signals.com.

Apart from fixing many bugs and edge cases, we’ve enhanced Solid Queue with the following:

• Safe and atomic batch operations to discard, retry and unblock jobs, used from Mission Control – Jobs.
• Enqueueing jobs in bulk (enqueue_all for Active Job’s perform_all_later).
• Recurring (cron-style) jobs.
• Proper logging and instrumentation.
• Lifecycle hooks for the supervisor and workers.
• A better installation, with a single schema file, separate DB configured by default and a binstub to easily start the supervisor.

More importantly, we’ve completely migrated HEY, our email and calendar service, over from Resque. In fact, HEY Calendar was launched in January directly using Solid Queue for all jobs. Some issues we had after that launch is what drove batch operations support. We’ve also started moving jobs in our biggest app, Basecamp 4.

Our production setup

Currently in HEY we’re processing about 20 million jobs per day, using 800 workers, 4 dispatchers and 2 schedulers, spread over 74 VMs running in two datacenters, that we deploy using Kamal. Some of the queues are quite overprovisioned from our past Resque setup; we’ll reduce the number of workers, VMs and servers in the future, and reassign them to other work. These 74 VMs are sized differently depending on load and priority, with 2 to 6 cores (we have AMD EPYC 9454 48-Core processors) and 4G to 27G memory allocated for each of them. The physical servers where these live are running a mix of VMs for web, jobs and other apps, we don’t run dedicated machines only for HEY.

We’re running Solid Queue in a separate database from the main app. We also have the tables for Active Storage and Action Text in separate DBs. We run a primary and two replicas for each, and the four databases share the same physical servers. For the Solid Queue database, we’ve allocated 32 CPUs, 64G memory and 350G disk.

This is how our deploy configuration looks like, a bit simplified:

solid_queue_jobs:
  solid_queue_jobs:
    cmd: bin/jobs -c config/solid_queue/<%= ENV.fetch("JOB_POOL", "default") %>.yml --skip_recurring
  hosts:
    - hey-mail-jobs-101: mail_jobs
    - hey-mail-jobs-102: mail_jobs
    - hey-mail-jobs-103: mail_jobs
    - hey-mail-jobs-104: mail_jobs

# ...

    mail_jobs:
      JOB_POOL: mail
# ...

We use Mission Control – Jobs to manage the queues and Prometheus and Yabeda for our monitoring needs. Here’s how some of the metrics we export for alerting look like:

module Yabeda
  module SolidQueue
    def self.install!
      Yabeda.configure do
        group :solid_queue

        gauge :jobs_failed_count, comment: "Number of failed jobs"
        gauge :jobs_scheduled_and_delayed_count, comment: "Number of scheduled jobs that have over 2 hours delay"

        collect do
          if ::SolidQueue.supervisor?
            solid_queue.jobs_failed_count.set({}, ::SolidQueue::FailedExecution.count)
            solid_queue.jobs_scheduled_and_delayed_count.set({}, ::SolidQueue::ScheduledExecution.where(scheduled_at: ..2.hours.ago).count)
          end
        end
      end
    end
  end
end

Then we start the metrics server like this:

SolidQueue.on_start do
  Yabeda::Prometheus::Exporter.start_metrics_server!
end

PostgreSQL and SQLite: big thanks to the community

We use MySQL at 37signals so we don’t have as much expertise or visibility on issues involving the other two databases we support, PostgreSQL and SQLite. Thankfully, there are great experts in the community that are helping tremendously with that. Andrew Atkinson, the author of “High Performance PostgreSQL for Rails”, has published a very informative article about using Solid Queue with PostgresSQL that I recommend you to read if you are using Postgres. Hal Spitz contributed directly to Solid Queue with a very helpful fix for a PostgreSQL-only issue related to concurrency controls. Andy Croll, Marco Roth, Stephen Margheim, Nick Pezza and, especially, Mike Dalessio were of incredible help in diagnosing and fixing a scenario where the SQLite database used for Solid Queue could end up being corrupted. Stephen himself has published a number of articles on using SQLite effectively with Rails, such as this one, and contributed valuable changes in that direction to the framework. I’m really grateful for all this.

What’s next

After Rails World, we plan to focus again on Mission Control – Jobs for a bit, with the goal of releasing v1.0 soon. As for Solid Queue, since we’ll be working on migrating our biggest app, Basecamp 4, where we run about 4 times more jobs than in HEY, we’ll explore sharding support and other scalability improvements that I’m sure we’ll need as we progress with the work.

We hope you find this useful!

Kamal 1.0 was designed with 37signal’s use case in mind — deploying an application across multiple hosts, served with an external load balancer.

With Kamal 2.0 we’ve focused on making it simpler to use at any scale, whether you are deploying your app to 50 servers or deploying 5 apps to a single server.

What’s new?

• kamal-proxy, a new proxy for gapless deployments
• Automatic HTTPS with Let’s Encrypt
• Deploy many applications to one server
• Create aliases for common kamal commands
• Simplified secret management, with commands for pulling secrets from password managers

New proxy

Kamal uses a proxy to seamlessly switch between application versions.

Previously we used Traefik, but its declarative discovery model made it a poor match for Kamal’s imperative design.

Kamal 2 use a custom-built proxy, kamal-proxy. This simplifies deployments by providing a 1-1 mapping between kamal commands and proxy commands.

This has allowed us to build in new features like maintenance mode, pausing requests and canary deployments that will be coming to Kamal soon.

Upgrading

Kamal 2.0 will be installed by default in Rails 8.0. It’s not just for Rails applications though — you can use it for web apps written in any language or framework.

If you already use Kamal 1, we’ve written an upgrade guide.

We are already running HEY deployments on Kamal 2 and are benefiting from faster deployments and simpler configuration. And we will have all our apps running it soon.

Background

But first, let me provide some context. Last year, shortly before Rails World 2023, we open sourced Strada, which allows you to build high fidelity native features in your hybrid Turbo Native apps. The launch was met with a lot of excitement and development. However, there was a consistent piece of feedback that we heard: the relationship between the Hotwire umbrella of libraries, Turbo + Stimulus + Strada, was quite confusing.

Strada is meant to leverage Turbo Native apps, yet the documentation and marketing for Turbo Native apps has been nearly non-existent. Further, integrating Strada into your Turbo Native apps requires effort that shouldn’t be necessary.

Hotwire Native

As we thought about it more, we saw an opportunity to make the relationship between the libraries much more clear. It made sense to consolidate the Turbo Native and Strada libraries under one roof, which would also allow us to more easily build new layers of out-of-the-box features for developers.

We’ve been working this year to make the idea a reality and the result is Hotwire Native! It’s the best way to build native apps for iOS and Android while leveraging your existing Hotwire web apps.

A web-first framework for building native mobile apps

Hotwire Native is the consolidation of the Turbo Native and Strada libraries into a single framework for iOS and Android. Going forward, the Hotwire umbrella of libraries now includes: Turbo + Stimulus + Native.

A Hotwire Native app lets you take a unique web-first approach towards development with smaller teams:

• Leverage your web screens from your existing web app.
• Build Bridge Components (formerly Strada components) to add high fidelity features in your web screens.
• Build fully native screens for the most important areas of your app that require the highest fidelity.

Hotwire Native’s web-first approach means upgrading to native isn’t an all-or-nothing decision. You are free to choose specific screens or even specific components to write natively in Swift or Kotlin when you’re ready. It truly is progressive enhancement.

Improvements

We didn’t simply consolidate the Turbo Native and Strada libraries and call it a day. We’ve made substantial improvements in key areas such as:

• Building a new iOS or Android app requires just a few lines of code to get started.
• A whole new set of configuration options to make it easier than ever to customize your native app.
• A brand new navigation layer that handles all the complex navigation stack situations that you may run into.
• More comprehensive Path Configuration rules that are consistent across iOS and Android.
• Bridge Components (formerly Strada components) work out-of-the-box without additional integration required.
• It’s easier than ever to replace web screens with native screens and route to them via URLs.

There’s plenty of other improvements throughout the iOS and Android libraries. We have thorough documentation and examples that you can see at native.hotwired.dev.

A big thank you

We couldn’t have done all this work without the incredible help from Joe Masilotti. Joe has been an amazing advocate and educator for building Turbo Native apps for many years and he helped the 37signals team put together all the great resources available for this launch.

And maybe most significantly, he made a big effort bringing his powerful TurboNavigator library directly into the new Hotwire Native iOS library. It’s the foundation of the new built-in navigation in the iOS library and it brings navigation parity to the Hotwire Native Android library.

Joe is rather famously known as the “Turbo Native guy” and now that Hotwire Native is public, he may want to consider a name change!

Deprecations

In light of the new Hotwire Native libraries, in the near future we’ll be deprecating the existing Turbo Native and Strada libraries for iOS and Android. The code in those original libraries serves as the new foundation for Hotwire Native and we’ll be focused on Hotwire Native development going forward. You can start transitioning over to the new libraries whenever you’re ready and start taking advantage of all the new out-of-the-box capabilities.

Check it out

Take a look at all the resources available on native.hotwired.dev and we’re excited to see the new apps you build! We’re looking forward to your feedback and contributions!

I’ve done a poor job here countless times in my career. And I have seen many others struggle here, too. If you like your craft, it’s natural that you want to do your best all the time. But the best and good enough are different things: you must learn when and where to aim for each.

This is easy to understand, but it’s very hard to put into practice when it is time to do the work. How do you convince creative individuals who love building things to build less? How do you get someone who enjoys technical challenges to try to make things simpler and more boring? There is a joke about never asking a surgeon if you need surgery. What about asking developers if they need to build the perfect interaction, abstraction, or library? Of course they do!

I believe the solution here is not about good practices or coaching. Instead, it’s about creating a working environment that makes costs a first-level concern for those doing the work. Aiming for good enough won’t happen as an intellectual exercise in the abstract but out of pure necessity at execution time.

How can you create such an environment? With constraints! I’ve seen 37signals apply these two consistently:

First, timeboxing. Time is not infinite, so don’t pretend it is. Set a deadline and give people doing the work the power to adjust what’s built. Fixed time, variable scope is probably the single most powerful concept I have ever learned about shipping.

Second, scarcity. Tiny teams that want to ship need to make the most of their time. Why would you restate problems to simplify them when you have an army of colleagues to work on whatever is pending to ship? On the contrary, how won’t you do it if shipping depends on completing a long list of additional things on your plate? The specific team sizes will vary, but I believe the principle prevails: a sense of scarcity helps shipping.

It may sound counterintuitive but consider the alternative. If neither time nor development bandwidth is a concern, how could people doing the work make the right call for the millions of decisions ahead?

When it comes to shipping, constraints are not to be avoided or resisted but rather embraced and even induced.

What is an homograph attack

Homograph characters look the same or very similar to other characters, but are different. For example, the letters “l” and “ӏ” (Cyrrilic “ӏ”) look the same but are different characters. A homograph attack is a phishing attack where the attacker uses homograph characters to spoof another identity. Homograph attacks became a real security concern after the introduction of Internationalized Domain Names (IDNs), which allowed domain names to contain Unicode characters. For example, a malicious actor could send you a fake reset password email from “support@paypaӏ.com” (note the Cyrrilic “ӏ”) with a link to their phishing website “https://paypaӏ.com” that looks like PayPal (and has a valid SSL certificate) stealing your credentials.

How to protect from homograph attacks

The most common way to protect from homograph attacks, implemented by modern browsers and most email clients, is to convert Unicode characters into Punycode: a representation of Unicode with the limited ASCII character subset used for Internet hostnames. But converting all international domains using non-ASCII Unicode characters to Punycode would be gross because it’d make legitimate domains look like gibberish: Punycode conversion should occur only when there is a real security threat. To help with this, the Unicode Technical Standard #39 (UTS #39) provides guidelines to detect security issues with Unicode character usage.

State of the art in Ruby

In HEY, our email service, we had to include a mechanism to protect our users from homograph attacks. We didn’t find any Ruby implementations for UTS #39, so we had to implement our own. Recently, we extracted it to an open-source Gem so you can all benefit from it. Our implementation is complete and in line with UTS #39. You can use it to protect both IDNs and email addresses. We hope you find it useful!

We also use Prometheus metrics to monitor our Rails apps. The metrics can tell us a lot, including:

• request rates and latencies
• how many background jobs have succeeded or failed
• how YJIT is performing

In production, we run multiple processes on multiple containers and on multiple servers so we need a solution to aggregate and collate those metrics to form one overall picture.

This post will cover how we export metrics from our Rails apps, including how we export them from Docker containers managed via Kamal.

Yabeda

Prometheus scrapes metrics from our Rails servers and so requires the app to serve, or export, those metrics. To help us do that we use Yabeda. Yabeda is a collection of Ruby gems from Evil Martians that makes it easy to export Prometheus metrics from a Rails app. Yabeda Prometheus supports aggregating metrics from individual processes, so we only need to scrape the metrics at the container level.

Once set up with yabeda-prometheus, yabeda-puma-plugin, and yabeda-rails metrics can be viewed by running:

$ curl localhost:9394/metrics

# HELP rails_requests_total Multiprocess metric
# TYPE rails_requests_total counter
rails_requests_total{action="show",controller="Boxes::ImboxController",format="html",method="get",status="500"} 1
# HELP rails_view_runtime_seconds Multiprocess metric
# TYPE rails_view_runtime_seconds histogram
rails_view_runtime_seconds_bucket{action="show",controller="Boxes::ImboxController",format="html",le="+Inf",method="get",status="500"} 1
rails_view_runtime_seconds_bucket{action="show",controller="Boxes::ImboxController",format="html",le="0.005",method="get",status="500"} 1
...

Serving metrics on port 9394 and at the /metrics path is typical for Prometheus Ruby clients.

Performance

We ran into some performance problems with the yabeda-prometheus gem, which struggled with our large number of metrics. To fix this we switched over to use yabeda-prometheus-mmap based on the more performant prometheus-client-mmap from the team at GitLab.

One pitfall which we ran into is noted in the prometheus-client-mmap README:

PID cardinality

In multiprocess setup e.g. running under Unicorn or Puma, having worker process restart often can lead to performance problems when processing metric files. By default each process using Prometheus metrics will create a set of files based on that process PID. With high worker churn this will lead to creation of thousands of files and in turn will cause very noticeable slowdown when displaying metrics

To reduce this problem, a surrogate process id can be used. Set of all such IDs needs have low cardinality, and each process id must be unique among all running process.

What this means is that if you use Unicorn, Puma, or even Resque it’s important to follow the instructions on configuring a PID provider. We do that like so:

# config/initializes/yabeda.rb

require "prometheus/client/support/puma"

Prometheus::Client.configuration.pid_provider = Prometheus::Client::Support::Puma.method(:worker_pid_provider)

Production

Once we were happy with the metrics being served by our app, we needed to think about how to expose a single endpoint on each server for Prometheus to scrape, when we can have multiple Docker containers running on a single host.

To do that we’re using OpenTelemetry Collector as a Kamal accessory. OpenTelemetry Collector scrapes metrics from containers, on the Docker internal network, and exposes them in one external location, port 9394.

• each container exposes port 9394, serving metrics
• OpenTelemetry Collector uses Docker service discovery to locate these containers and scrape metrics
• the Collector then collates and exposes those metrics on port 9394 and labels them with the container name

Finally, Prometheus scrapes and stores those metrics, allowing querying metrics across the system.

Here’s a simplified diagram showing it works:

The benefits of this system:

• a single endpoint per host to scrape metrics from
• it avoids port conflicts with multiple containers on a single host
• it avoids complex Traefik configuration to get it to proxy metrics

The configuration is fairly simple:

# Kamal
# config/deploy.yml

...

accessories:
  otel_collector:
    image: otel/opentelemetry-collector:0.100.0
    port: 9394
    files:
      - config/otel_collector.yml:/etc/otelcol/config.yaml
    volumes:
      - /var/run/docker.sock:/var/run/docker.sock
    options:
      user: 0 # if you run Docker as root
    roles:
      - web

# OpenTelemetry Collector
# config/otel_collector.yml

receivers:
  prometheus:
    config:
      scrape_configs:
        - job_name: "otel-collector"
          scrape_interval: 5s
          docker_sd_configs:
            - host: unix:///var/run/docker.sock
              refresh_interval: 10s
          relabel_configs:
            # Only keep targets with role label as 'web' or 'jobs'
            - source_labels: [__meta_docker_container_label_role]
              regex: (web|jobs)
              action: keep
            # Only scrape port 9394
            - source_labels: [__address__]
              regex: (.*):9394
              action: keep
            # Add container label
            - source_labels: [__meta_docker_container_label_role, __meta_docker_container_id]
              separator: "_"
              target_label: container

exporters:
  prometheus:
    endpoint: 0.0.0.0:9394

service:
  telemetry:
    logs:
      level: "debug"
  pipelines:
    metrics:
      receivers: [prometheus]
      exporters: [prometheus]

I’ve created a repository showing how these pieces fit together.

• Sample application, showing Rails, Yabeda and Kamal configuration

Check the commits for more detail and the Kamal documentation to see how to manage accessories.

Questions?

I hope you find this post useful and would be interested in hearing from you if you do. Have a question, comment or idea? Get in touch at lewis@hey.com or x.com/lewispb

Today I’m pleased to announce that we are open sourcing Mission Control — Web and you can find it on GitHub.

Here’s how it looks:

Admin dashboard of Mission Control — Web

What’s it for?

Mission Control — Web allows immediate control of web requests during incidents, by denying access to parts of the application, defined by specific paths.

My colleague Jorge proposed Mission Control — Web in 2022. He identified the need for such a tool during previous incidents, where a whole application became unavailable due to specific features being unperformant.

This is admittedly a simple tool, but it’s a nice lever to be able to pull in case of emergency.

What does it do?

It has two parts: an admin dashboard and a middleware. These can be configured in the same app, or ideally, two separate apps. They share a Redis database.

The Admin dashboard is used to set up a list of paths, defined as regex patterns. Requests to these paths will be denied in the application configured with the middleware.

Version 0.2.0 is available now. We hope you find it useful! If you are interested in contributing, we have prepared a list of improvements and new features we’d like to see in the future, some of them great for first-time contributors.

In this post we’ll take a look at how we’re using some of these features and share some helpful patterns discovered along the way.

Colors

Campfire uses oklch() to define colors in CSS. oklch() offers access to wider color spaces (like Display-P3) and greatly improves developer ergonomics when working with colors. For example, let’s take a look at these greys used in Campfire’s UI.

:root {
  --lch-gray: 96% 0.005 96;
  --lch-gray-dark: 92% 0.005 96;
  --lch-gray-darker: 75% 0.005 96;
}

At first glance they may seem unfamiliar but they’re actually more readable and quite easy to use once you get acquainted.

LCH stands for:

• Lightness: perceptual lightness ranging from 0%—100%;
• Chroma: the amount of color from pure grey to full saturation, 0–0.5;
• Hue: the color’s angle on the color wheel, 0–360deg.

With that in mind, we can read the colors without much effort. We can see that they all share the same hue and chroma, only the lightness differs. It’s apparent just from reading the code that --lch-gray and --lch-gray-dark are relatively close in lightness, but --lch-gray-darker is significantly darker. It’s also simple to adjust them programmatically or manually tweak them without using a color picker and without inadvertently shifting the hue. If you’ve ever tried to do that with RGB colors you know how tricky that can be.

We started by defining the pure color values above but we wrap them in the oklch() color function and define a set of abstract custom properties that consume the values for use in our other stylesheets.

--color-border: oklch(var(--lch-gray));
--color-border-dark: oklch(var(--lch-gray-dark));
--color-border-darker: oklch(var(--lch-gray-darker));

Sure, you might be thinking, grey is easy but what about other colors? Here’s a set based on blue for links and selections.

--lch-blue: 54% 0.23 255;
--lch-blue-light: 95% 0.03 255;
--lch-blue-dark: 80% 0.08 255;

--color-link: oklch(var(--lch-blue));
--color-selected: oklch(var(--lch-blue-light));
--color-selected-dark: oklch(var(--lch-blue-dark));

A quick read of these values reveals that all three are in the same color family, indicated by the same hue angle (255º). Further we can observe that links are medium lightness and saturation. The light variant has a much higher lightness value and much lower saturation making it more grey, while the dark variant is not quite as light or desaturated. We generally use the darker variants for borders around the lighter values.

And even better, oklch() makes it trivial to add variants that use alpha transparency, too.

--color-link-50: oklch(var(--lch-blue) / 0.5);

Custom Properties

Variables in CSS are certainly not new but we’ve developed some general usage patterns that make working with them a pleasure. Let’s look at some styles from Campfire’s buttons.css to demonstrate.

Declared vs. Fallback values

Often when using custom properties in the past, we’d set something up like this in which you declare all the custom properties at the top of the rule (or in :root) and then use them immediately below. Something like this:

.btn {
  --btn-background: var(--color-text-reversed);
  --btn-border-color: var(--color-border);
  --btn-border-radius: 2em;
  --btn-border-size: 1px;
  --btn-color: var(--color-text);
  --btn-padding: 0.5em 1.1em;

  align-items: center;
  background-color: var(--btn-background);
  border-radius: var(--btn-border-radius);
  border: var(--btn-border-size) solid var(--btn-border-color);
  color: var(--btn-color);
  display: inline-flex;
  gap: 0.5em;
  justify-content: center;
  padding: var(--btn-padding);
}

And that works fine but it feels like a lot of boilerplate and it’s a little defensive in that you may never use those variables again. That’s where fallback values come in handy. Instead of a litany of properties at the top of the rule, we can set the default values inline but expose a custom property that will accept another value when present. It looks like this:

color: var(--btn-color, var(--color-text));

Here --btn-color is optional. If it’s set, the rule will use that value; if not, it will fall back to --color-text. The fallback value can be a straight value or another variable. Now we can re-write the rule above like this:

.btn {
  align-items: center;
  background-color: var(--btn-background, var(--color-text-reversed));
  border-radius: var(--btn-border-radius);
  border: var(--btn-border-size, 2em) solid var(--btn-border-color, var(--color-border));
  color: var(--btn-color, var(--color-text));
  display: inline-flex;
  gap: 0.5em;
  justify-content: center;
  padding: var(--btn-padding, 0.5em 1.1em);
}

This is tighter and all the default values plus exposed variables are together, inline.

But how do we decide where to use custom properties? There are really two cases: 1) whenever we need to use the same value in more than one place (DRY) and 2) when we know a value is going to be changed.

A good example of the first case is the --btn-size variable. Almost all of Campfire’s buttons are circles with an icon inside. To make sure they line up nicely with input fields we set their block-size using this variable.

Because that size is exposed at the :root level we can use it for buttons and input elements. And even better, we can use that value to calculate the height of the chat footer in our layout. No magic numbers in sight!

:root {
  --btn-size: 2.65em;
}

body {
  --footer-height: calc((var(--block-space)) + var(--btn-size) + var(--block-space));

  grid-template-rows: 1fr var(--footer-height);
}

The footer’s height consists of the button’s height plus padding above and below using the global --block-space variable.

The other case for custom properties is when we know that we’ll want to change some values to create variants of an element. We think of it like a mini API for our CSS classes. Going back to our button class, we can declare variants simply by changing the value of custom properties instead of redefining a property.

/* Variants */

.btn--reversed {
  --btn-background: var(--color-text);
}

.btn--negative {
  --btn-background: var(--color-negative);
}

:is(.btn--reversed, .btn--negative) {
  --btn-color: var(--color-text-reversed);
}

.btn--borderless {
  --btn-border-color: transparent;
}

.btn--success {
  animation: success 1s ease-out;

  img {
    animation: zoom-fade 300ms ease-out;
  }
}

This makes it very clear what’s changed by these variants. Even better, as in the case of .btn--success, it makes on obvious distinction between changing a default property value and adding a new property (the animation property in this case).

CSS :has()

We started using :has() in the early stages of Campfire’s development because it offers a number of conveniences and opportunities to do with CSS what we previously had to do in server side code. We were so bullish on :has() that we literally shipped the first beta version of Campfire a week before Firefox shipped its release with support for :has()—the last of the major browsers to do so.

You can think of :has() as a way to query an element about what’s inside it.

This makes our button class very flexible. You can throw about any combination of things inside it, and it will adjust accordingly. Text only, image and text, image only, inputs (like radio buttons), or multiple images with text.

For example, when our .btn class finds an image inside of it (that’s not an avatar photo), it can apply sizing and make sure it gets inverted in dark mode—without needing any kind of special classes.

.btn {
  ...

  img {
    -webkit-touch-callout: none;
    user-select: none;
  }

  &:where(:has(img):not(.avatar)) {
    text-align: start;

    img {
      filter: invert(0);
      inline-size: 1.3em;
      max-inline-size: unset;

      @media (prefers-color-scheme: dark) {
        filter: invert(100%);
      }
    }
  }

Most of the buttons in Campfire contain an icon image plus a hidden text element for screen readers.

<%= form.button class: "btn btn--reversed center", type: "submit" do %>
  <%= image_tag "check.svg", aria: { hidden: "true" }, size: 20 %>
  <span class="for-screen-reader">Save changes</span>
<% end %>

With :has() our button class can know if these elements are present and turn it into a circle icon button with the image centered inside it. Notice that we’re using our --btn-size variable from earlier.

&:where(:has(.for-screen-reader):has(img)) {
  --btn-border-radius: 50%;
  --btn-padding: 0;

  aspect-ratio: 1;
  block-size: var(--btn-size);
  display: grid;
  inline-size: var(--btn-size);
  place-items: center;

  > * {
    grid-area: 1/1;
  }
}

Just dump whatever you want into .btn and it’ll take care of the rest.

That’s really satisfying to use as a developer but you could do this without a lot of extra effort using utility classes like .btn--circle-icon or .btn--icon-and-text. What really opened our eyes was when we were able to replace Ruby on Rails code with just CSS.

Take, for example, the menu button that toggles the sidebar when using Campfire with a narrow viewport.

Because the sidebar (which lists all of your chat rooms) is hidden when closed we wanted to display a small dot on the menu button to indicate that you have rooms with new, unread messages in them. Normally we’d have to write some Ruby on Rails code to handle that condition something like this:

<% if @room.memberships.unread.any? %>
  // render the dot
<% end %>

But with :has() we can do it with pure CSS alone!

#sidebar:where(:not([open]):has(.unread)) & {
  &::after {
    --size: 1em;

    aspect-ratio: 1;
    background-color: var(--color-negative);
    block-size: var(--size);
    border-radius: calc(var(--size) * 2);
    content: "";
    flex-shrink: 0;
    inline-size: var(--size);
    inset-block-start: calc(var(--size) / -4);
    inset-inline-end: calc(var(--size) / -4);
    position: absolute;
  }
}

Here the we’re querying the sidebar element to 1) make sure it isn’t open (because you don’t need to see the dot if you’re already looking at the rooms list) and 2) to see if it has any elements inside it that have the .unread class. If those are true, draw the dot and position it. Notice that we’re using a custom property (--size) here for both the dimensions of the dot and to calculate its border radius and position. It’s harmonious and avoids magic numbers.

Elsewhere, on Campfire’s account profile screen we used :has() to solve a problem that was nearly impossible to do even with server side code. The screen features a list of all the chat rooms you’re in and a button to toggle the state of each room. If you’ve made the room invisible in your sidebar we also wanted to be able to grey out the row to visually reinforce this critical status.

The problem is that toggle button is a completely separate element using a different controller, rendered in a Turbo Frame. It’s the same toggle we show in the room, itself. That means the code that renders the row has no idea what status of the button is, nor does it know when the status changes.

<li class="flex align-center gap margin-none min-width membership-item">
  <%= link_to room_path(membership.room), class: "overflow-ellipsis fill-shade txt-primary txt-undecorated" do %>
    <strong><%= room_display_name(membership.room) %></strong>
  <% end %>

  <hr class="separator" aria-hidden="true">

  <span class="txt-small">
    <%= turbo_frame_tag dom_id(membership.room, :involvement) do %>
      <%= button_to_change_involvement(membership.room, membership.involvement) %>
    <% end %>
  </span>
</li>

Now we could, of course, use Javascript to get the state, observe changes, and update the view. Or we could re-write this code to re-render the entire row when the notification state changes, but then we’d be writing a duplicate toggle that is only slightly different than the one used elsewhere.

A third option is to write a single CSS rule!

.membership-item:has(.btn.invisible) {
  opacity: 0.5;
}

If the row has a button in toggled to the .invisible class, dim it.

Advances in CSS have been slowing replacing Javascript code over the last few years, now it’s coming for server side code!

One more?

Campfire’s direct message feature, which we call Pings, displays all of your active conversations across the top of the sidebar. Depending on how many people are involved, Campfire displays one, two, three, or four avatars to represent the chat.

Normally our view template would need to count the number of participants and conditionally apply a class to the element so the CSS knows how to render each layout group. But with :has() we can effectively count the number of elements and adjust the display accordingly.

/* Four avatars */
.avatar__group {
  --avatar-size: 2.5ch;

  block-size: 5ch;
  display: grid;
  gap: 1px;
  grid-template-columns: 1fr 1fr;
  grid-template-rows: min-content;
  inline-size: 5ch;
  place-content: center;

  .avatar {
    margin: auto;
  }

  /* Two avatars */
  &:where(:has(> :last-child:nth-child(2))) {
    --avatar-size: 3.5ch;

    > :first-child {
      margin-block-end: 1.5ch;
      margin-inline-end: -0.75ch;
    }

    > :last-child {
      margin-block-start: 1.5ch;
      margin-inline-start: -0.75ch;
    }
  }

  /* Three avatars */
  &:where(:has(> :last-child:nth-child(3))) {
    > :last-child {
      margin-inline: 1.25ch -1.25ch;
    }
  }
}

Magic 🪄

Responsive design

In this last section, we’ll take a look at Campfire’s approach to responsive design. The first thing to know is that Campfire has zero/none/nada viewport based @media queries. There are no attempts to assert that viewports narrower than x are mobile devices. Campfire’s layout fully adapts to whichever device you’re using in whichever configuration or orientation, without attempting to declare any state as “mobile”. Here’s how.

Layout

Campfire has a single @media breakpoint—one value, used in a number of places.

@media (max-width: 100ch) {
  ...
}

This breakpoint largely determines how the CSS grid layout must adjust when the viewport is too narrow to display the sidebar alongside the chat transcript. When the document is narrower than 100 characters, it’s not practical to render them side-by-side, so instead Campfire hides the sidebar and reveals a menu button to toggle it.

Using characters as the unit of measure ensures that we get the right behavior no matter which device you’re using and in a number of other scenarios such as multitasking on iPad or even if you simply enlarge the font size past a certain point. Type is the heart of web pages so it makes sense for the layout to respond to it.

Feature enhancements

The other place we use media queries is to respond to the kind of input device the user has. It’s never been fair to assume a device with a narrow viewport has a touch screen, nor that a device with an enormous viewport does not. This blurry line is not getting clearer. But thanks to @media queries we can actually get useful information about a device’s capabilities. First up, any-hover.

@media (any-hover: hover) {
  &:where(:not(:active):hover) {
    /* hover effect */
  }
}

This queries the user’s device to see if it has any input mechanism that is capable of hovering (probably a mouse). It won’t match on touch screen devices and will opt out of Mobile Safari’s annoying behavior that makes you double-tap things that have a hover effect. Not bad.

But let’s look at something a little more impressive. Every message line in a Campfire chat has a ••• button that reveals a menu of extra actions (edit, Boost, copy, share) that you can do.

On devices with a mouse or trackpad the ideal is to only reveal the menu when you hover over the message but that would make it inaccessible on touch devices. No problem. We can use any-hover along with the pointer query to get the behavior we want on each kind of device.

@media (any-hover: hover) and (pointer: fine) {
  /* Reveal the button only on hover */
}

@media (any-hover: none) and (pointer: coarse) {
  /* Show the button all the time */
}

This is especially magical with a device like the iPad Pro. Which can match both queries under certain conditions, and change on-the-fly. When it’s docked on the Magic Keyboard with built-in trackpad, it matches the first query and the ••• buttons are hidden until you hover. Lift it off the Magic Keyboard and it becomes a purely touch device—the ••• buttons magically appear. It’s very cool.

What’s next?

Campfire 1.0 shipped in January 2024 and by March we had already started to work on the next ONCE product. While Campfire supported bleeding edge features when was released the web platform is rapidly changing and we’re already exploring new features that have gained browser support since then. It’s a fantastic time to be working on the web.

If you haven’t tried Campfire yet, it’s available now at once.com, the first of a family of products that you buy once, own forever (including source code), and can do what you want with.

Questions?

Have a question, comment or idea? Want to see more posts like this? Get in touch at jz@37signals.com or x.com/jasonzimdars

Thruster is a minimal HTTP/2 proxy server that we wrote to make it easier to serve a Rails application with great performance and security. It runs alongside your existing Puma process, and adds:

• HTTP/2 support
• SSL (via Let’s Encrypt, for automatic certificate management)
• HTTP caching, for public assets
• Efficient static file serving via X-Sendfile and compression

Puma already does a great job of serving Rails applications. Thruster is designed to give it an extra little boost, so you’ll have all you need for your application to perform well on the open Internet.

Why we built Thruster

The idea for Thruster came about when we started working on the ONCE project last year. We needed a way to package Rails applications that customers could run simply, by themselves. The applications should require very little effort to set up, and no ongoing maintenance, but they still need to be fast and secure.

We realized that – out of the box – Rails with Puma actually gets you pretty close, but on its own it’s not quite enough. To reach the performance you’d expect, you’d typically need to deploy Puma along with additional web servers, CDNs, and the like. You usually end up with a few moving parts to set up and maintain.

At a certain scale that’s still the right thing to do, but for a lot of applications, it feels like more complexity than you should need. And for our ONCE project, since our customers would be running the software themselves, it was more complexity than we could afford.

So we designed Thruster to be a simple, zero-config answer to those missing pieces we weren’t yet getting from Rails & Puma. Something that we could just drop in to a project, but otherwise not have to think about.

Although we initially designed it for the ONCE project, we’ve since found it useful when deploying other Rails applications too.

How Thruster works

Thruster wraps your Puma process so that you don’t have to worry about running multiple processes or configuring them to know about each other.

Typically you’d start your Rails application with something like:

rails server

To run the same command with Thruster, you’d use:

thrust rails server

This starts the Thruster process, which in turn starts your Rails application. If your application quits, Thruster will quit too. Their lifetimes are tied together. This is particularly useful when running in a container environment, since you can simply prefix your existing CMD with thrust, and still get the same container restart behavior that you had before.

To enable automatic SSL, you just need to tell Thruster which domain it should accept traffic for. You do that with the SSL_DOMAIN environment variable. For example:

SSL_DOMAIN=example.com thrust rails server

As long as you have a valid DNS record pointing that domain to your application, Thruster will take care of provisioning and renewing the SSL certificates as needed.

You can also use Thruster in an environment where you only need some of its features. For example, if you already have SSL termination taken care of, you can still use Thruster for its caching. Or if you’re already using a CDN, you can still use Thruster to get rapid serving of private static files via X-Sendfile.

You can install Thruster by adding its gem to your Gemfile, and you can dive into the code on GitHub.

We hope you find it useful!

The beginning

Everything started with a report from our Operations team mentioning that HEY memory usage was often getting close to boiling over. We rarely deploy HEY during the weekends, and that’s when the issue manifested itself the most:

The leak

After a few days without deploying the app, the memory usage almost reached 100%, and then a deployment would reset it back to normal. This signals a slow and steady memory leak.

Checking Ruby heap allocations

The first approach I tried was logging and analyzing the Ruby Heap allocation stats. In particular, I looked for endpoints where:

1. A minor (or major) GC run happened.
2. heap_available_slots increased greatly.

The theory is that when GC runs, it should clear most of the newly allocated objects, hence an increase of heap_available_slots paired with a GC run shouldn’t happen unless there is a leak.

The analysis exposed 2 endpoints:

• MessagesController#update
• TopicsController#show

These are the most heavily used endpoints of the app, so the result wasn’t conclusive at all.

Heap dump

I decided to proceed by extracting a Ruby heap dump. Luckily this can be done by running rbtrace in the app servers:

bundle exec rbtrace -p $worker_pid -e 'Thread.new{GC.start; require "objspace"; File.open("/tmp/0.json","w"){|f| ObjectSpace.dump_all(output: f) }}'

This command writes the $worker_pid heap dump to /tmp/0.json.

Unfortunately, a heap dump extracted without enabling object trace allocation doesn’t contain key information such as:

• The GC generation it was allocated in
• The filename and line number it was allocated in
• A truncated value
• Object bytesize

So even the basic heap dump wasn’t enough to find the culprit.

ObjectSpace.trace_object_allocations_start

Enabling object trace allocation greatly slows down the response times and increases memory usage but I had no more options left, so I still decided to proceed. To mitigate this performance degradation, I enabled object trace allocation only on one app host and only for the time needed to extract the heap dumps I needed. I also monitored the host during the process: If something went wrong, I could’ve immediately stopped the host traffic from the load balancer to mitigate it.

I managed to extract 3 heap dumps without any issues:

1. 0.json: ~15min after deploy
2. 1.json: ~40min after deploy
3. 2.json: ~1h after the deploy

It was time to analyze them!

Analysis

The first analysis was via heapy diff:

jacopo-37s-mb 3.3.0-preview2 ~/Desktop/Hey heap dumps/1-tracing heapy diff 0.json 1.json 2.json |head -n 10
Retained STRING 18947 objects of size 1290614/16768824 (in bytes) at: /usr/local/bundle/ruby/3.3.0/bundler/gems/okra-d3937f2a023c/lib/okra/html.rb:19
Retained OBJECT 14380 objects of size 1150400/16768824 (in bytes) at: /usr/local/bundle/ruby/3.3.0/bundler/gems/okra-d3937f2a023c/lib/okra/html.rb:19
Retained ARRAY 13588 objects of size 608192/16768824 (in bytes) at: /usr/local/bundle/ruby/3.3.0/bundler/gems/okra-d3937f2a023c/lib/okra/html.rb:19
Retained STRING 3634 objects of size 247720/16768824 (in bytes) at: /usr/local/bundle/ruby/3.3.0/gems/prometheus-client-mmap-1.0.0/lib/prometheus/client/histogram.rb:21
Retained STRING 2895 objects of size 939094/16768824 (in bytes) at: /usr/local/bundle/ruby/3.3.0/gems/json-2.6.3/lib/json/common.rb:312

Which outlined lib/okra/html.rb:19, and I thought this was the root cause; but after digging further, I wasn’t able to find any leaking code related. Luckily, the heap dumps extracted with object trace allocation enabled contained the object dependency tree, so I decided to take a look at it and figure out what was retaining lib/okra/html.rb:19; and for that, I used sheap.

Initially, I extracted from the heap dump a few addresses associated with lib/okra/html.rb:19:

jacopo-37s-mb 3.3.0 ~/Desktop/Hey heap dumps/1-tracing grep "file.*lib/okra/html.rb.*line.*19" 2.json | head |  grep -o "address\":\"[^\"]\+\""
address":"0x7fbfec2c0098"
address":"0x7fbfec2c0110"
address":"0x7fbfec2c0138"
address":"0x7fbfec2c0408"
address":"0x7fbfec2c0430"
address":"0x7fbfec2c0480"
address":"0x7fbfec2c04d0"
address":"0x7fbfec2c04f8"
address":"0x7fbfec2c0908"
address":"0x7fbfec2c0980"

Then, I checked their dependency tree with sheap, and found a common pattern:

jacopo-37s-mb 3.3.0 ~/Desktop/Hey heap dumps/1-tracing sheap 0.json 2.json
irb#1():004> $diff.after.find_path($diff.after.at("0x7fbfec2c0098"))
=>
[#<ROOT vm  (3194 refs)>,
 #<DATA 0x7fc02024b070 yjit_root (11268 refs)>,
 #<IMEMO 0x7fbfecc9b4a0 ment (4 refs)>,
 #<IMEMO 0x7fbffd0b2948 iseq (111 refs)>,
 #<OBJECT 0x7fbfbb8b53d0 (0x7fbff0a537c0) (8 refs)>,
 #<OBJECT 0x7fbfecd3e150 ActiveModel::AttributeMutationTracker (2 refs)>,
 #<OBJECT 0x7fc01ab32900 ActiveModel::LazyAttributeSet (6 refs)>,
 #<HASH 0x7fbff0f58b80  (4 refs)>,
 #<OBJECT 0x7fbff0f56ec0 ActionText::Content (1 refs)>,
 #<OBJECT 0x7fbfec64da08 ActionText::Fragment (1 refs)>,
 #<OBJECT 0x7fc007f95140 Okra::HTML::Node (1 refs)>,
 #<ARRAY 0x7fc01ab32040  (7 refs)>,
 #<OBJECT 0x7fc007f95190 Okra::HTML::Node (3 refs)>,
 #<ARRAY 0x7fbfc485d960  (2 refs)>,
 #<OBJECT 0x7fc007f95230 Okra::HTML::Node (3 refs)>,
 #<ARRAY 0x7fbfc485d870  (3 refs)>,
 #<OBJECT 0x7fc007f95410 Okra::HTML::Node (3 refs)>,
 #<ARRAY 0x7fbfbb8b5150  (21 refs)>,
 #<OBJECT 0x7fc007f95690 Okra::HTML::Node (3 refs)>,
 #<ARRAY 0x7fbfec37e4f8  (3 refs)>,
 #<OBJECT 0x7fc007f95780 Okra::HTML::Node (3 refs)>,
 #<ARRAY 0x7fbff09af5a8  (11 refs)>,
 #<OBJECT 0x7fc007f959b0 Okra::HTML::Node (3 refs)>,
 #<ARRAY 0x7fbff09ae9c8  (11 refs)>,
 #<OBJECT 0x7fc007f95e10 Okra::HTML::Node (3 refs)>,
 #<ARRAY 0x7fbfec378530  (3 refs)>,
 #<OBJECT 0x7fc007f96040 Okra::HTML::Node (3 refs)>,
 #<ARRAY 0x7fbfec378490  (3 refs)>,
 #<OBJECT 0x7fc007f96180 Okra::HTML::Node (3 refs)>,
 #<ARRAY 0x7fc0202ff520  (39 refs)>,
 #<OBJECT 0x7fc007f96310 Okra::HTML::Node (3 refs)>,
 #<ARRAY 0x7fbfec36acf0  (3 refs)>,
 #<OBJECT 0x7fc007f96400 Okra::HTML::Node (3 refs)>,
 #<ARRAY 0x7fbff09a4ba8  (11 refs)>,
 #<OBJECT 0x7fc007f96540 Okra::HTML::Node (3 refs)>,
 #<ARRAY 0x7fbfec331e50  (3 refs)>,
 #<OBJECT 0x7fc007f966d0 Okra::HTML::Node (3 refs)>,
 #<ARRAY 0x7fbfec331dd8  (3 refs)>,
 #<OBJECT 0x7fc007f96810 Okra::HTML::Node (3 refs)>,
 #<ARRAY 0x7fc0202fe120  (39 refs)>,
 #<OBJECT 0x7fc007f96a40 Okra::HTML::Node (3 refs)>,
 #<ARRAY 0x7fbfec311ad8  (3 refs)>,
 #<OBJECT 0x7fc007f96b30 Okra::HTML::Node (3 refs)>,
 #<ARRAY 0x7fbff0999118  (11 refs)>,
 #<OBJECT 0x7fc007f96db0 Okra::HTML::Node (3 refs)>,
 #<ARRAY 0x7fbfec300800  (3 refs)>,
 #<OBJECT 0x7fc007f96f40 Okra::HTML::Node (3 refs)>,
 #<ARRAY 0x7fbfec3007b0  (3 refs)>,
 #<OBJECT 0x7fc007f97080 Okra::HTML::Node (3 refs)>,
 #<ARRAY 0x7fbfbe42eed0  (21 refs)>,
 #<OBJECT 0x7fc007f971c0 Okra::HTML::Node (3 refs)>,
 #<ARRAY 0x7fbfec2ca138  (3 refs)>,
 #<OBJECT 0x7fc007f972b0 Okra::HTML::Node (3 refs)>,
 #<ARRAY 0x7fbff0993538  (11 refs)>,
 #<OBJECT 0x7fc01b8fdb20 Okra::HTML::Node (3 refs)>,
 #<ARRAY 0x7fbfec2c2b90  (3 refs)>,
 #<OBJECT 0x7fc01b8fdcb0 Okra::HTML::Node (3 refs)>,
 #<ARRAY 0x7fbfec2c0138  (2 refs)>,
 #<ARRAY 0x7fbfec49ff30  (2 refs)>,
 #<STRING 0x7fbfec2c0098 "MsoNormal">]

All these objects were retained by YJIT, this is clearly shown by the yjit_root node in the top of the dependency tree! After finding the root cause I opened an issue for the YJIT team, which promptly fixed it.

Conclusion

The current tooling in Ruby to troubleshoot memory leaks is pretty advanced. Generally, an effective approach is to use rbtrace, to extract a heap dump (with object trace allocation enabled); and then analyze it via heapy, sheap, or any other similar tool.

Happy hunting!