Blog
2025 Year in Review
We are wrapping up the third year of SadServers, this 2025 has been our best one yet and I wanted to recap some of the things that happened. The number of Linux and DevOps scenarios almost doubled, there are close to 100 challenges now. We also introduced new scenarios around applications like Clickhouse, Harbor, Podman or Vault. The number of SadServers users have more than doubled. As a side effect there were times where for example we were running out of AWS spot instances in some AZs. We’ve had no major incidents though, uptime has been close to 100%. This doesn’t mean we didn’t make mistakes and that could be a good topic for a separate blog post. There are many new features introduced this year like the achievements page, the scenario invites, the command line history logs or the gift page. One system we decommissioned because it wasn’t used much and it was hard to maintain was the Asciinema replays. We also introduced the Business accounts, for companies doing DevOps assessments, either for internal training or for interviewing job candidates. We keep working on the next iteration of SadServers, the “SRE Simulator”. This is mostly work beign done for a separate product but in SadServers we use the system for the Kubernetes sandboxes (still PoC) and also some scenarios run as k8s pods on the new platform. Of course I have to mention AI. We use products like Cursor or Copilot for coding with mixed results and who knows what even the near future will bring. What we know is that there is a race by both established and startup companies working on AI agents for SRE. SadServers is a great platform for these AI companies to evaluate their products. Earlier in December we ran the “Advent of Sysadmin”; 12 days with a new scenario every day. This event turned out to be a huge success with about 10,000 people participating. I’m looking forward to 2026; my team and I are very excited for the new projects we are working on, as always focused on bringing value to our users.
December 22, 2025
Destroy and Deploy: the Joys of Immutability
Brought to you by SadServers – Hands-On Linux & DevOps; Real Challenges. Real Infra. Real Skills. Sysadmin Friend: “Hey, look, my application server in production has an uptime of over a thousand days! Talk about stability eh?” Me: “Well, that’s actually not a good thing.” Sysadmin Friend: “What!?” Me: “Sorry, let me tell you a grandpa story.” In the old, pre-cloud days, we would make changes in our server (upgrades, application deployments) half-manually, typically running our own custom Bash scripts in big servers named after Star Wars characters. These were our “pets” or “snowflakes” because they were hard to reproduce from scratch and we didn’t want to touch them too much, afraid they could die from a light breeze. If the sysadmin tending to the pet left the company, and as usual there was no good documentation, then, well, there would be a world of pain.
October 11, 2025
Test Your Infrastructure with Game Days
Brought to you by SadServers – Hands-On Linux & DevOps; Real Challenges. Real Infra. Real Skills. This article was published in the book 97 Things Every Cloud Engineer Should Know by O’Reilly. What is a Game Day “You don’t have a backup until you have performed a restore” is a good aphorism and in a similar way, we can say that your service or infrastructure is not fully resilient if you haven’t tried breaking it and recovering.
October 8, 2025
Don't SSH into Production
Brought to you by SadServers – Hands-On Linux & DevOps; Real Challenges. Real Infra. Real Skills. This article was published in the book 97 Things Every Cloud Engineer Should Know by O’Reilly. (I’m very aware of the apparent contradiction of publishing this in SadServers, a service based - so far - on working with an “SSH” terminal). Routine server system administration tasks should be handled with automation and services; through code and software.
October 4, 2025
Migrating Kubernetes out of the Big Cloud Providers
Brought to you by SadServers – Hands-On Linux & DevOps; Real Challenges. Real Infra. Real Skills. “Move to kubernetes to save costs” they said in the early days of the k8s frenzy. This was trusting that an efficient pod bin (node) packing would save on node costs (there’s also autoscale but regular cloud already offers that). The reality is that the overhead costs of running the control plane and auxiliary services in each node (DNS, metric and log collectors etc) plus extra easy ways to make costly mistakes turns most Kubernetes installations a more expensive proposition than running the workloads without it.
September 24, 2025
Technical Interviews: We Can do Better
Brought to you by SadServers – Hands-On Linux & DevOps; Real Challenges. Real Infra. Real Skills. Hiring is very expensive for companies (and for candidates too); it not only takes a lot of employees’ time to go through multiple rounds of interviews with several candidates, but even worse, hiring a bad engineer - only to let them go weeks or months later - sets everything back, including morale. And yet, companies and teams spend a surprisingly small amount of time preparing good interviews.
June 21, 2025
What the F is DevOps?
Brought to you by SadServers – Hands-On Linux & DevOps; Real Challenges. Real Infra. Real Skills. Who knows? Whenever I’ve talked to a recruiter or hiring manager, often the first minutes are dedicated to explaining and scope what the role of the DevOps team is. Origin of DevOps Everybody knows this right? In the traditional way of working, there were developers writing code and sysadmins taking care of servers, doing operations. Software engineers would “chuck their code over the wall” to the sysadmins and these sysadmins would deploy the code. Developers want to make changes and sysadmins want stability (“if it works don’t touch it”) thus generating a conflict of interests. “Developer Operations” or “DevOps” was born to break down that wall, or so goes the legend.
October 31, 2024
Automation Is a Panacea
Brought to you by SadServers – Hands-On Linux & DevOps; Real Challenges. Real Infra. Real Skills. Panacea noun a solution or remedy for all difficulties or diseases. Obviously there are no panaceas, let’s downgrade it to “something that is good to do in almost all circumstances”.
December 21, 2023
