In this edition:

• Graham wonders whether the idea of a hierarchical file system makes sense at all in 2026.
• Adrian goes back in time to find the origin of the “everything is a file” moniker.
• In our Vidéothèque section, Adrian reviews a documentary about the Unix file system on the AT&T Tech Channel.
• In the Library section, Graham reviews the Apple Computer, Inc. developer manual explaining the file system in the 1990s.

Download this issue in DRM-free PDF or EPUB format, and read it on your preferred device. You can also subscribe to our RSS feed, featuring the full content of our articles.

We would like to thank our patrons who generously contribute every month (or have contributed in the past) to our work and help us run this magazine. Thank you so much! In alphabetical order: Adam Guest, Adrian Tineo Cabello, Benjamin Sheldon, Christopher Nascone, Colin Powell, Franz Lucien Moersdorf, Guillermo Ramos Álvarez, Jean-Paul de Vooght, Dr. Juande Santander-Vela, Patryk Matuszewski, Paul Hudson, Quico Moya, Roger Turner, Szymon Licau, and countless more leaving anonymous tips every month.

Enjoy this issue! Please share our articles on social media, or contribute if you would like to support our work with a donation via Liberapay.

Cover photo by Maksym Kaharlytskyi on Unsplash.

Early examples of such file systems have a single level of structure; you create directories at the top level, and files in those directories (this actually means that there are two levels of directory, because there must be a top-level index to find the directories you create). These have one of two purposes:

1. Help a person organise their files in a way that makes it easier to find them.
2. Keep multiple people who use a time-sharing system from getting in each others’ way, by giving them separate areas to store their files.

In 1965, someone on the Multics project notices that the laws of physics do not apply to computers, and you can fit a folder inside another folder. The arbitrarily-nested hierarchical file system is born. Now users on Multics and its progeny, Unix, can organise their files, and do so while keeping out of each others’ way! Access control stops any account reading, writing, or deleting another account’s files, unless someone who can change the rules gives that account explicit permission.

Unfortunately, as anybody who ever deleted /vmunix or C:\COMMAND.COM knows, saving user accounts from each other is not the whole story. We also need to save the operating system from the people using the computer. This means a new requirement:

3. Keep multiple people who use a time-sharing system from interfering with the operating system, by making operating system files inaccessible from regular accounts.

But because we already shipped the multi-user filesystem and designed the file-access APIs around it, we cannot add this requirement to the model. So we invent a fiction: the operating system is another user account, or a collection of user accounts, that share the filesystem with the actual people who use the computer. So maybe the mail server has its own user account, and the print spool daemon, and so on.

Unfortunately, even this does not solve the problem, because the person who runs the computer needs to be able to configure, patch, and update the operating system. That person then gets access to an account that can work with all the files on the filesystem, starting at the root of the whole hierarchy: call it the root account.

They get that access, but they do not need that access, and indeed they should not have that access, because a person with the root account can violate requirements 2 and 3: they can get in another user’s way, and they can interfere with the operating system. Ideally the filesystem would help with requirement 1 and strictly enforce requirements 2-3, but the user-permissions model does not lend itself to that possibility. So add another layer to the cake: mandatory access controls stop people from fiddling with particular files, even if their user account has discretionary permission to operate on those files.

Unfortunately, computers no longer exist in isolation (nor did they at the time; but we assumed that someone who connected up a tape drive and spooled the contents of the tape into their local installation had a reason for doing so). People download files from the internet, and they download files from the internet that are programs, and they expect to run those programs, and they do not want for one bad (or well-intentioned but poorly coded) program to break the files they use in other programs, or even to read those files unless that is their explicit intent. Time to add a new requirement:

4. Keep the documents that a person creates using a computer private and intact, even when that person runs arbitrary software.

Mandatory access control no longer cuts it; now the filesystem needs a dynamic level of protection, in which files created by different processes are maybe usable from other processes run by the same user account, but maybe not. It depends on what those processes are, and what the files are, too. And where the files came from, and whether the person using the computer intended to get the file, or whether the process they ran just sorta created it for them without telling them. And also, those processes want to keep their own files, to cache downloads, to store the working set from complex computations, to save configuration options, and for whatever purposes.

5. Keep the files that arbitrary software creates private and intact, even when a person uses that or other arbitrary software.

Oh, and by the way, while all of that is going on, there are now so many files that even requirement 1 is long forgotten. People cannot remember whether that important report due tomorrow is in their Desktop folder, or in Desktop/Desktop.old, or in Desktop/Desktop.old/from_dell, or in their Documents folder, or in some unique folder that is managed by whichever app it is they use to edit the file.

6. Help a person to find documents in a labyrinthine structure of folders that they do not understand.

…it is always possible that they accidentally deleted the file they need, or that they kept the file but deleted a critical section of the report that, on reflection, they realised the audience is actually going to care about.

7. Help a person to browse and restore previous versions of a file, and deleted files.

…or maybe they were just saving the file as the battery on their laptop finally discharged?

8. Provide resilience to common hardware failures that lead to incorrectly or partially committed filesystem changes.

…or, no, wait, I remember, the correct version of the report is on my desktop, I have been editing the version on my laptop, I will get that other version once it syncs via my cloud provider.

9. (to a high number, probably around 94). Act as a cache to a remote file storage system; maintain most recently-used files locally while still evicting files as needed to make space for other files a person requests; resolve version conflicts when the remote version and the local version both have changes since the last sync point; resolve conflicts when one version has changes and the other version is deleted; retain confidentiality and integrity over the remote sync interface…

And we need to go back and redefine the idea of “users of a time-sharing system” that we mentioned in requirements 2-3, and invented in about 1965. You see, those time-sharing systems have changed somewhat. Some are little portable devices, so the person whose “account” is “logged in” gives them to a passing acquaintance to share a fun meme they saw. Some are big wall-mounted devices, so the person is using the system alongside six other people who can all see and interact with it.

But yeah, sure, I guess it is neat that the last time we improved the file system was ZFS circa 2005, and that the other file systems are still catching up.

Photo by Jack Krzysik on Unsplash.

CTSS’s fundamental model of human-computer interaction (…) can still be found today at the heart of MS-DOS, Windows, Macintosh, Linux, and many other operating systems. So can such innovations as CTSS’s “hierarchical” file system, a scheme devised by Corbató and his team to enable individual groups and individual users to organize their personal files into “directories” and “subdirectories” on the 7094’s hard disks. (In modern-day Windows and Macintosh systems, of course, such directories are called folders.)

Waldrop forgot to mention Linux, itself a derivative of Unix, where a quick man 7 hier command will display the complete description of the filesystem hierarchy, finally providing some light on the intricacies of the various /etc, /usr, /home, and /var folders.

Beyond the mere organization of data, the Unix lineage of operating systems took Corbató’s hierarchical concept and elevated it into something akin to an arcane voodoo artiluge. This is true of Linux, macOS, and the BSDs, but not so much Android and iOS, because let us be honest, it is quite tough to use a command line with these last two, and iOS’s “File” application is an absolute clusterfuck of an idea.

This “feature” of a voodoo-like file system is probably one of the most salient aspects of Unix-based systems, and it is captured in a famous, four-word axiom of the Unix philosophy: everything is a file. This is one of those phrases that have echoed throughout the ages, similar to you are doing it wrong or the network is the computer; or at least, it did until file systems disappeared beneath the rugged interface of an S3 bucket somewhere in 2005.

To understand the reason behind this idea, we have to look back to the creators of Unix at Bell Labs: Ken Thompson, Dennis Ritchie, and Douglas McIlroy. When they were building Unix from the ashes of the overly complex Multics project, they sought simplicity. Instead of creating a dozen different APIs to talk to a hard drive, a keyboard, a printer, and a network connection, they created just one.

Scarcity is the actual mother of invention, because nobody needed this, really.

They decided that if an abstraction worked for a text document, it should work for everything. (I know, stay with me.) In their worldview, a file was simply a stream of bytes. Consequently, the same system calls used to manipulate a text file (open(), read(), write(), and close()) could be used to interact with virtually every component of the computer. Because why not.

(See? The world was much simpler back in the day.)

Take a walk through the /dev (device) directory on any modern Linux or macOS machine, and you are no longer looking at data stored on a magnetic platter or solid-state drive; you are looking at the hardware itself, masquerading as files!

It goes even deeper. When you type on your keyboard or watch text scroll on your screen, you are interacting with a TTY device (like /dev/tty), a direct descendant of the physical Teletype machines of the 1970s. If you wanted to send text directly to a parallel-port printer in the early days of Linux, you did not need a dedicated printing application; you simply redirected your text into the printer’s file: echo "Hello World" > /dev/lp0. Of course, you would still need a printer, which can be a problem in and of itself.

There are “block devices” like /dev/sda or /dev/nvme0n1, which represent your actual physical hard drives. You can read them directly, bypassing the traditional file system entirely, simply by pointing a tool like cat or dd at them.

Then there are the esoteric, conceptual devices. Need an endless stream of zeroes to wipe a hard drive? Read from /dev/zero. Need a place to send error messages so you never have to see them again? Throw them into the abyss of /dev/null, the file system’s built-in black hole. Need cryptographic randomness? Just read from /dev/urandom, where the kernel distills the chaotic entropy of hardware interrupts into a stream of random bytes. Try it by yourself: run cat /dev/urandom | hexdump -C | head -n 5 in your nearest terminal (well, do not try it if you are running Windows) and you will see some… well, random garbage printed out.

The “everything is a file” philosophy enters the realm of LSD when we look at Inter-Process Communication, or “IPC”. How do two completely isolated programs talk to each other in Unix? By reading and writing to files, of course!

The most iconic manifestation of this is the pipe (represented by the | character in your shell). Conceived by Douglas McIlroy, the pipe takes the standard output (file descriptor 1) of one program and connects it to the standard input (file descriptor 0) of another. But Unix also allows for named pipes: again, you guessed it: persistent files residing on the disk that act as FIFO (First In, First Out) queues. One program opens the file and writes to it, another opens the file and reads from it. To the programs involved, they are just doing standard file I/O.

This abstraction scales up to networking. When you browse the web, your browser communicates over a network socket. In Unix, that socket is treated as a file descriptor. The program writes its HTTP request to the “file” and reads the web server’s response from that same “file.” The underlying networking stack (TCP handshakes, IP routing, Ethernet frames) is completely hidden behind the familiar read() and write() interface. Even complex virtualization tools like Docker rely on this, listening for instructions via a Unix domain socket, usually located at /var/run/docker.sock. Anyone reading this who has ever debugged REST APIs using curl knows this mechanism very well.

If Unix established the “everything is a file” paradigm, the research operating system Plan 9 from Bell Labs perfected it, and modern Linux heavily borrowed from there, taking it from LSD to full-on Ayahuasca level.

Boot up a Linux machine and navigate to /proc. The files you see there take up exactly zero bytes on your hard drive. They exist entirely in the computer’s RAM, dynamically generated by the kernel on the fly. Magic!

The /proc (process) filesystem is an illusion that maps the internal state of the operating system into the file hierarchy. Want to know what CPU you are running? You do not need a specialized hardware diagnostic tool; you just type cat /proc/cpuinfo. Want to see how much memory is free? cat /proc/meminfo.

Every running program on your computer has a dedicated directory in /proc named after its Process ID (PID). Inside that directory are files containing everything from the exact command used to launch the program, to the environmental variables it holds, to the specific areas of memory it is currently using.

Similarly, the /sys (sysfs) directory provides a window into the Linux kernel’s device model. If you are on a Linux laptop and want to check your battery percentage, you do not need a graphical widget. Oh noes, that is for simpleton users. No, you, my dear IT expert, you can simply cat /sys/class/power_supply/BAT0/capacity. Want to change the brightness of your screen? Just write a new number into /sys/class/backlight/intel_backlight/brightness. Und so weiter.

Good luck remembering all of those commands, but hey! Now you know you can.

The Unix file abstraction is considered by many one of the greatest engineering triumphs in software history. Sadly, those “many” are not simple computer users, but actual experts of the industry. Of course, if you look at it closely, yes, it enables composability. Keyboards, displays, network connections, random number generators, and kernel diagnostics all share the exact same interface as a simple text file, making this standard toolkit universally applicable.

I personally think that the Unix file system analogy is not enough. We should be able to listen to Mark Snow’s famous “X-Files Theme” by just typing cat /music/spotify/track/3pDZSfOSszYWMYygDiuPJx. We should be able to swipe Tinder profiles using left /social/tinder/34564576891457. We should be able to search and buy plane tickets with fly /AF/CDG/PPT/business/2027-03-02/2027-04-10. You get the idea. And to be honest, none of the scenarios above would be too complicated to implement (well, except the airline thing, really).

But the truth is that in terms of usability, let us be honest, it is a terrible system. Even for people from the world of IT, it is usually impossible to actually remember all of the myriad combinations available, and which one represents what. You might have a few in mind, or you might create an alias to quickly type one or the other, but the fact that “everything is a file” is barely more than a curiosity to pretty much everyone. Add some complexity points if you switch from Linux to FreeBSD, and suddenly you must cat for files in other locations.

As I said at the beginning, we have not cared about file systems for more than 20 years now. (Remember when you could not have spaces in DOS filenames until circa 1995? If you can, I have bad news for you: you are old.) In Linux, arguably the most nerdy of all operating systems, you might still be choosing between ZFS, EXT, BTRFS, or XFS when formatting a new drive, but nobody cares anymore, really. Windows users are fine with NTFS, Mac users do not even know what APFS is. Gone are the days of using fdisk. Nostalgic OS/2 users might even remember HPFS and shed a tear. Most users probably even use some kind of encryption on top of their file system, just in case. And some ultra-nerd Linux users might be even be interested in Gobolinux, a distribution that literally reorganized the whole hierarchy for… reasons.

These days people are more interested in knowing whether we have a fast SSD in our home computer, or if we have properly configured Ceph, Gluster, or Longhorn within our Kubernetes cluster. And if all else fails, you can always format your USB thumbdrives with FAT32 and make sure that virtually every operating system on Earth can read and write to it.

But whatever you do, remember that this article only caters to systems deriving from Unix, and in particular, do not even think of creating a file named CON on your Windows hard disk. You have been warned.

Cover photo by Alexey Demidov on Unsplash.

The video is a masterpiece of technical communication, explaining everything from pipelines to the YACC parser generator. But for our purposes this month, as we delve into the nature of file systems, there is a specific segment of this documentary that deserves our attention. It reminds us of a time when data storage was a field ripe for disruption.

To truly understand the revolution that the Unix file system represented, one must first understand the bureaucratic nightmare of the systems that preceded it. In the days of the IBM OS/360 (a system the documentary slyly critiques by quoting Fred Brooks’ estimation that it required “5,000 staff-years of effort” to produce) data was not merely stored; it was administered. Files were heavily structured entities. A programmer had to define record lengths, block sizes, access methods (sequential, indexed, direct), and physical disk geometries. You had to negotiate fiercely with the operating system just to write a string of characters to a persistent medium.

(COBOL developers enjoying their well-deserved retirement while reading this magazine will surely remember that you can specify such attributes when creating files in mainframes. Ah, the good old times.)

The narrator of the documentary captures this historical friction perfectly:

“In many systems, you have to say an awful lot about a file before you can do anything with it. You have to say where it is, and how big it is, and what kind of information that’s going to be in it—all kinds of things that are basically, utterly, completely irrelevant.”

Notice the epistemological shift contained in that last word. Irrelevant. What was considered the very substance of systems programming by the establishment was dismissed by the Bell Labs team as unnecessary friction. This is what rebellion tastes like in the world of hackers: Unix was a masterpiece of reductionism. It stripped away the metadata, the proprietary formatting, and the structural baggage, presenting a radically new, universally applicable definition:

“A file is simply a sequence of bytes. Its main attribute is its size. By contrast, in more conventional systems, a file has a dozen or so attributes to specify or create a file; it takes endless amounts of chitchat. If you want a Unix system file, you simply ask for a file and you can use it interchangeably wherever you want a file.”

A file is simply a sequence of bytes. This single sentence is arguably one of the most important conceptual leaps in the history of software engineering. By rendering the file formatless, the operating system relinquished the responsibility of understanding the data. The OS became a dumb, reliable conduit; the interpretation of those bytes was left entirely to the user-space programs.

This separation of concerns is the bedrock upon which the entire Unix philosophy was built. Before exploring the deeper implications of this design, let us consider the practical demonstration Kernighan provides in the video. To build a spellchecker, he does not write a monolithic C program. Instead, he orchestrates a ballet of small, focused utilities. He pipes a sentence into a program to break it into individual words, then into another to convert them to lowercase, then to sort, then to uniq to remove duplicates, and finally to a program that checks the stream against a dictionary.

(This demo will also be used by generations of IT experts to dismiss the bloat of word-processing software entirely. Whether they are right or wrong depends on whether your income depends on finishing that report before the meeting.)

“So what we have here is five separate programs cooperating to do this job, and in one giant pipeline,” Kernighan explains, noting proudly that he accomplished a complex task without writing any new code.

This seemingly simple trick is entirely dependent on the file system’s minimalist architecture. Because the file system imposes no record structure and no proprietary encoding, text becomes the universal interface. The output of sort is just a sequence of bytes separated by newlines; uniq natively understands this sequence because it expects nothing else. If Unix files had retained the heavy metadata of the mainframes, this pipeline would have required endless format conversions and API negotiations at every step.

But a sea of formatless files is useless without organization. Here, Unix introduced its next stroke of genius: the hierarchical directory structure.

“The Unix system consists of a hierarchy of directories. A directory is simply a file that contains the names of either other directories or files, and this whole thing goes on recursively.”

Again, we see the principle of recursive simplicity at play. A directory is not a special, magical entity with its own esoteric API; it is simply a file. It is a file that the operating system happens to read to figure out the mapping of names to underlying identifiers. Because a directory is just a file, the structure can nest infinitely. Every folder on your modern desktop, every directory on a cloud server, and every path in a URL traces its lineage directly back to this exact abstraction.

Yet, the Unix file system wonders go beyond mere data storage. It is the concept of device independence: the realization that if a file is just a sequence of bytes, then hardware peripherals can be treated as files too.

“In addition, the I/O devices, the peripheral devices connected to the computer, are also files in the file system. For example, the line printer, and the tape drive, and even the thing that dials telephone numbers, are all devices in the file system. And the same program that will copy information from one disk file to another disk file will also copy information from a disk file to the line printer.”

This is a moment of orthogonal perfection. You do not need a special “print” API or a “modem” API. You just redirect the byte stream. The tools do not know, and do not care, whether the bytes are resting on a magnetic platter, being displayed on a cathode-ray tube, or being punched onto paper by a line printer. The file system acts as the universal namespace, the grand unified theory of the machine’s resources.

Watching this 1982 artifact from our vantage point today feels almost melancholic. We live in an era where the simplicity of “a file is a sequence of bytes” is increasingly obfuscated. We build walled gardens on mobile apps, social networks, and LLMs, where applications cannot see each other’s data, actively subverting the shared namespace of a local file system. We rely on opaque cloud storage buckets, proprietary databases, and complex APIs that require reams of authentication tokens and structured JSON payloads just to persist a piece of text.

We have, in many ways, returned to the “endless amounts of chitchat” that Thompson and Ritchie sought to abolish. Le sigh.

Yet, underneath all our modern layers of abstraction (beneath the containers, the hypervisors, and the distributed key-value stores) the Unix file system remains. The heart of the system is still beating. There is an enduring power in getting the fundamentals right. When you strip away the noise, the hubris, and the bureaucratic complexity of software engineering, what remains is the silent, elegant truth: it is all, in the end, just a sequence of bytes.

Watch “AT&T Archives: The UNIX Operating System” on YouTube, a website that sadly does not allow you to save videos locally as a file on your computer.

Cover snapshot chosen by the author.

What good technical documentation does need to do is five things. Four of these are expressed clearly in the Diátaxis framework, and I have augmented that to fill in the obvious gap. Those things are:

1. Guide you through learning the technology.
2. Give you recipes for solving particular problems.
3. Help you understand how the technology works.
4. Provide a reference to the technology’s capabilities.
5. Create a common model that you share with the people who build, support, and manage the technology.

People use the different types of documentation to different extents; when someone says that a particular technology “has good documentation”, they typically mean that it has the types of documentation they tend to lean on, presented in ways they find usable. When someone says that a particular technology “has bad documentation”, they either mean that the types of documentation they would like to see are missing, or they exist but are not to their liking.

This means it is possible for people to talk across purposes about the same collection of documentation, not seeing the other’s point of view because they get different things out of the documentation. So one person might think their project has great docs because they have a lot of fantastic how-to articles and reference, but another person wishes they would add conceptual guidance because they do not want to know how to complete a task, they want to think about how they might achieve any task.

Inside Macintosh, the official source of documentation for Mac software development in the classic era, caters to (almost) everyone. This being the De Programmatica Ipsum issue on file systems, I will dive into the Inside Macintosh: Files volume for a closer look. For those who are reading along at home, I am using the (printed and bound, thank you very much, by Addison-Wesley) edition that corresponds to version 7.0 of the Macintosh System.

The first chapter, Introduction to File Management, includes conceptual guidance (“About Files”), how-tos (in the section on “Using Files”), and reference (“File Management Reference”). There is even a sort of reference to the reference, in a pair of “Summary of File Management” sections at the end. One is the Pascal Summary and the other is the C Summary, and they are everything that a module interface or a header file would tell you about the constants and functions available.

Note: no tutorial content. If someone new to Macintosh wants to write software and never used random-access storage before, and does not know how an application might work with multiple files and what operations to expose in their UI, they are not learning that from Inside Macintosh: Files. In fact, they are not learning that from Inside Macintosh: Human Interface Guidelines either, which explains that File menu operations “pertain to housekeeping tasks for documents” and describes each of the common items in turn (in which we notice an inconsistency: the File > Close menu item does not pertain to housekeeping tasks for documents at all, it is a window management operation). This might seem like a weird niggle but it is not, it is a huge gap: the 1980s-1990s microcomputer platforms like the Macintosh, Amiga, Atari ST, or Archimedes were the first exposure many people had to graphical user interfaces with windows, icons, menus, and pointers, and simultaneously the first exposure many had to non-sequential storage. You have to remember that this was an era when broadcast TV included minutes-long segments describing how to use a mouse. it is entirely possible that someone wanted to write an application without understanding how a computer could find multiple files, or how a user might be expected to juggle multiple files, and without knowing how the New, Save, and Open actions interact.

If you do understand that, you will get a lot of nerdy details on the innards of the system! You learn about how the Hierarchical File System allocates storage blocks on the device to files; how it separates a file’s data from its resources using two forks (surely it should be one fork with two tines?); how catalog node IDs, B*-trees, directory records, and directory thread records all work together to organise the hierarchical structure of the storage. You could write your own implementation of HFS from scratch, but still not know how apps model documents as files. Anyone who lived through the various transitions in identifying files on the Mac—from file system specifications, to FSSpec records, to FSRef structures, to -[NSString filesystemRepresentation], to NSURL, to bookmarks—can relive that history if it is really what you want to do, by diving in to the documentation. Anyone who wants to know how Apple’s developers conceptualize files on the Macintosh, and how they think Apple’s users should work with files on the Macintosh, needs to fill in that gap themselves.

Nonetheless, I still think that Inside Macintosh is “good” documentation. That is mostly because I learned what the expectations for files, and random-access storage, were from other sources, so I do not need the tutorial content it lacks. Plenty of successful developers created applications for the Macintosh that used files in ways that people with Macs expected them to. But you have to wonder how many people who wanted to write Mac apps did not, because the Inside Macintosh documentation did not bootstrap their understanding of the inside of the Macintosh.

Cover photo by the author.

In this edition:

• Graham makes us realize that you cannot get rid of programmers.
• Adrian is dismayed reviewing the history of Software Factories.
• In our Vidéothèque section, we listen to Damon Edwards tell us the story of how the DevOps movement got started.
• In the Library section, we review “MDA Explained”, “Aspect-Oriented Software Development with Use Cases”, and “Science of Software Product Lines”.

Download this issue in DRM-free PDF or EPUB format, and read it on your preferred device. You can also subscribe to our RSS feed, featuring the full content of our articles.

This issue closes the 8th volume of this magazine, also available as DRM-free PDF and EPUB files.

We would like to thank our patrons who generously contribute every month (or have contributed in the past) to our work and help us run this magazine. Thank you so much! In alphabetical order: Adam Guest, Adrian Tineo Cabello, Benjamin Sheldon, Christopher Nascone, Colin Powell, Franz Lucien Moersdorf, Guillermo Ramos Álvarez, Jean-Paul de Vooght, Dr. Juande Santander-Vela, Patryk Matuszewski, Paul Hudson, Quico Moya, Roger Turner, Szymon Licau, and countless more leaving anonymous tips every month.

Enjoy this issue! Please share our articles on social media, or contribute if you would like to support our work with a donation via Liberapay.

Cover photo by Alex Simpson on Unsplash.

To quote from the aforementioned report, conference attendee Jules Schwartz says “Programmers must be controlled, so that they don’t invent beyond the requirements which are assigned to them.” This is such a big problem, that the software engineering memeplex has phrases that describe both its occurrence (“Gold-Plating”) and its avoidance (“You Aren’t Gonna Need It” or “YAGNI”); the latter being a technique that rightfully got labeled “Extreme” on its introduction.

In both theory and in practice, you do not need programmers to write programs. Once you have a description of your system’s capabilities and design, you can automatically generate a program that does what you need. No programmer to create features you do not need; to turn a quick design tweak into a general data-driven layout engine; to rewrite the whole application using a technique they just read about in a Yourdon Press book.

In practice, this approach has drawbacks. Firstly, the implementation that you mechanically derive from a specification probably is not optimised for your target platform; it does not use design idioms from that platform’s UI, it does not use all of the hardware facilities available, and so on. You either tweak the implementation for the platform (by hiring programmers), or you optimise the compiler for the platform (by hiring programmers). Ugh. Programmers.

Secondly, a specification that is precise enough to turn directly into a functional implementation has many of the features of a computer program anyway. You know who can work with computer programs? Ugh, that is right. Programmers.

So through a combination of coercion—managers weaponised Agile, throwing so many tasks into each short iteration that there is no time for gold-plating—and compromise—programmers stopped reading books about programming—programmers and managers reached an uneasy truce. Programmers would (mostly) only write the software that their employers needed (or pretend to, with gold-plating saved for “corporate open source”), in return for which managers would (mostly) tolerate the programmers’ continued presence (or pretend to, until the next software factory hype cycle).

But now, everything has changed. Now we have a new magic pixie dust we can sprinkle over our projects to make everything beautiful and to get rid of those pesky programmers: AI. Does AI now succeed where computer-aided software engineering (CASE) failed? Can we make a software factory out of LLMs?

Yes, and no. Firstly, I will quickly dismiss the idea that Spec-Driven Development takes us back to the CASE days. This is quick to do because I already made a video on it.

But a software factory is a specific kind of CASE product, and a specific application of specification to the software world. It is a collection of pre-made components, with templates and recipes that an implementor can use to adapt it on to their specific problem. That kind of thing is so much in the wheelhouse of an LLM-based coding assistant that there are two different technical ways to solve the problem.

A software factory vendor could package their factory as an Agent Skill. The SKILL.md file tells the LLM how the software factory is structured, and what to do. Recipes/playbooks are scripts in the scripts/ folder that the LLM runs to fulfil certain tasks. Pre-made components and templates are in the assets/ folder, and documentation, of course, goes in documentation/. To build an instance of the factory software, invoke the skill.

But they could also deliver it as an MCP server. This time, the recipes and playbooks are tools that the LLM uses, and the components, templates, and documentation are resources that the server makes available to the LLM. The server also shares prompts over MCP, providing the same templated interaction with the coding assistant that an Agent Skill would.

Now would this work? Yes, of course it would. The real question is: who would make it work? Who understands how to tweak the factory’s output to meet the specific demands of the organisation? Who knows how to integrate it into the existing system? Who knows how to tailor the configurable components to suit the goals of the specific system you are building? Who knows how to fix problems when the LLM does not quite generate the correct code? Ugh, that is right. Programmers.

Cover photo by Pavel Neznanov on Unsplash.

Of course, this has not happened yet, and it will not likely happen anytime soon.

But to learn the history of programming is to see that every major advance in the field was driven by the quite explicit aim of getting rid of those programmers altogether, in one way or another, and to achieve a degree of control over the quality of the final product.

It is safe to say that such control is an illusion, as all control is.

Let us enumerate some famous examples of technologies that promised to bring some kind of control into the process of programming, or at least to free it from the all-powerful hand of programmers: FORTRAN and COBOL in the 1960s; structured programming and spreadsheets in the 1970s; object-oriented programming and CASE tools in the 1980s; design patterns and UML in the 1990s; Model-Driven Architecture and TDD in the 2000s; DevOps and Agile in the 2010s; and the latest fad, Agentic AI via statistical bullshit machines LLMs, in the troubled 2020s during which these words are being poured.

Given all of these efforts, one would think that software in 2026 would be a flawless product of the human mind, a perfect contraption, reliable in its feature set, and easily maintainable by the teams in charge. Built with safety and usability in mind, with respect for privacy and ecology concerns, and even with aesthetic considerations. Would that not be a safe assumption? Have we not been building software for over 65 years at this point?

(This author will wait for the reader of these lines to stop laughing uncontrollably.)

Of course not. Quite the opposite. If anything, software has bloated to the point where all the common sense that could be safely thrown out of the window has suffered precisely that fate, where every new wave of hype takes the industry into yet another roller coaster tour and where all rationality has long ago left the building.

No wonder many of us are seriously thinking of raising sheep in the Kerguelen Islands, as far away as possible from any software or human beings, for that matter.

(Catches breath.)

The idea of “software factories” is not new, and as many things in our industry, it was an IBM engineer who came up with it: Bob Bemer, one of the original designers of COBOL and ASCII. He published a paper titled “The Economics of Program Production” in annus mirabilis 1968, in which he described the idea of a software factory, outlining three clear goals:

a. Maximizing programmer effectiveness and personnel resources.
b. Minimizing time and costs for original production, changes and checkout.
c. Maintaining the best-conditioned system from a quality viewpoint.

Generation after generation of programming team managers have been trying to optimize these three variables by whichever means required. Developers, like Schrödinger cats aware of the fact that they are in a box with a lethal isotope next to them, have time and again silently boycotted these efforts (for better or for worse), and the resulting patchwork of technologies and approaches has had an effect akin to that of Taylorism in our field.

Precisely, this author made a painful comparison in this magazine almost 5 years ago:

Software workers are the new factory workers of the 21st century; a group of workers whose obesity, deafness induced by earphones worn in noisy open spaces, and carpal tunnel syndrome, are all visual marks of stress and abuse. The invisible ones having mental health issues; burnout, harassment, depression, all conveniently ignored by insurances, managers, and other board members.

And in this new age of LLMs and Agents, they are also the ones that are conveniently laid off, for the sake of efficiency… and also to raise the market price of the stocks of their respective companies. Those CEOs could always use yet another yacht, after all.

Eliezer Talon, in a discussion about yet another modern software factory concept called the “pull request”, wrote that

The problem is that software development organisations are not assembly lines. Most of the effort goes into understanding the problem, validating assumptions, and adapting when we discover we were wrong. Those activities benefit from collaboration rather than isolation.

Bingo. Somehow, even in the age of LLMs, programmers still have a job; if anything, at least to check that the output of those chatbots makes any sense whatsoever. This raises another crucial question, quite uncomfortable for many, about how to train future generations of programmers to have precisely the skills required to distinguish the wheat from the chaff.

As an old man screaming to a cloud, this author thinks that we could still bring a sense of ethics in our future generations of programmers and software engineers, instead of teaching them yet again how to fool an AI interview process to get a job. I thus hope for a management style where control is neither the carrot nor the stick but a healthy consequence of a well-working dialogue between managers and programmers. Maybe it is not too late to dream of that.

Or perhaps, as usual, I am just being too idealistic.

Cover photo by Steve A Johnson on Unsplash.

Hence, this month’s Vidéothèque entry is “The (Short) History of DevOps” by Damon Edwards, a 12-minute video published in September 2012, explaining the early memories of DevOps and how it evolved from 2009 to then. Pay attention to the release date of this video: we are talking about DevOps before Terraform, Docker containers, or Kubernetes.

The video explores the origins and evolution of the DevOps movement, documenting how it transitioned from a niche frustration into a globally recognized IT buzzword. Damon aims to clarify the often-misunderstood roots of DevOps, highlighting key individuals, conferences, and community efforts that shaped its decentralized development. The most important part here is to understand that no individual vendor took part in the creation of DevOps; this is quite an important point.

So the history begins in 2007 in Belgium with a certain Patrick Debois, who experienced deep frustration at his job (we can all relate to him here) due to the constant friction and conflicting workflows between agile development teams (“Dev”) and traditional operations (“Ops”) while working on a data center migration. This disconnect led to a pivotal moment at the Agile 2008 conference in Toronto, where Andrew Clay Shafer proposed a session on “agile infrastructure” (minute 01:34) that nobody attended. Like, literally nobody, except for Patrick. Their resulting conversation spawned a Google group, but the true turning point occurred during the 2009 O’Reilly’s Velocity conference, when John Allspaw and Paul Hammond delivered a famous presentation on “Dev and Ops cooperation” at Flickr, as explained at minute 03:03.

Stuff went fast from that point. Inspired by the Velocity presentation, Patrick Debois organized a local gathering in Ghent, Belgium, in late 2009 to bring developers and systems administrators together, named “DevOpsDays,” a term that was quickly shortened to the hashtag “#DevOps” by attendees wanting to save character space on Twitter (minute 04:42). Somehow, this sparked a global community out of nowhere, leading to more face-to-face meetings in places like Australia and the United States and eventually causing the movement to cross the chasm into mainstream adoption.

In just 3 years, from 2009 to 2012, DevOps became the definitive software factory concept: a set of best practices and activities, mindsets, and (yes) rituals so that software could be put into production not just 2 times a year but potentially 20 times per hour, with better quality and less friction. Ah, the dream.

DevOps is, then, an experience-based, grassroots movement created by practitioners, for practitioners (minute 09:21). It was never designed as a rigid standard, a specific product, or a corporate job title imposed by vendors. Keep that in mind the next time you get to use the Azure DevOps Server console for your project. No, Microsoft did not come up with the name or the concept.

Watch “The (Short) History of DevOps” by Damon Edwards on YouTube.

Cover snapshot chosen by the author.

When you build a Linux kernel (or someone builds it for you), an important step is configuring the build. The project uses a domain-specific language called kconfig to define various parameters and feature settings. You (or the person who does it for you) choose what features you want in the kernel, and what values you want for the other parameters, and those define the way that the build system builds a (not the) Linux kernel for you. The kernel build system is a factory, Linux is a software product line, and the specific Linux kernel you build is a product in the product line.

Or, as Don Batory points out in the important, if somewhat idiosyncratically written, “Science of Software Product Lines”, the output of kconfig is the Linux kernel you might build, if you get lucky. 87% of the configurations you can create using kconfig result in a product that does not even compile. Clearly Linus Torvalds needs to read this book.

Much of the time, software designers think about features and about codebases as orthogonal concerns. Yes, “the business” wants me to add a feature to the code, so what I am going to do is to read the code and work out some changes to make that I think integrate the feature into the existing ball of mud architecture. OK, maybe they want me to soft-launch the UI behind a feature flag, so I will find some places where I can put if(featureIsEnabled) into the code so that the feature gets turned on when the flag is set.

Technically, that product with the feature flags in is really a software product line. Each person gets their own version of the product determined at runtime, based on the state of the collection of feature flags defined for that person. But, like the Linux kernel, the creators of that feature-flag-rich app have not read “Science of Software Product Lines”, so they are not getting the full benefit of decades of research in the area.

Those of us with long memories—or who still build venerable GNU projects like Emacs or GCC from source—remember the “fun” to be had with the GNU autotools, specifically GNU autoconf and GNU automake. The concept was simple: you could make your software both portable, and flexible, by defining macros that configure build settings and enable or disable features. In fact, and perhaps surprisingly, it did not work like that at all. What it did was to inspect the target system and the options someone used to run the script, and define a collection of configuration points that instruct the build system to create different products, with distinct feature sets and internal behaviour. Yes, that is right, a GNU autoconf build system defines a software product line, and none of us had read Don’s book, either. How could we? It was published in 2026.

The typical way that autotools projects configure a software product line is to provide C preprocessor macros that the compiler uses to conditionally include or exclude feature code, switch implementations of routines, or set concrete values for abstract ideas, like the size of a memory region. Batory shows us that the C preprocessor (CPP) is one of the most powerful and capable tools available for creating software product lines—and also one of the most dangerous, as its text-substitution approach readily leaves code in an uncompilable state, particularly when features compose in unexpected ways (hence the dice roll every time you try to configure the Linux kernel). This mess is to the compiler what DLL Hell is to the runtime environment.

His book shows the theory behind slicing software by feature, and composing the code fragments that add features, and the second-level code fragments that ensure two features play nicely when they are both selected. He shows that a feature map is the space of all possible software products that a software product line produces, and that it is finite, even if potentially humungous; the Linux kernel SPL has far more possible configurations than the number of particles in the universe (when I worked at Facebook in 2014, so did the set of feature flags in their mobile apps).

He shows practical tools, like the X15 refactoring engine that uses Java annotations to build and transform SPLs in semantically valid ways. He demonstrates how object-oriented modelling can yield flexible designs that are amenable to supporting SPLs; including novel OO capabilities as in the BETA programming language. BETA is similar to the SIMULA language, but inheritance runs in the opposite way than you are used to. A parent class can inherit behaviour from its subclasses, and include instructions in its method definitions that tell the runtime to invoke the subclass’s implementation of the method. It is no surprise that OOP is a great technique for building SPLs from components, given that Brad Cox showed us how to do this with Software ICs, and how to monetise it with Superdistribution.

By far the most interesting chapter is towards the end, where Batory lists open challenges in SPL research with their complexity, reading like the higher-difficulty “exercises” in “The Art of Computer Programming”. These range from the most modern of 2026 challenges (how to work with SPLs in the LLM era) to the most basic and surprising of fundamental knowledge gaps: nobody has ever provided a scientific basis for believing that modularity is beneficial in software design. Even the most worthy of luminaries, Edsger Dijkstra, axiomatically accepts “the separation of concerns” as a good idea, as if its alternative is something that should be considered harmful with no further thought. This, ironically, in a paper called “On the role of scientific thought”!

Cover photo by the author.