Hermit Shell

I've been curious about the idea of building a shell that uses an ahead-of-time compilation strategy for a while now. Essentially, the shell is composed of many functions written to perform different tasks. Some of these functions are exposed as commands to the interactive mode, but rather than having an interpreted language to glue programs together, you would write your new command as a function that simply calls any other functions it needs to. Building complex behaviours no longer relies on piping and redundant code being used to serialise and parse data moving between hundreds of discrete programs: the shell is now one monolithic program that just passes data around in memory.

History

Terry Davis built a pretty damn functional implementation of this with the prompt in TempleOS, which, if you've never heard of it, is a scratch-built operating system that the term 'esoteric' does not even begin to cover. Crucially, HolyC, the for-purpose language in which most of TempleOS was implemented, could be compiled ahead-of-time or interpreted just-in-time. The interactive command prompt seems to have been implemented with the latter, which side-steps the issue I faced choosing to build Hermit in a pure AoT language.

I first started thinking about this idea in a roundabout way after researching TempleOS a few years ago, but I had no idea what it might look like and wasn't nearly a knowledgeable enough programmer to conceive of any possible design solution for it.

Then, relatively recently, I saw an old snippet of Jonathan Blow talking about, essentially, his independently invented version of the same principle, in which commands are just functions/procedures you chain together with actual code, and where external programs exist, piping and serialisation are replaced with direct memory mapping of data, the set-up for which is facilitated by the shell.

My Attempt

I'm going to preface this by saying that Hermit isn't a truly sincere attempt at the highest level version of this concept. For me, it was more about exploring the construction of an interactive shell and whether the 'userspace' scripting can be done entirely in a systems language without significant interruption to a daily workflow — can I edit a script and run it, instantly, without needing to restart my shell, despite the fact that that's literally what I need to do for this program to work?

So my very simple solution to have shared functions-as-commands AoT code be editable and immediately available without having to rebuild and reload the shell was to build a very thin, central shell program (about 250 lines of Odin) alongside a user library, which consists of a big pile of pre-declared, shared utilities. Specific procedures in this library — Odin's term for functions — that are marked for export in the compiler will become commands.

The core shell program provides a reserved command, reload, which unloads the user library, recompiles the user code into a DLL and mounts and indexes it again. If a compilation fail occurs during this step, the DLL isn't overwritten so no runtime damage is done. The old one is just mounted again and the compilation error is forwarded to the console. In the current version, there's no facility for dealing with this while multiple shells are running simultaneously, so the compilation fails because the DLL is unwritable. This doesn't crash or cause any running instances any problems, so I've left it for now. I believe I have a fairly simple solution to make this workable, where each instance of the shell program can choose how to react to the changes.

Local Scripts

One problem I immediately noticed with the design was the inability to localise scripts — you can't have a build.sh script in a project folder that's just written in Odin.

I mean... you can... I even built a run command in Hermit that simply calls odin run against the targeted script while adding all the relevant library and linking paths to it. Okay, it leaves an executable behind, but it technically works. Even though Odin is a very fast compiler, that 800ms compilation time is very, very noticeable when you're expecting a command to just run. You can get into comparing the file with the executable and only rebuilding when the code is new— oh who gives a shit.

So I made a tiny linear script format instead, which is the most turbo-basic text file of linear commands. Each line is a separate command and its arguments, and if any of them fail, it exits there. There's no control flow or anything, it just runs through linearly.

Anyway it turns out that this solves like 80% of daily reusable scripting cases, because, as much as we terminal folk love to automate everything with our little magic text files, most of the shit we actually have to do is chain a few simple commands together that we regularly need to run.

Demo

Here's an example of how the prompt is implemented in Hermit —

@export
hermit_prompt :: proc() {
    path := os.get_current_directory()

    // calling 'base' on a Windows drive root 'X:\' will bork it
    // so we use a little safety check to prevent that

    if !strings.has_suffix(path, ":\\") {
        path = filepath.base(path)
    }

    hermit.set_color(.GREEN)
    hermit.print(path)

    hermit.set_color()
    hermit.print(" >>> ")
}

This gives us —

project_dir >>> |

This isn't significantly more complex than my identical Fish prompt, so that's cool.

Building with External Code

One of the cool things I did while experimenting was to directly source my game framework Forest into my Hermit user code. Forest has a particular design philosophy where it exposes everything that's of use, which also means the framework code itself becomes the source of truth for a lot of things. Rather than building tools separate programs, I'll simply implement them directly into Forest, so my games can execute those operations for me automatically.

Sometimes I still need those tools outside of games for one-off operations and such, so Forest also has a tooling folder with individual stub files that expose those internal APIs to the command line. Each one is a simple argument parser with some error handling and help text. For a normal shell, building and placing these on path are a necessary intermediate step.

But Hermit is written in a systems language, and in fact, already does argument handling and the basic error handling required of a shell program. So what if, instead of building these separate program stubs, I just import my game engine into my shell?

Well, I can. I now have my entire game engine's tooling system directly integrated into Hermit as user-space commands. These get rebuilt all the time whenever changes to the shell occur, so I rarely, if ever, have a tool fall out of sync with a game because it's an old build.

Obviously this is trivial for me because Forest and Hermit are both Odin and they share an inherent design philosophy that I bring to things I build — so they're naturally able to gel together in this way very quickly.

Odin can import and deal with anything with a C-compatible ABI — instead of the wildly complicated scripts I make for ffmpeg (because ffmpeg's command line interface is crazy), I can just import ffmpeg as a library and interface with it much, much faster. You could also build a Hermit directly in C, Zig, Rust or Go or any other number of systems languages. With the wealth of libraries they have available to them, most of the things you want your shell to do can be implemented trivially.

The reality is, 90% of my Hermit code will be me implementing my regular, reusable scripts folder in Odin anyway. This was just a cool outcome of pursuing the idea to its extreme.