Theseus OS Blog

Porting Wasmtime to no_std atop Theseus

2022-06-21T00:00:00+00:00

Bringing Wasmtime to `no_std`*

🎉🎉 Wasmtime now builds and runs on Theseus! 🎉🎉

This is the first port of Wasmtime to a no_std* environment, to the best of our knowledge, despite it being a very hot topic of conversation. We completed the initial minimal version of Wasmtime and successfully ran it on Theseus on May 17th, 2022.

This milestone marks the culmination of a very long journey, as evidenced by our previous several posts on this topic and the fact that our initial port started with a version of Wasmtime from mid-October 2021. Yeesh!

November 2021: Introducing the goal of running WASM on Theseus
December 2021: Porting the low-level "simple" Wasmtime crates and dependencies
February 2022: Porting the largest crate, wasmtime-runtime, Part 1
April 2022: Porting wasmtime-runtime, Part 2
This post: Porting the top-level wasmtime crate and tying everything together

In this post, we aim to

Provide a high-level overview of Wasmtime and its multi-crate architecture,
Enumerate the changes needed to build Wasmtime on no_std*, and
Itemize the dependencies/functionality that Wasmtime requires from the underlying platform.

While we don't intend this to be a step-by-step porting tutorial, we do hope it makes it easier to port and run Wasmtime on other platforms in the future.

Why is there an asterisk by no_std?

Theseus doesn't yet support the std library, so this is a legitimate no_std port. However, one cannot simply take this port and run it within any other no_std environment, because Wasmtime still relies on lots of functionality from the underlying platform.

We did have to add some std-like components to Theseus in order to satisfy Wasmtime's needs, as shown below.

"C'mon, just skip to the ~~recipe~~ code already!"

No problem, here is the full changeset to Wasmtime. Note that this doesn't include the many changes and extensions we made to Theseus to support this; those are described in the rest of the article.

1. Summary of Wasmtime's key parts

As described in Wasmtime's documentation, the project is architected as one top-level user-facing crate called wasmtime that re-exports and connects together key functionality from several internal crates.

wasmtime-cli: a CLI app that offers simple interactive access to standard Wasmtime features
wasmtime: exposes a safe, embeddable API for interacting with WASM modules, e.g., compiling, instantiating, and invoking them
wasmtime-jit: facilitates JIT compilation and execution of WASM modules using a compiler's code generator (currently cranelift)
wasmtime-runtime: implements the majority of the runtime logic for executing WASM binaries atop of a give host platform
wasmtime-environ: standalone abstract definitions of core WASM concepts and environment types, enabling integration with the cranelift backend
wasmtime-types: definitions for core WASM types and execution concepts
wasmparser¹: an external (non-Wasmtime) tool for parsing WASM binaries

The diagram below depicts the above major components of Wasmtime¹ and their dependencies, with a focus on those that did not already support no_std when our work began or required other forms of modification. This intentionally excludes ubiquitous dependencies like error handling, heap allocation, and logging to keep the graph legible (...ish). Note that the project also contains many other crates that realize optional features, such as module caching, fibers, etc, but these (and wasmtime-cli) aren't necessary for an initial port.

(click diagram to open it in a full-size window)

2. Changes made to Wasmtime components

We took a bottom-up approach to iteratively port lower-level dependencies until they compiled on no_std on Theseus, and then moved on to the next highest layer in the dependency stack. The lowest-level crates, wasmparser, wasmtime-types, and wasmtime-environ were relatively simple to port to no_std, requiring only trivial changes described in the section below.

By far, the most complex crate to port was wasmtime-runtime, which has not only myriad dependencies that each need porting, but also plenty of platform-specific code that is necessarily unsafe and quite intricate. We have already described our efforts to port wasmtime-runtime to Theseus in two previous posts starting here.

Moving up, porting wasmtime-jit and then wasmtime was fairly straightforward once wasmtime-runtime was done, as they share many dependencies. There were a few issues that needed fixing, as described below, but nothing strenuous.

Finally, the real fun begins once all Wasmtime crates are able to be compiled for your platform! You then attempt to run it for the first time only to discover that nothing works and you don't know why.

Welcome to integration hell! 😈

... Ok, time to come clean; it turns out that this wasn't so bad. We only had a few days of debugging struggles that centered around differences in how bincode performs (de)serialization across different versions and how region handles memory management on Unix vs. Theseus. Both are explained in the following sections.

Logic and structural changes

As Wasmtime is cross-platform, it contains several platform-specific modules that we must implement for Theseus. These typically look something like:

if #[cfg(unix)] {
    mod systemv;
    pub use self::systemv::*;
} else if #[cfg(target_os = "theseus")] {
    mod theseus;
    pub use self::theseus::*;  
} else ...

Here's an example of one such module in wasmtime-jit that handles registering unwind info for new code regions that were JIT-compiled from WASM modules.

Beyond these modules, we have encountered only a few places in which platform-agnostic code must be modified to support Theseus. Two such examples are:

The aforementioned registration of unwind info:
```
UnwindRegistration::new(
    text.as_ptr(),
    #[cfg(target_os = "theseus")]
    text.len(), // Theseus needs the text section's length
    unwind_info.as_ptr(),
    unwind_info.len(),
)
```
- Here, Theseus's unwinder benefits from knowing both the starting and ending bounds of the .text section(s) that a given piece of unwind info covers; it helps accelerate the search for the unwind info that corresponds to a given stack frame's caller virtual address.
- This stems from Theseus's structure of many small crate object files all being loaded and linked at runtime, which results in many separate .eh_frame sections that each contain unwind info for different .text section address ranges.
The representation of memory-mapped region in wasmtime-runtime:
```
pub struct Mmap {
    ptr: usize,
    len: usize,
    file: Option<File>,
    #[cfg(target_os = "theseus")]
    theseus_mp: theseus_memory::MappedPages,
}
```
- A Theseus-specific version of Mmap::accessible_reserved() and other related functions were also necessary to accommodate the changed Mmap struct.
- This design choice avoids unsafe or leaky code that would be required to get around the ownership-based type invariants guaranteed by MappedPages.

Naturally, we made many other changes, but those were generally much smaller in scope and described in following separate sections. The inquisitive reader can search the full list of changes for `target_os = "theseus to see other platform-specific modifications.

Minor yet tedious changes for `no_std` compatibility

Many other changes are trivial and thus glossed over in this post, such as changing import statements from std::xyz to core::xyz or alloc::xyz and modifying or patching dependency chains to disable default-features. However, I don't want to understate the pure tedium of such changes because they require a lot of effort for no actual difference in functionality. Many of the code diffs look like the collage of boring tedium below:

- use std::borrow::Borrow;
- use std::marker::PhantomData;
- use std::mem;
+ use core::borrow::Borrow;
+ use core::marker::PhantomData;
+ use core::mem;

[dependencies]
- anyhow = "1.0"
+ anyhow = { version = "1.0", default-features = false }
  cranelift-entity = { path = "../../cranelift/entity", version = "0.77.0" }
- wasmtime-types = { path = "../types", version = "0.30.0" }
+ wasmtime-types = { path = "../types", default-features = false, version = "0.30.0" }
- wasmparser = "0.81"
+ wasmparser = { version = "0.81", default-features = false }
  indexmap = { version = "1.0.2", features = ["serde-1"] }
- thiserror = "1.0.4"
+ thiserror = { version = "1.0.4", optional = true }
- serde = { version = "1.0.94", features = ["derive"] }
+ serde = { version = "1.0.94", default-features = false, features = ["derive"] }
  object = { version = "0.27", default-features = false, features = ['read_core', 'write_core', 'elf'] }
+ hashbrown = { version = "0.11.2", features = ["ahash"], default-features = false }
  ...

+ [patch.crates-io]
+ wasmparser = { path = "./path/to/ported/wasmparser/crate" }
+ object = { path = "./path/to/ported/object/crate" }

... and the procedure to patch dependencies that have been ported to no_std is even more tedious, shown below:

It would be a fantastic quality-of-life improvement if std's re-exports of items from core and alloc could be transparently converted to their no_std equivalents, but that's a topic for another day. In fact I've already suggested this in a presentation to select Rust team members entitled "How Theseus uses Rust, plus Rust challenges", slides 47-51.

Here are a few specific examples of changes that were conceptually minor yet tricky or tedious:

Error handling, particularly usage of anyhow
- This boils down to error type mapping: .map_err(anyhow::Error::msg).
Substituting no_std versions of specific crates
- std::io → core2::io
- thiserror → thiserror_core2
Using bincode for (de)serialization
- In accordance with bincode's migration guide, we migrated to version 2.0 that supports no_std, typically like so:
```
- bincode::deserialize(...)
+ bincode::serde::decode_from_slice(..., bincode::config::legacy())
```
- Side note: be sure to use the same versions of all types when you serialize and deserialize things... otherwise you'll get those lovely mysterious errors and spend a whole two days trying to figure out which struct field is causing the problem. 🙄

3. Full list of functionality needed to support Wasmtime

Heap allocation for alloc types
- Sorry, you ain't gonna run Wasmtime without a heap!
Basic memory management: allocating memory regions, creating memory mappings, etc.
- Unix-like OSes leverage crates like libc, region, rsix for this
Stack unwinding for proper panic handling and ensuring drop handlers run
- wasmtime-runtime requires both catch_unwind() and resume_unwind()
- Also need to support registration of external unwind info that comes from WASM modules compiled into native code, plus an unwinder that can utilize this info
backtrace for capturing a stack trace
- Plus optional symbolication of addresses, i.e., addr2line functionality or similar
Signal handling for trapping between WASM and native execution contexts
- On Theseus, this means registering custom handlers for CPU exceptions (see prior post)
Editing object files
- We contributed a PR to object that allowed reading and writing of object files in a no_std environment
bincode + serde for (de)serialization
- Just had to upgrade to newer major version that already supported no_std
- Required some minor changes to how wasmtime uses bincode, in accordance with bincode's migration guide
Basic support for accessing filesystem paths and reading files
- Only needed when reading a AOT pre-compiled WASM module from a file for purposes of deserialization
Multitasking
- Full preemption is essentially required, as cooperative multitasking would require a significant rework of Wasmtime
Thread-Local Storage (TLS), see prior post
- TLS via the thread_local!() macro is used to statefully handle traps and support stack unwinding during execution of native WASM module code
Stack access
- Typically via the psm crate, for portable stack manipulation
- Only used to obtain the current value of the stack pointer register
I/O traits
- Mostly just the basic ones from std::io, which are usable in core through a variety of no_std crates like core2 (previously bare_io), core_io, etc
Basic error handling via anyhow and thiserror
- anyhow already supports no_std, but has a different API due to its reliance on std::error::Error
- thiserror cannot support no_std yet, but there are no_std-compatible drop-in alternatives like thiserror_core2 which derive core2::error::Error instead
- Once the std::error::Error trait is moved into core, all of these pain points will be permanently healed!
A source of randomness
- Easiest option is to use OS-provided RNG via rand's interface, or its no_std SmallRng feature
Basic I/O for printing output to a screen, console, or log
Mutual exclusion
- The spin crate provides an easy yet inefficient no_std spinlock, but we do prefer to use Theseus's own Mutex that blocks the current task by putting it to sleep
crc32fast, a dependency of the object crate
- We contributed a PR to make this compile properly for custom targets
more_asserts for easy assertions
- We contributed a trivial PR to make this no_std

Concluding remarks + next steps

Phew, that was a long ride — but it's only the beginning of the journey to bring Wasmtime to Theseus. We still have several optional features to port and implement, starting with full support for live JIT compilation of WASM modules. We then plan to integrate Theseus's existing WASI implementation with our port of Wasmtime such that we can more easily run legacy programs in a WASM environment.

This is a major addition to a safe-language OS like Theseus, which runs everything in a single address space and single privilege level, foregoing hardware protection in favor of reliance on language-provided safety to realize isolation between both applications and system components. Importantly, this will enable Theseus to safely execute unsafe code or legacy components written in unsafe languages, using WASM as a "sandbox" to prevent unsafe code from circumventing the type-based restrictions ensured and obeyed by the rest of the OS's safe Rust environment.

Further down the line, we would like to propose design changes to Wasmtime such that it makes greater usage of traits to abstract away its platform-specific dependencies as much as possible. This would make it easier to port Wasmtime to new platforms, and ideally reduce the amount of mandatory unsafe code.

Small demo

Given the following very simple WASM module hello.wat:

(module
    (import "host" "hello" (func $host_hello (param i32)))
    (func (export "hello")
        i32.const 3
        call $host_hello)
)

we can precompile it using the wasmtime-cli on Linux or another machine:

wasmtime --compile --target x86_64-theseus hello.was --output hello.cwasm

and then run that precompiled hello.cwasm binary using the following host code on Theseus:

use wasmtime::{Caller, Engine, Func, Instance, Module, Store};

/// Taken from the docs example code in `wasmtime/crates/wasmtime/src/lib.rs`.
fn hello(cwasm_file_contents: &[u8]) -> Result<...> {
    let engine = Engine::default();
    // Deserialize the AOT pre-compiled WASM binary
    let module = unsafe {
        Module::deserialize(&engine, cwasm_file_contents)? 
    };
    let mut store = Store::new(&engine, 4);
    let host_hello = Func::wrap(&mut store, |caller: Caller<'_, u32>, param: i32| {
        println!("Got {} from WebAssembly", param);
        println!("my host state is: {}", caller.data());
    });
    let instance = Instance::new(&mut store, &module, &[host_hello.into()])?;
    let hello = instance.get_typed_func::<(), (), _>(&mut store, "hello")?;
    // Invoke the WASM `hello` function
    hello.call(&mut store, ()).map_err(anyhow::Error::msg)?;
    ...
}

If all goes according to plan, you'll see the expected output below:

For simplicity, we depict wasmparser as part of the set of Wasmtime crates, even though it is actually part of the separate wasm-tools project. ↩

Theseus is Hiring!

2022-04-29T00:00:00+00:00

Theseus is looking for systems programmers!

The hiring window has closed. Check back again later this year, we may be hiring again!

Theseus is proud to be a member of a diverse ecosystem of open-source Rust projects sponsored by Futurewei Technologies. Thanks to the generosity of Futurewei's Rust initiative, Theseus is able to hire developers for the first time!

To apply, send your resume or C.V. to theseus.systems@gmail.com.

Check out the below sections for more info.

Job Title: Operating Systems Developer

As an OS developer, you will independently lead a major development project within Theseus OS.

To lend some context, Theseus OS is a novel operating system written from scratch entirely in Rust, with the objective of realizing next-generation safety and efficiency guarantees for workloads in a variety of execution environments. Theseus's goals span the gamut from supporting cutting-edge exploratory research topics to more practical concerns of achieving legacy compatibility and usability.

Lately, our focus is on deep support for WebAssembly (WASM) interfaces and runtimes, as well as porting Theseus to additional architectures (e.g., ARM). As such, your development project (more ideas below) will likely be related to one of those areas.

Other job responsibilities include:

Collaborating with the general public in an open-source environment
Addressing issues and solving bugs/problems expediently
Writing easily readable code and documenting it well
Meeting with other Theseus OS developers, Rust developers, and Futurewei team members

Location and more

Fully remote
Flexible work hours
- Ideally 2+ hours of daily overlap with the US Pacific time zone
Work-life balance is highly valued and respected
Required equipment: any standard PC capable of running Linux/Windows

Compensation and Term

$35-$65 USD per hour (approx. $5,500-$10,000 USD monthly), commensurate with experience and performance.

The initial employment term is:

3-4 months if full-time (~40 hours per week)
6-9 months if part-time (~20 hours per week)

Terms can be extended if the candidate performs well.

Eligibility

Theseus is a U.S.-based organization but does not require citizenship, permanent residency, or work visa/authorization.

Anyone from any country is welcome to apply, including but not limited to students, professionals, open-source developers, hobbyists, and more. If in the U.S., you will be hired as a standard 1099 independent contractor, as with typical freelance work.

Desired Skills

The ideal candidate would have experience with at least one of the following:

Systems programming languages
- Rust preferred
- C, C++, or Assembly appreciated
Operating Systems concepts
- Multithreading and concurrency
- Memory management, virtual memory
- Architectural knowledge, e.g., ARM/x86
Basic compiler or programming language concepts
- Type safety, memory safety
WebAssembly, sandboxing, software isolation
Source code versioning via git and the general GitHub workflow
Working independently in a self-driven context
Clear communication and technical writing skills (for documentation)

If you're not an expert in these areas, don't worry! You'll pick up any necessary skills while working with us. As academics at heart, we love teaching these concepts, and it's always exciting to watch new OS concepts click.

What You'll Learn

Deep knowledge of how operating systems work under the hood
Expertise in the Rust programming language
How to write safe, easy-to-read, and robust code
Experience with navigating/understanding large projects
Skills for producing good documentation
Best practices for contributing to public open-source projects

Project Ideas

Projects are fairly open-ended and flexible; the topic is ultimately up to you, but should ideally relate to one of Theseus's current development goals. We do encourage you to work on something that interests you personally.

If you've ever caught yourself thinking “I've always wanted to know how XYZ works in an OS” then this is where you'll shine (and get paid to follow your curiosity!).

Your project(s) will expand or improve upon the existing functionality of Theseus or add new features entirely. We have an extensive list of exciting project ideas that you can tackle or use as inspiration for an idea of your own:

Extending support for running WebAssembly (WASM) on Theseus
- Supporting more WASM interfaces, e.g., WASI extensions
- Improving WASM performance (e.g., with Wasmtime) atop Theseus
Creating a generic universal driver abstraction for cross-OS driver reuse
- Ask us about WASI-dd: a future WASM-based interface for reusing drivers on any WASM-compliant OS
Deeper graphics support and a better graphics stack
- Supporting WASM+WebGL workloads
- Creating a new more featureful window manager or compositor
Legacy compatibility: improving Theseus's implementation of Rust's std library and/or libc
Porting Theseus to another architecture, e.g., ARM aarch64, RISC-V
Driver development for additional devices and peripherals
- Storage devices, GPUs, USB devices, audio chips, etc.
- Proper power management, ACPI shutdown, device/CPU suspend & resume
Language-level: deeper integration and support for Rust's async/await syntax
See more project ideas here

About Us: our Vision and Culture

We strongly believe that Theseus OS is well-positioned to be the next great system for supporting complex dynamic workloads on both low-end and high-end embedded systems, datacenter servers, and any other single-operator computing environment where safety and efficient performance are key.

Theseus began as a research-oriented OS that focused on investigating the benefits of a unique OS structure and novel methods of state/resource management.

Theseus has received positive feedback from experts in academia, appearing in top-tier conferences like the Symposium on Operating Systems and Principles (SOSP) in 2020.

We are in the process of transitioning Theseus from a research prototype to a more usable fully-fledged operating system that will support arbitrary workloads, both existing popular applications and libraries as well as up-and-coming interfaces, e.g., WASM and its interface extensions.

However, we still retain that academic culture and feel of fearlessly exploring wild and wacky ideas without concern for commercial usability.

As with most open-source projects, the work environment is casual and chill. While we do expect you to make progress, we are also all too familiar with the challenges inherent in OS development and are very understanding of unforeseen delays or complications. We'll be there to guide you to overcome any difficulties/setbacks along the way.

You'll work directly with Kevin Boos, the creator of Theseus OS and founder of Theseus Systems. Kevin is the full-time lead developer and will work personally with you to specify your project, implement your design, and foster knowledge of the fundamentals of Rust, Theseus, and OS functionality.

You'll also interact with a variety of other contributors, such as technical managers and experts from Futurewei or systems professors and PhD students from Yale University and other academic institutions.

Theseus OS is a fully open-source project that welcomes contributions from anyone. Thus, you may interact with the Rust or OS community at large, both of which are often recognized as the most welcoming of all online developer communities. Rustaceans are super friendly!

We recognize your time is valuable: meetings are kept to a minimum! That being said, Kevin and others are always happy to meet with you and help you out at any time of day. Project guidance can be as hands-on or hands-off as you like, according to your preferred work style.



Kevin Boos Founder & Creator of Theseus PhD, Rice University	Puma Chief Morale Officer PhD in Ball Fetching, sneaky Zoom ninja

Thanks, and we look forward to meeting you!

More progress porting Wasmtime to Theseus

2022-04-12T00:00:00+00:00

The quest port continues

This post covers the highlights of our ongoing work to port Wasmtime to Theseus; see the previous post(s) for more information. Interested folks can follow along with the ported Wasmtime code here.

📢 Good news: wasmtime-runtime now builds on Theseus! 📢

Completed initial port of `wasmtime-runtime`!

Last time we left off having finished several key features to support the needs of the wasmtime-runtime crate:

Thread-Local Storage (TLS)
Creation and management of Unix-like memory-mapped areas via the region and libc crates
- Plus additions to tlibc, the Theseus-specific implementation of basic libc functions
Reading/writing of object files via the object crate
Improvements to Theseus's page_allocator and theseus_cargo build tool to handle more complexity

The full list of required, target-agnostic dependencies is as follows:

[package]
name = "wasmtime-runtime"
...

[dependencies]
wasmtime-environ = { path = "../environ", version = "0.30.0" }  ## Ported to `no_std` previously
libc = { version = "0.2.82", default-features = false }         ## Ported to Theseus in prior post
region = "2.1.0"         ## Ported to Theseus in prior post
log = "0.4.8"            ## Supports `no_std`
memoffset = "0.6.0"      ## Supports `no_std`
indexmap = "1.0.2"       ## Supports `no_std`
thiserror = "1.0.4"      ## Use `thiserror_core2` instead
more-asserts = "0.2.1"   ## Ported to `no_std` in prior post
cfg-if = "1.0"           ## Supports `no_std`
backtrace = "0.3.61"     ## Ported to Theseus in this post
lazy_static = "1.3.0"    ## Supports `no_std`
rand = "0.8.3"           ## Offers `no_std`-compatible `SmallRng`
anyhow = "1.0.38"        ## Supports `no_std`, with code changes

Over the past several weeks, we have completed our modifications to wasmtime-runtime such that it now builds properly on Theseus! The key missing parts were:

Crate / Feature	Summary	Reason Needed for `wasmtime-runtime`
`backtrace`	Cross-platform crate for capturing stack traces	For capturing and analyzing stack traces to see if any WASM module functions exist on the call stack
`std::path`	Module for manipulating and parsing file paths	To refer to WASM module files, and for (de)serialization
`resume_unwind()`	Continues a panic action (e.g., unwinding), but skips the registered panic hook	To continue propagating a panic across a native code-WASM boundary
`thiserror`/`anyhow`	Helper crates for convenient error handling	To derive the `Error` trait and easily return error types
Signal handling	Registering signal handlers, e.g., for `SIGSEGV`, `SIGILL`	For catching OS-level exceptions that occur while executing native code compiled from WASM modules

`Back`(`trace`) to the Future

The wasmtime-runtime crate uses backtrace to capture a stack trace when a trap occurs, such as a fault during WASM execution or another systems-level problem like Out Of Memory (OOM). This trace is used to both:

Traverse the call stack to see if any stack frames from WASM code exist, and
Provide the user or caller of Wasmtime with more context about a runtime failure.

Porting backtrace was relatively simple; the primary changes required were to publicly expose more details about Theseus's custom unwinder. Feel free to check out the full changeset here, summarized below:

Theseus's unwinder calculates the register values for each stack frame. We simply use those values to provide backtrace with:
- The stack frame's instruction pointer (i.e., call site address)
- The current stack pointer at that execution point
- The starting address of its containing function
Theseus supports symbolication: resolving an address into a symbol
- This uses Theseus's crate management metadata, a CrateNamespace, which contains a map of all public symbols in that namespace
- The key function is get_section_containing_address()
  - This is similar to the addr2line tool, but it works with Theseus's dynamically loading and linked code structure

The one remaining feature that our port of backtrace lacks is connecting a resolved symbol to its location: file path, line number, and column number. This is conceptually easy to do but requires debug information to be parsed from an object file. Although Theseus does support parsing DWARF debug info, it isn't always available because debug info is typically stripped from object files to keep their size down. Fortunately, the backtrace crate treats this information as optional, and thus Wasmtime doesn't require it to be available, so we can simply return None when asked for symbol location details.

The `Path` Forwards

Many Wasmtime crates use std::path::{Path, PathBuf} to refer to WASM module files that are JIT-compiled and loaded into a Wasmtime engine. Thus, we must implement a version of path types that are API-compatible with Rust's std::path types in order to minimize the number of changes to Wasmtime itself. You can find the code for that here, which is primarily a quick & dirty copy of the code from std::path.

Theseus already offers its own Path type, which is similar but not identical to those in Rust std::path. All we need is a simple glue code layer between std::PathBuf and Theseus's path::Path.

One notable difference between std::path and theseus_path is that Theseus uses Rust String and str types natively, so there is no need for the equivalent types OsString and OsStr; these types become simple typedefs in Theseus:
pub type OsString = String;
pub type OsStr = str;

Relax and unwind

The resume_unwind() function is used in Wasmtime to carry on with the unwinding procedure after it has been caught. This is currently only used in the runtime's trap handling logic, which essentially continues unwinding after a trap that stemmed from a Rust-level panic, i.e., one deemed irrelevant to handling in-WASM traps.

Here is Theseus's implementation of resume_unwind() with code that tests it. Although this is conceptually tricky, the implementation is quite straightforward — simply start unwinding from the current point without starting from the regular panic handler. You can test this in Theseus by invoking the test program unwind_test -c.

To actually use our new resume_unwind() function in wasmtime-runtime, we add the following very simple code block.

#[cfg(feature = "std")]
std::panic::resume_unwind(panic);
#[cfg(target_os = "theseus")]
theseus_catch_unwind::resume_unwind(
    KillReason::Panic(PanicInfoOwned::from_payload(panic))
);

The notable difference between Theseus's resume_unwind and Rust's std::panic::resume_unwind is that we allow unwinding to occur from both a language-level panic and beneath the language level after a CPU-level "machine" exception. Thus, Theseus's panic payloads expect a KillReason rather than an erased type Box<Any>, so we must handle that minor difference.

Oh, an `Error` Occurred? Anyway Anyhow...

In yet another tribute to D. Tolnay, Wasmtime uses his superb anyhow and thiserror crates for convenient error handling across nearly every source file and function. While anyhow technically supports no_std environments like Theseus, it cannot accommodate the same API.

Thus, we must change every. single. usage. of anyhow in the whole Wasmtime code base.

The majority of the changes simply require use to add this snippet to any Result type before returning or unwrapping it:

.map_err(anyhow::Error::msg)?  // convert the `Err` into `anyhow::Error`

because in no_std environments, anyhow::Error does not impl std::error::Error and thus cannot perform the implicit conversion; we must do it explicitly. This rough edge is currently unavoidable, as evidenced by anyhow's documentation.

As you can see, most of these changes are functionally unnecessary. The real reason for such tedium is that the Error trait is defined in Rust's std library and is thus unavailable for use in no_std environments that only use core or alloc.

Thankfully, supporting thiserror is a bit easier, thanks to the thiserror_core2 crate that ports it to emit a derivation of the core2::error::Error trait. We can simply use this as a drop-in replacement for thiserror because we already use core2::error::Error as a substitute for std::error::Error.

💤💭💤 We dream of the day when @Jane Lusby's excellent work on moving the Error trait into core is completed! Once that lands, we won't need to bother with all this pablum.

The Last Jedi Dependency: Signal Handling

The last remaining feature needed to finish porting wasmtime-runtime is signal handling. This is needed for Wasmtime to be able to catch traps that occur when executing WASM code that was JIT-compiled into native code, among other purposes.

Theseus doesn't offer POSIX-like signals because they're unsafe and unnecessary in a safe-language OS, but it does implement handlers for CPU exceptions, e.g., page faults, general protection faults, etc. However, we previously did not allow third-party crates to register handlers (callbacks)

Registering "signal" (CPU exception) handlers is now supported in Theseus as of commit . The initial implementation is limited to the following four categories of "signals", which was loosely based on the set of signals that wasmtime-runtime cares about.

/// The possible "signals" that may occur due to CPU exceptions.
pub enum Exception {
    /// (SIGSEGV) Bad virtual address, unexpected page fault.
    InvalidAddress     = 0,
    /// (SIGILL) Invalid opcode, malformed instruction, etc.
    IllegalInstruction = 1,
    /// (SIGBUS) Bad memory alignment, non-existent physical address.
    BusError           = 2,
    /// (SIGFPE) Bad arithmetic operation, e.g., divide by zero.
    ArithmeticError    = 3,
}

Each task can register one handler function for each category of exceptions:

pub trait ExceptionHandler = FnOnce(&ExceptionContext) -> Result<(), ()>;

pub fn register_handler(
    exception: Exception,
    handler: Box<dyn ExceptionHandler>,
) -> Result<(), ()> {
    HANDLERS.with(|handlers| {
        let handler_slot = &handlers[exception as usize];
        if handler_slot.borrow().is_some() {
            return Err(());
        }
        *handler_slot.borrow_mut() = Some(handler);
        Ok(())
    })
}

and the Thread-Local Storage (TLS) areas are used to efficiently access the handlers registered for each given task:

thread_local!{
    /// The "signal" handlers registered for each task.
    static HANDLERS: [RefCell<Option<Box<dyn ExceptionHandler>>>; 4] = Default::default();
}

When an exception occurs, the CPU jumps synchronously to the kernel function specified in the interrupt descriptor table (IDT). We simply add a condition to each IDT exception function to check for and obtain the relevant registered exception handler (if one exists). The registered handler is then invoked with the following contextual information about said exception; the key information is the instruction_pointer.

/// Information that is passed to a registered [`ExceptionHandler`]
/// about an exception that occurred during execution.
pub struct ExceptionContext {
    pub instruction_pointer: VirtualAddress,
    pub stack_pointer: VirtualAddress,
    pub exception: Exception,
    pub error_code: Option<ErrorCode>,
}

This feature is used in the trap handling component of our port of wasmtime-runtime, which needs the above context to determine whether the exception occurred in a section of code that came from a compiled WASM module. Of course, the handlers must first be registered as part of the platform-specific init procedure shown here.

Onwards and Upwards

With that, our port of wasmtime-runtime is complete! We're nearly done with the "minimum viable port" of wasmtime functionality, as the only remaining crates are wasmtime-jit and the top-level wasmtime crate itself. Once those are complete, we will publish a longer-form post about the journey to get wasmtime running on Theseus.

Be on the lookout for such good news soon!

Miscellaneous Contributions

Fixed an issue with TLS sections, which required adding support for loading .tbss (TLS BSS) sections
@Jacob Earle implemented a rate-monotonic scheduler for pseudo-real-time task scheduling
- Adds a sleep crate that handles blocking a task for a given number of system ticks
- Supports the notion of "periodic" tasks that run a short workload, go to sleep, and then wake back up at the beginning of every period
Updated to the latest Rust 1.61 nightly version
- This was required to fix the ABI issues in LLVM, wherein the "x86-interrupt" ABI for exception handlers didn't properly match what the CPU pushes onto the stack before jumping to said handler
- Needed as part of the aforementioned effort to support registering third-party "signal" (exception) handlers in Theseus
Kevin contributed a tiny PR back to the x86_64 crate
- We had been using our own outdated fork for a while, but we now use the latest version of the x86_64 crate with a tiny addition to support a LockedIDT type.
@Ramla Ijaz finished implementing packet tranmission for the Mellanox 100GiB NIC

Progress porting wasmtime-runtime to Theseus

2022-02-03T00:00:00+00:00

Porting `wasmtime-runtime`, the key Wasmtime crate

A previous post from late 2021 chronicled our ongoing journey to port Wasmtime to Theseus. While our bottom-up approach got off to a strong start, we quickly encountered our first major challenge when examining the wasmtime-runtime crate, as it contains many dependencies on platform-specific and legacy system interfaces:

Unix-like memory mapping and protection
Signal/trap handling
Thread-local storage
Stack introspection and backtracing
File and I/O abstractions
Exception (panic) handling and unwinding resumption

This post describes our progress over a few weeks of working to add these features to Theseus in order to support wasmtime-runtime's many complex dependencies.

Porting & reorganizing third-party libraries

We first re-organized Theseus's repository to include two folders for third-party crates:

libs/: contains standalone crates that don't depend on Theseus, but can be used by Theseus and others.
ports/: contains crates that have been ported to depend directly on Theseus-specific crates, e.g., those in kernel/ and are thus not standalone.

The main features ported over the past couple of months (early winter 2021-2022) are shown in the table below:

Crate / Feature	Summary	Reason Needed for `wasmtime-runtime`
`libc`	Rust wrapper around the actual platform-specific `libc` implementation	Used to establish memory mappings and register signal handlers
`region`	Cross-platform APIs for virtual memory functions	Used to allocate large chunks of memory and remap/protect memory regions as exec/read/write as needed
TLS	Thread-Local Storage areas	Used for `thread_local!()` macro, which is needed to handle traps and stack unwinding upon exceptions that occur while executing native code that was JIT-compiled from a WASM binary
`object`	Helper crate for reading/writing object files, e.g., ELF	Used to read object files generated by `cranelift`'s backend and to manage unwind info

Porting `libc` to Theseus

Support for a minimal subset of libc functionality has been an ongoing but low-priority effort, mostly for two reasons:

Running C code directly on Theseus is inherently unsafe, as C is not a safe language and can thus violate Theseus's language safety-based isolation and resource usage guarantees.
No crates that Theseus depends on have needed an underlying libc, thus Theseus as a platform did not need to offer one... until now.

Theseus's libc implementation is called tlibc. which is described in this book chapter. So far it has been loosely based on the Redox project's relibc.

Our efforts of late were focused on supporting mmap for POSIX-style memory mappings, which Theseus has traditionally eschewed because they are unsafe and poorly-designed from a state management perspective¹. In the future, we also may support POSIX-style signal handlers, but for now we have chosen to re-implement Wasmtime's signal handling directly in safe Rust atop Theseus rather than going through an unsafe libc FFI boundary for no good reason.

The bulk of the mmap implemenation for tlibc was added in commit fffda85. The key aspects of this are:

tlibc exposes a POSIX-style mmap() function that calls Theseus APIs to instantiate new MappedPages objects, and then saves them in a private list so that they aren't dropped until the corresponding munmap() call is invoked.
- This is required because Theseus's abstraction of a virtual memory mapping, MappedPages, is auto-unmapped upon drop to guarantee safety.
Currently, mlock and munlock are dummy functions because Theseus doesn't perform any swapping or paging to disk.
Memory protection (mprotect) is offered, but is currently limited because Theseus forces all current memory mappings to be marked as "present" in the page table.
- Thus, stripping read permissions from a mapping technically works, but it violates the guarantees of the MappedPages type, i.e., the mapping is present and valid for the entire lifetime of a MappedPages object.

While we were at it, we went even further with additional improvements to theseus_cargo, libc, and tlibc to facilitate integration of Rust and C code atop Theseus.

Each time we implement a new feature in tlibc, we must also update Here's an example of that for mmap, with some testing functions included.
theseus_cargo now supports building out-of-tree components that depend on both Rust and C code, e.g., native libraries.
- It also now supports building rlib and staticlib crate types.
We added basic stdio features to tlibc, e.g., for printf, which is useful for testing purposes.

In summary, we fixed all the issues with tlibc, libc, and theseus_cargo.

Now, Rust and C code (both in-tree and out-of-tree components) can all be compiled and loaded/linked together in Theseus.

Porting `region` to Theseus

With tlibc now supporting basic libc memory mapping functions, porting the region crate was fairly straightforward.

However, importantly, we chose to not force region on Theseus to depend on tlibc, mainly because it would introduce another layer of unsafety. The primary implementation of alloc and free are here, which are similar to the mmap() implemenation in tlibc. We also must express the region::Protection type in terms of Theseus's page table EntryFlags, which was generally straightforward.

The one tricky part of region that we disliked is QueryIter, which allows the caller to query all virtual memory areas across the entire current virtual address space to find ones that span or overlap with a certain range of addresses.
This is problematic for a few reasons:

Theseus's state management philosophy dictates that it does not maintain a centralized list of all memory mappings, so there's nothing to iterate over by default.
Theseus provides a very safe and clear API for interacting with memory mappings, which region::query_range() completely ignores because it assumes a POSIX-style virtual memory API.
Theseus strives to prevent TOCTTOU attacks by avoiding the concept of a handles that point to a resource indirectly. By design (and by necessity atop conventional OSes), QueryIter separates the "time of check" from the "time of use", leading to potentially confusing behavior and errors in which a memory region returned from a query no longer exists by the time one attempts to use it.

In the end, our solution was to allow QueryIter to expose and return only references to the memory areas already created by the region crate itself. This strives to mitigate safety issues that could arise by exposing all memory regions maintained by Theseus to higher-level Rust code that may use them unsafely through the region APIs. Hopefully this feature restriction doesn't pose a problem in the future.

Supporting Thread-Local Storage on Theseus

Thread-Local Storage (TLS) allows one to declare a variable that will be instantiated on a per-thread basis, with each thread having its own local copy that other threads cannot access. This is useful for many reasons, e.g., programming conveniencce, performant access to thread-specific data without locking, etc. Our motivation for finally supporting it in its ultimate flexible form -- the ELF standard TLS areas -- stemmed from wasmtime-runtime, which uses it in myriad ways.

Note: previously, Theseus offered a cheap imitation of TLS using the GS register to store limited, targeted data about each task, but it wasn't usable by any applications, libraries, or even other non-task kernel crates.

We implemented TLS support across several commits. This was a suprisingly complex and tricky implementation that required a lot of trial-and-error experimentation to determine how to correctly layout each TLS object in the per-task TLS area.

Another complicating factor is that Theseus loads and links all crates at runtime, which means that our implementation must support both statically-linked TLS areas from the base kernel image as well as newcomers found in dynamically-loaded crates. There are a lot of tradeoffs herein as it relates to reserving and allocating offset ranges in the TLS space for TLS data sections, tracking TLS data sections per namespace, per crate, etc -- but these are best saved for a separate post about TLS.

We went a step further by implementing Rust's thread_local!() macro for any Theseus crate, which offers lazy dynamic initialization and cleanup of TLS areas. This overcomes the limitations of standard ELF TLS sections, which behave like static globals in Rust: they are const-initialized and never dropped.

Porting the `object` crate to Theseus

The object crate is standalone and doesn't need to be ported to Theseus specifically, thus we can simply port it to no_std and place it in Theseus's libs/ directory. The only real difficulty here is that while object does support no_std, no previous users of object needed to write to an object file in a no_std environment. Thinking about it, we do agree that's kind of weird, but Theseus is just like that sometimes. 😊

Once we convinced the maintainers of object that this feature was necessary, the changes required to do so weren't very involved. It boiled down to a rearrangement of object's Cargo features and configuration blocks: check out the PR we submitted (that was accepted) for more details.

Miscellaneous Improvements

We improved the page allocator to allow it to lazily merge contiguous freed chunks of pages.
- This happens lazily after an allocation request first fails; it is possible to also do it proactively in AllocatedPages::drop(), but that makes deallocation more expensive.
- Needed for loading C object files or static libraries with entry points at a fixed address, e.g., the default entry point of 0x400000.
- Future work: support building and loading position-independent executables (PIE, and PIC). This is required to simultaneously load multiple C executables at the same fixed address, because Theseus only offers a single virtual address space.

See our OSDI 2020 paper for an in-depth discussion of this. ↩

WASM has landed!

2022-01-25T00:00:00+00:00

Running WASM + WASI modules in a safe OS kernel

🚀🚀 WASM has now landed in Theseus! 🚀🚀

With PR #472 being merged in, Theseus's main branch allows you to execute WASM modules (as precompiled WASM binaries) that use both the core WASM specification and basic WASI extensions.

See our previous blog post for more about this effort.

How to use and run WASM on Theseus

Theseus doesn't yet support running an actual compiler, so you first need to download a pre-compiled WASM binary on the Theseus command line or package one up into the OS image itself during build time. We describe the latter approach below.

Theseus's build tooling now supports including arbitrary "extra files" in the generated OS .iso image; click here to read more about that feature. All you have to do is put whatever files you want included into the extra_files/ directory in the source code repository. Presto-magic-change-o, these extra files will be automatically loaded into Theseus's in-memory filesystem upon boot, in the /extra_files directory.

Once you have a WASM binary, running it is easy with the wasm command:

wasm /path/to/wasm/binary [args]

For example, try out the exorbitant interactive calculator like so:

wasm /extra_files/wasm/exorbitant.wasm

Our work on Wasmtime atop Theseus is still ongoing. Please check back for more updates soon!

2021: A Year in Review

2022-01-13T00:00:00+00:00

Theseus's First Full Year

Although 2021 not the first year of Theseus development, it was the first whole year in which:

Theseus was fully open-sourced and publically known to the community.
Theseus received interest from academic and industry collaborators.
Theseus received funding for open-source development from industry (yay!).
Our focus shifted from prototyping research concepts to feature completeness, stabilization, and legacy compatibility.

Thanks to all the folks who contributed, advised, and interacted with myself and the rest of the Theseus team this year!

We had an explosion of interest on GitHub:

And official funding from Futurewei to continue Theseus development!

We look forward to another productive year!

Time to get Theseus onto some real devices! ^{hint hint}

Recap: major new developments

Ability to build out-of-tree crates against Theseus using theseus_cargo
- A novel extension of cargo to support building against prebuilt dependencies
Basic WASM execution (using wasmi interpreter) with core WASI support
- Coming soon!
Continuing efforts to support headless, no-graphics operation
General legacy compatibility improvements, including a fuller libc implementation
Generic, dynamic, and arbitrary thread-local storage (TLS), plus a Rust-like thread_local!() macro
Tons of documentation, plus auto-published source docs and book docs!
A full redesign of ergonomic and composable traits for device I/O, plus FAT FS support
Performance and ergonomics improvements to the page and frame allocators
Deferred interrupt handling tasks for better device driver performance and system interactivity
and many more!

Thanks to 2021's Contributors!

Beyond our usual contributors, we had several newcomers from both Yale University and the open-source community at large who generously devoted their time to make some excellent improvements to Theseus. Our sincere thanks to:

Futurewei Technologies, especially Sid Askary and Yong He, for generously offering technical advice and funding for Theseus development.
@apogeeoak, who improved documentation quality and implemented GitHub workflows to autogenerate docs.
Vikram Mullick, who began and nearly finished support for running WASM+WASI binaries atop Theseus.
Jacob Earle, who began support for logging output on ARM microcontrollers and a pseudo-real time scheduling algorithm for Theseus.
Josh Triplett, who served as a valuable font of advice and sounding board for some of my wild Theseus ideas.
Philipp Oppermann, whose project Blog OS helped kickstart Theseus development a few years ago.

November-December 2021 Update: We have WASM liftoff!

2021-12-31T00:00:00+00:00

Theseus now supports WASM + WASI

Great news -- executing WASM modules atop Theseus is now working!

As proposed in a previous post, the initial implementation covers basic WASI system calls and uses the wasmi crate to execute WASM binaries using an intepreter.

Big thanks to Vikram Mullick for leading this effort. Our initial work was also inspired by early work from redshirt, a WASM-based proof-of-concept system that also used wasmi.

Note: this feature is not yet merged into the theseus_main branch. It's coming soon! In the mean time, you can follow its progress in PR #472.

Demos of running WASM on Theseus

We have a few demo applications to show basic WASM + WASI functionality.

First, a classic text adventure game called Zork. This demonstrates using stdio through WASI-defined APIs.

Second, a simple version of grep version 2.0 compiled to WASM, which demonstrates using WASI's filesystem I/O interface.

Once WASM support is officially in the main branch, we will offer an easier command line interface for loading and executing WASM binaries.

Progress on Porting Wasmtime to Theseus

While the interpreted WASM project is wrapping up, the port of Wasmtime is just getting started.

As described in Wasmtime's documentation, the project is architected as one top-level user-facing crate that re-exports and connects together key functionality from several smaller internal crates. The top-level crate is aptly named wasmtime, and it primarily exposes a safe API for interacting with WASM modules, e.g., compiling, instantiating, and invoking them.

Bottom-up approach -- off to a strong start

We have taken a bottom-up approach such that we can iteratively port each crate to the Theseus environment, building and testing them as we go. A diagram of the key crates that we care about and need to port is below.

As such, the top-level wasmtime crate will be the last one that we port to Theseus. So far, we have been able to quickly adapt the following bottom-most crates (shown in green above) to no_std environments, as they are relatively standalone:

wasmparser: an external (non-wasmtime) tool for parsing WASM binaries
wasmtime-types: definitions for core WASM types and execution concepts
wasmtime-environ: support for abstract definitions of compiler environment and features, enabling easy use of the cranelift backend for JIT
cranelift-entity: core data structures used by the Cranelift code generator

In addition, we had to modify the following crates that were dependencies of wasmtime in order to support no_std environments. We submitted PRs to each crate's upstream repository, some of which have already been accepted. Thanks to those authors/maintainers!

object: PR to allow no_std environments to write, not just read, object files
more_asserts: a trivial PR to add no_std support
crc32fast: a tricky fix to compilation for targets that lack some SIMD hardware features
indexmap: PR to unify the API between std and no_std users of indexmap data structures
- Unfortunately, this one has been delayed due to indexmap's non-standard usage of features and auto-std feature detection

Looking forward in (wasm)time

The main challenge in porting Wasmtime to Theseus is porting the wasmtime-runtime crate, which implements the majority of the runtime logic for executing WASM binaries atop a given host platform. This crate is massive and will likely take months to complete, due to its many dependencies and robust usage of standard POSIX-like functionality that Theseus currently lacks.

We'll start by tackling the main dependencies of wasmtime-runtime, most of which are based around legacy POSIX-style interfaces and traditional OS platform abstractions:

Unix-like memory mapping and protection
Signal/trap handling
Thread-local storage
Stack introspection and backtracing
File and I/O abstractions
Exception (panic) handling and unwinding resumption

Look for updates in the next post!

Miscellaneous Contributions

Kevin wrote a new book chapter about how Theseus's task management subsystem works.
Fixed a rare bug in the frame allocator, originally identified by Ramla. Thanks to her for catching that!
Fixed a classic off-by-one bug that caused buffered stdio reads to drop one character when the buffer was full. Thanks to Vikram for catching this!

October 2021 Update: All about WASM

2021-11-01T00:00:00+00:00

Bringing the Wonderful World of WASM to Theseus

Theseus's newest goal is to be a WASM-native system, in which a fully-featured WASM runtime can execute in a bare metal environment.

WebAssembly (WASM) is a powerful new binary instruction format that offers a "sandboxed" execution environment based on a simple machine model. WASM's goal is to allow code from a variety of programming languages to be easily deployed on and performantly executed within a web browser-provided environment, effectively realizing the portability dream once envisioned by Java's bytecode format.

In addition, multiple extensions to the standard help to expand WASM's functionality beyond just what is offered by most browsers. The most notable is WASI, the WebAssembly System Interface, which extends WASM's core functionality with common system-provided features like standard I/O, filesystems, clocks and timekeeping, and more.

Why WASM on Theseus?

WASM is the solution to one of the major downsides of safe-language OSes: all components must be written in a safe language in order to uphold the isolation and safety guarantees provided by said language compiler. This can make it tedious or impossible to support legacy components and interfaces.

With a WASM runtime, Theseus could safely load and run software modules written in any arbitrary unsafe language! All you'd need to do is compile them into a WASM module, which is quite easy thanks to most major languages supporting WASM targets. This will also make it significantly easier to run legacy components with complex dependency chains atop Theseus, as we can bundle them all up into self-contained WASM modules with little to no external dependencies.

How do we get there?

To bring WASM to Theseus, we have started two concurrent projects:

The simple approach: use the wasmi intepreter crate from parity-tech
- Relatively simple, as wasmi is no_std-compliant and requires only minimal interfacing with the host platform in order to use it
- We can implement WASI system calls as needed, which act as the glue between the WASM environment and the rest of Theseus's subsystems
The complex approach: port the Wasmtime WASM runtime project to Theseus
- Massively complex with dozens of platform-specific logic and API calls
- Tons of legacy dependencies, e.g., libc- and POSIX-style memory management, signal handling, system calls, and usage of many Rust libstd features

Vikram Mullick has begun working on part 1 above as part of his senior capstone project at Yale. Kevin Boos has begun working on part 2 above, and will also assist with part 1 as needed.

Due to the complex nature of Wasmtime with its many legacy dependencies, this two-pronged split approach is quite beneficial, giving us the best of both worlds:

Theseus can "quickly" get up and running with basic WASM support, allowing us to:
- Experiment with running legacy components as WASM modules
- Begin implementing support for WASI and other key WASM interfaces, e.g., WebGL
- Integrate Theseus's existing runtime loading and linking infrastructure with WASM
We can leverage the existing WASM and WASI infrastructure layers to more easily support Wasmtime
- This will realize a ~10x performance improvement over the initial wasmi, without wasting the initial wasmi-based efforts

We look forward to announcing WASM support for Theseus and realizing its full potential as a WASM-native system.

August/September Update: A Proper Terminal Emulator

2021-10-04T00:00:00+00:00

As mentioned in our previous update, we're working to support headless operation in Theseus such that it can run in an seL4 guest VM, which doesn't support graphical displays.

Going Headless with a Proper Terminal Interface

We took a three-step approach towards realizing a proper headless interactive terminal:

Removing the requirement that a graphical display (e.g., VGA device) must exist for Theseus to successfully boot.
- Now, if no display devices are not found, bootstrap and init can proceed, and headless operation is assumed.
Devising a new input dataflow and event manager, with the serial port as the focal data source and sink.
- Monitors input sources for new connections asynchronously and spawns handlers for them upon receival.
Create a new terminal emulator that:
- Offers abstraction layers to support both graphical and non-graphical terminals.
- Supports conventional escape and control codes, compliant with most ANSI, VT100, and xterm.

Yet Another Serial Port Driver Redesign

Our existing serial port driver was very simple -- it synchronously transmitted output bytes and simply dumped input bytes. Thus, the receive interrupt handler was trivially simple and executed quickly.

However, the added complexity of doing actual work (e.g., spawning a new console/terminal instance) in the receive interrupt handler caused it to become prohibitively expensive to run. Executing long-running operations in an interrupt handler is unacceptable: (i) it prevents that CPU from doing any other work, and (ii) it harms system interactiveness by preventing other interrupts from being handled.

To avoid this, we used Theseus's asynchronous channels to establish a communication channel between the serial port interrupt handler and a separate listener task. The listener task blocks until receiving a notification from any serial port, upon which it spawns a new console and terminal instance using that serial port and an input source and an output sink.

This design has one drawback: it requires proactively spawning a dedicated task just to listen for notifications from the serial port. That led us to realize that it was time to introduce a better abstraction: deferred and/or linked interrupt handlers.

Deferred Interrupt Handlers

Theseus's Deferred Interrupt Handlers are an extension of the concept of "bottom half" and "top half" interrupt handlers¹:

Interrupt Handler (aka "top half"): the short, latency-sensitive function that runs synchronously, immediately when the interrupt request is serviced. It typically does two things:
1. Notifies the deferred task that work is ready to be done (optionally providing details about that work), and
2. Acknowledges the interrupt such that the hardware knows it was handled.
Deferred task (aka "bottom half"): the more complex function that runs in a deferred manner to handle longer operations.
- Runs asynchronously in a non-interrupt context and can thus perform more long-running operations without blocking the rest of that CPU's workloads.

Theseus's deferred interrupt handling implementation is conceptually similar to but differs from tasklets and workqueues in Linux. The deferred task is uniquely tied to an interrupt handler in a 1-to-1 manner upon creation and are fully type-safe; there is a direct, strongly-typed channel of communication between them². The deferred task can also optionally be configured to execute immediately after the interrupt handler in order to reduce I/O device latency, thanks to scheduler integration.

Our favorite side benefit of this design is that itencourages adherence to the separation of concerns principle, as the interrupt handling functionality must be conciously divided into an "urgent" synchronous part and a deferred asynchronous part.

With this new interrupt handling architecture, we were able to adapt the serial driver to work efficiently without having unnecessary listener tasks wasting memory and CPU cycles. The serial port is a simple case, as the interrupt handler only need to notify the deferred task that data has been received on the port and then acknowledge the interrupt; no other data exchange is needed. The deferred task can then read all received data from the serial port in a non-urgent manner and do anything else necessary, such as spawning new console/terminal instances.

Splitting the Serial Port Driver

Theseus's design specifies a tiny minimal kernel boot image, the nano_core which must include only the bare minimum components to be able to bootstrap the OS and then dynamically load all other OS initialization components. One of these components is the serial port driver, as it's the only real usable choice for early boot/kernel logging in the absence of display, networking, or file support.

With the above additions to the serial port driver, namely the usage of deferred interrupts and channels, the driver was becoming far too complex. It had gone from having zero dependencies to having dozens. That bloated the size of the nano_core, thereby harming the dynamicness of Theseus.

To solve this, we split the serial port driver into two parts:

Basic driver: a standalone crate that exposes the SerialPort type, which only offers functions to initialize, read from, and write to the device.
Full driver: offers full I/O support with trait implementations, deferred interrupts, channel usage, task blocking, and more.
- Wraps the basic driver's SerialPort type with additional functionality.

This design enables two previously-conflicting goals:

The majority of system components can access the full functionality of the serial port and use it just like any other I/O device.
The kernel boot image is kept tiny with minimal dependencies whilst still being able to output logs to the serial port.

A Complete Terminal Emulator Rewrite

Finally, we began the most complicated component of true interactive headless operation: a terminal emulator. Theseus's existing terminal is fairly ad-hoc and doesn't conform to any real standards; it was implemented quickly as a summer intern project and offers just enough features to run commands and display their output. The same is true for Theseus's stdio, which is a "quick-n-dirty" inflexible implementation that uses heap buffers instead of a real file abstraction.

As anyone who has dealt with this knows, terminal emulators are exceedingly and unexpectedly complex. We experimented with multiple iterations of the design, primarily centered around how best to represent each displayable "unit" or "cell" on the terminal display:

The terminal grid stores a dynamic number of elements per line, in which each element can contain either a single character or a string of characters (no matter how wide) as long as they have the same style.
- Easy to output to a text backend.
- The most memory efficient.
- Calculating line wrapping and cursor navigation is overly complex because each unit may have a different columnar display width.
The terminal grid stores a dynamic number of elements per line, but each element in the line can only contain a single character.
- Slightly less memory efficient, as it duplicates units that have the same style.
- Easier to calculate displayed rows and cursor movement than design 1, but still complex.
The terminal grid stores a dynamic number of elements per line, but each element corresponds to a single displayed column on screen.
- Requires more memory and a little bit of extra effort to support wide characters, since they must be split across more than one element.
- Much, much easier to calculate cursor movements, line wrapping, and display commands.
The terminal grid stores a fixed number of elements per line, equivalent to the width of the screen row; blank units are inserted to fill each linas necessary.
- Very wasteful of memory, especially for short lines (common in terminal output).
- Trivial to calculate displayed rows and cursor movement.
- Very hard to re-size/re-flow the terminal.

After multiple iterations, we settled on Design 3 above. It selects the best set of tradeoffs in the spectrum of design points, and most importantly, simplifies the translation between on-screen coordinates and scrollback buffer coordinate.

One interesting aspect of Theseus's terminal emulator is how it's split into multiple components:

The frontend: a single entity responsible for handling incoming character and control/escape code bytes and determining what actions should be taken.
- The frontend is also reusable as a terminal driver, readline library, and line discipline layer.
The backend abstraction: a trait that represents the various display actions that a terminal might invoke.
The backend implentation(s): one of multiple entity types that handles display actions to "render" the terminal output in different formats:
- Graphical pixel framebuffers
- Classic 80x25 VGA screen with 16-color basic text display
- A serial port connected to a host-side pseudo-terminal
- A network connection or file or any other sink

However, this is still a work in progress and is not yet used for Theseus's main terminal display. It takes quite a lot of effort to get all the details right, but we have an MVP that supports:

Insert/replace mode switching
Automatic CR/LF behavior with wrapping
Multiple different PTY backends, e.g., screen, picocom, minicom, etc
Most ANSI escape/control codes, including colors and text styles
Cursor movement and scrolling

Fortunately, there are several Rust terminal emulators that serve as valuable examples, e.g., Alacritty, but they're standalone monolithic projects that rely on an underlying graphics stack like OpenGL and are thus inappropriate for in-kernel usage. We were able to use the excellent vte crate from the Alacritty project though, which saved a lot of time and effort in parsing ANSI/VT100 control codes.

Did you know that the "grid" of a terminal screen is indexed starting at (1,1) rather than (0,0) for the upper-leftmost character? ^{Oh, how I wish I knew that earlier...}

Other Improvements

Improved safety, complexity, and performance of interpreting a memory region as an executable function.
- Previously called MappedPages::as_func(), now moved into the crate management subsystem as a member function of the LoadedSection type.
- The new LoadedSection::as_func() method already knows both the size of the function's section and whether the underlying memory is executable, which omits multiple runtime checks.
- The function interface is much simpler, as the caller need not specify any size or offset values w.r.t. the memory region.
- Improves safety: offers a compile-time guarantee that an executable function instance can only be obtained from a LoadedSection (with proper alignment and size) rather than any chunk of executable memory.
Re-designed parts of the Task struct to reduce lock contention, eliminating it in most places.
- "Mutable" fields (those that may change after Task creation) are now moved into a TaskInner struct (excluding atomic fields), which allows most fields to be accessed without acquiring the lock on that Task.
- Immutable fields are kept in the "outer" Task struct.
- We ensure atomic types are properly aligned and sized such that native atomic instructions are actually used. We realized this by switching from atomic::Atomic to crossbeam_utils::atomic::AtomicCell, plus static assertions that all types wrapped in AtomicCell are actually eligible for native atomic access.
- Move a Task's ExitValue out of its RunState enum so that the RunState enum can be accessed atomically and independently of an ExitValue.
- The runstate and running_on_cpu fields of Task are now atomically accessible in a lock-free manner, allowing us to block/unblock tasks in lock-free contexts, e.g., within an interrupt handler.
- As more Task fields are now readable without locking, we can remove all now-unnecessary unsafe statements from the scheduler.

Other terminology is often used, including "first-level" and "second-level" interrupt handlers, or "hard" and "soft" interrupt handlers. ↩
It is also possible to have a pool of deferred tasks, in which each task can be tied to multiple interrupt handlers. ↩

June/July Update: Headless Operation on seL4

2021-08-02T00:00:00+00:00

One Very Busy, Very Hot Summer

In June of this year, the Seattle area hit record high temperatures of over 110°F (44°C) for three days. Ouch!

We began this hot summer with the goal of enabling Theseus to run atop seL4, using both its hypervisor and VMM functionality to present Theseus with a standard "bare metal" x86 environment. Unfortunately, we quickly discovered that while seL4 supports ARM and x86, it does not yet fully support x86_64 VMMs, which is the only architecture that Theseus currently runs on. The implementation of an x86_64 VMM was supposedly completed by Dornerworks, but we were unable to get it to successfully run any x86_64 guest OS (but x86 VMs did work properly). We have decided to postpone this particular effort until this PR that officially adds support for x86_64 VMMs on seL4 is accepted.

Headless operation

In the meantime, we started working towards Theseus-level support for headless operation over a serial port interface. The serial port is the only form of direct interactive access to guest VMs on seL4 (excluding network access), so it is a necessary component to debug and use Theseus therein. It's also useful for communicating with serial devices on other more limited platforms, e.g., our WIP port of Theseus to ARM Cortex-M4 microcontrollers.

Previously, Theseus had two problems in this area:

All input to the shell/terminal was assumed to come from a real keyboard and mouse.
The serial port was only used for basic logging output (not treated as a regular I/O device).

To achieve headless operation, we had to set two corresponding goals:

Abstract the terminal and input handling to work with any I/O source, not just a physical keyboard.
Enable full, bidirectional, arbitrary I/O across serial ports.

New I/O Abstractions: Stateless vs. Stateful

As the first step towards these goals, we created several new I/O traits to represent different categories of I/O.

The BlockReader and BlockWriter traits represent I/O streams which can be read from or written to at the granularity of a single block (as the smallest transferable chunk).
BlockIo is a "parent" trait that specifies the size in bytes of each block in a block-based I/O stream.
KnownLength: a separate trait that represents an I/O stream with a known length, such as a disk drive.
ByteReader, ByteWriter: traits that represent I/O streams which can be read from or written to at the granularity of an individual byte.
We also provide wrapper types that allow byte-wise access atop block-based I/O streams: ByteReaderWrapper, ByteWriterWrapper, ByteReaderWriterWrapper.
- The blocks_from_bytes() function is useful for calculating the set of block-based I/O transfers that are needed to satisfy an arbitrary byte-wise transfer.

Notably, these traits all offer stateless access to byte streams only, an important behavioral characteristic that helps simplify state management in Theseus. This means that they don't keep track of an internal offset within the stream.

For example, the ByteReader trait exposes only one function that requires the caller to specify at which offset the stream read should start.

fn read_at(&mut self, buffer: &mut [u8], offset: usize) -> Result<...>

These traits and types also stack on top of each other, e.g., you can use a ByteReader to realize byte-wise access to an underlying block-based I/O device that implements BlockReader. We make this easier with trait delegation, in which wrapper types "forward" the trait implementation through:

References (&dyn ByteReader)
Mutable References (&mut dyn ByteReader)
Locks (Mutex<ByteReader>) using the LockableIo type

We also offer stateful I/O types, which wrap stateless I/O streams (the above traits) to track the current offset into the I/O stream while reading or writing it. This is similar to classic POSIX I/O interfaces, but are strongly-typed and allow for limited permissions: ReaderWriter, Reader, and Writer structs.

Finally, all of the above types and traits implement the no_std version of std::io::Read/Write traits, which can come from crates like core_io or bare_io. This widely expands their compatibility to work with pretty much any other I/O-related code in the Rust ecosystem.

For example, we used these new I/O abstractions to integrate the fatfs Rust crate with Theseus, which allows Theseus to read and write the contents of a FAT filesystem on disk. More work is required to provide a more generic file abstraction that can represent arbitrary files across any filesystem type, as Theseus's current representation of files is quite ad-hoc and limited to in-memory filesystems.

Redesigned Serial Port Driver

With redesigned I/O traits, we can proceed to our second goal: improving the serial port driver.

On x86 machines, there are up to 4 serial ports, but commonly only one or two are available: COM1 and COM2. The OS can interact with them using different I/O ports, e.g., writing bytes to 0x3F8 and the subsequent 7 port addresses will allow you to communicate with the COM1 serial port. Here are three great resources for learning more about serial port behavior and how to write a proper driver: one, two, three.

The changes we needed to make are:

Implement and activate interrupt handlers for all serial ports, such that the hardware triggers an interrupt when input bytes are received on the port.
SerialPort instances are no longer exclusively owned by the logger, as they must be accessible from within the serial port interrupt handler and other kernel/application crates.
Implement the necessary read and write I/O traits for the SerialPort type, so we can use them in all I/O stream contexts.

With these changes in place, Theseus is now able to read from and write to serial ports freely, as if it were any other I/O stream like a file or disk.

Other Improvements to Theseus

Updated Theseus's Rust compiler to version 1.54, which entailed many changes:
- Refactoring all inline assembly to Rust's new asm!() syntax.
- Complying with the restrictions on naked functions: Rust ABI is no longer allowed, and only one assembly block is permitted per naked function.
- Use the new compare_exchange_weak() family of functions, which is more efficient on some architectures (ARM) because it is allowed to spuriously fail.
Refactored code for memory-related types to unify their APIs:
Book documentation: thanks to @apogeeoak, the Theseus Book now has clearer structure, automatic spell check, and is built and published online via GitHub Actions CI workflows.
Mellanox 100GiB NIC: Ramla Ijaz added basic support for initializing and configuring this high-performance NIC. Packet transmission is an ongoing work in progress.
Added support for parsing the ACPI DMAR table, which specifies details about the system's IOMMU.
- This is part of our quest to protect Theseus's single address space execution environment from errant or malicious I/O devices that attempt to access arbitrary system memory without permission, the one final frontier in the "chain of safety" that cannot be checking by the compiler.

Contributions to other Open-Source Projects

We added compile-time configuration of logging flexibility to the rust-fatfs crate.
We ported the MPMC Queue crate to support no_std environments on the latest version of Rust.

Next Steps

Now that we have flexible, generic I/O abstractions available in Theseus, the next step for achieving full headless operation is to enable a terminal/CLI to handle I/O to and from arbitrary sources, such as a serial port.

Theseus OS Blog

Porting Wasmtime to no_std atop Theseus

Bringing Wasmtime to no_std*

1. Summary of Wasmtime's key parts

2. Changes made to Wasmtime components

Logic and structural changes

Minor yet tedious changes for no_std compatibility

3. Full list of functionality needed to support Wasmtime

Concluding remarks + next steps

Small demo

Theseus is Hiring!

Theseus is looking for systems programmers!

Job Title: Operating Systems Developer

Location and more

Compensation and Term

Eligibility

Desired Skills

What You'll Learn

Project Ideas

About Us: our Vision and Culture

More progress porting Wasmtime to Theseus

The quest port continues

Completed initial port of wasmtime-runtime!

Back(trace) to the Future

The Path Forwards

Relax and unwind

Oh, an Error Occurred? Anyway Anyhow...

The Last Jedi Dependency: Signal Handling

Onwards and Upwards

Miscellaneous Contributions

Progress porting wasmtime-runtime to Theseus

Porting wasmtime-runtime, the key Wasmtime crate

Porting & reorganizing third-party libraries

Porting libc to Theseus

Porting region to Theseus

Supporting Thread-Local Storage on Theseus

Porting the object crate to Theseus

Miscellaneous Improvements

WASM has landed!

Running WASM + WASI modules in a safe OS kernel

How to use and run WASM on Theseus

2021: A Year in Review

Theseus's First Full Year

Recap: major new developments

Thanks to 2021's Contributors!

November-December 2021 Update: We have WASM liftoff!

Theseus now supports WASM + WASI

Demos of running WASM on Theseus

Progress on Porting Wasmtime to Theseus

Bottom-up approach -- off to a strong start

Looking forward in (wasm)time

Miscellaneous Contributions

October 2021 Update: All about WASM

Bringing the Wonderful World of WASM to Theseus

Why WASM on Theseus?

How do we get there?

August/September Update: A Proper Terminal Emulator

Going Headless with a Proper Terminal Interface

Yet Another Serial Port Driver Redesign

Deferred Interrupt Handlers

Splitting the Serial Port Driver

A Complete Terminal Emulator Rewrite

Other Improvements

June/July Update: Headless Operation on seL4

One Very Busy, Very Hot Summer

Headless operation

New I/O Abstractions: Stateless vs. Stateful

Redesigned Serial Port Driver

Other Improvements to Theseus

Contributions to other Open-Source Projects

Next Steps

Bringing Wasmtime to `no_std`*

Minor yet tedious changes for `no_std` compatibility

Completed initial port of `wasmtime-runtime`!

`Back`(`trace`) to the Future

The `Path` Forwards

Oh, an `Error` Occurred? Anyway Anyhow...

Porting `wasmtime-runtime`, the key Wasmtime crate

Porting `libc` to Theseus

Porting `region` to Theseus

Porting the `object` crate to Theseus