Hexops' devlog

Announcing pkgmirror: self-host your own Zig mirror

Thu, 26 Mar 2026 00:00:00 +0000

pkgmirror is an open-source, self-hosted Zig toolchain and package mirror service so your builds no longer rely on third-party services or Microsoft GitHub.

Mach now hosts its own pkgmirror at pkg.hexops.org.

Features

Zig toolchain mirroring: acts as a caching reverse proxy for Zig toolchain downloads from ziglang.org
Package mirroring (optional) - instead of writing GitHub URLs in your project’s build.zig.zon, use your own server and have it mirror GitHub, Codeberg, or your project host of choice. No more Microsoft-induced downtime harming your builds.
Artifact mirroring (optional) - mirror prebuilt binaries and CI artifacts the same exact way.
Nominated Zig support - support for Mach’s nominated Zig versions: something less of a moving target than nightly Zig, but more frequently updated than Zig stable releases.
Proactive cache warming - automatically fetches all stable and nominated Zig versions, for each OS/arch variant, so that your mirror is ready to go.
Automatic LetsEncrypt support - Server can invoke acme.sh for you, no reverse proxy needed!
Single binary - written in Zig, ~9 MB, super fast - no runtime dependencies beyond libc and acme.sh (optional)

Why run a Zig toolchain mirror?

The Zig community maintains a list of community mirrors for toolchain downloads which are used throughout the community.

For example, setup-zig is a GitHub Action that installs Zig and caches your build directory between runs, it cycles through the Zig mirror list automatically to reduce load on ziglang.org.

anyzig - which is probably the nicest way to seamlessly switch between Zig versions in your Zig projects, uses Zig mirrors as well.

Note: minisig verification is used to ensure you got a genuine ziglang.org binary, and not something malicious - these tools handle that verification for you automatically.

pkgmirror is a Zig toolchain mirror server, written in Zig, so you can host a mirror of your own and contribute!

Zig package mirrors

pkgmirror goes beyond just simple Zig toolchain mirroring and allows mirroring Zig packages and binary artifacts, too. This is optional.

In Mach for example, it’s really important to us that someone can check out an old repository with an old game/app that uses Mach, and be able to get it building quickly still: a key part of this is ensuring that all of Mach’s dependencies in build.zig.zon are hosted through reliable URLs that won’t break or dissappear over time. Sure, we could link to a Github-hosted tarball of one of our libraries - but GitHub repositories get deleted, or Microsoft enshittification of GitHub may occur and we have to move to a self-hosted forgejo instance, code.hexops.org (which we did recently) - so we need stable, long-term URLs our dependencies can be hosted at.

pkgmirror allows configuring mirroring of Zig packages (and binary CI artifacts, too) that are hosted on GitHub, Codeberg, etc. - the approach is simple: pkgmirror caches the file on disk for you forever, and gives you a super stable URL to write in your build.zig.zon file.

Not everyone should run a mirror

Not everyone should run a Zig toolchain or package mirror: it’s a serious responsibility, you should handle backups, etc. especially if others are depending on your mirror to be stable.

But when you do want to host a Zig mirror, pkgmirror is now a solid option for that - written in Zig unlike the myriad of other Go-based solutions out there.

Special thanks

pkgmirror would not be possible to write in Zig today if not for karlseguin’s http.zig library, and our automatic LetsEncrypt support via acme.sh wouldn’t be possible without ianic’s excellent tls.zig library providing a TLS 1.3 server.

Thanks

You can now find pkgmirror and Mach itself on our self-hosted forgejo instance, code.hexops.org
Join the Mach Discord server to follow development
Consider sponsoring our work so we can do more of it!

Building the DirectX shader compiler better than Microsoft?

Fri, 09 Feb 2024 00:00:00 +0000

This is a ~~story~~ nightmare about the messy state of Microsoft’s DirectX shader compiler, and trying to wrangle it into a nicer experience for game developers. In some respects, we now build the DXC compiler better than how Microsoft does.

Setting the stage

For Mach engine we’ve been building an experimental graphics API called sysgpu using Zig, aiming to be a successor and descendant of WebGPU for native graphics. It will support Metal, Vulkan, Direct3D, and OpenGL backends. As part of this, we need to compile shader programs into something that Direct3D 12 can consume. But what does it consume?

A brief history lesson

The DirectX graphics API uses HLSL as its shading language of choice. In the past, with Direct3D 11 and earlier, this compiler was called ‘FXC’ (the ’effects compiler’)

FXC is deprecated, DXC enters the scene with Direct3D 12

Unfortunately, FXC as a compiler is rather notoriously slow among game developers, with suboptimal code generation - meaning shaders often both compile and execute fairly suboptimally.

With the release of Direct3D 12 and Shader Model 6.0 (SM6), Microsoft officially deprecated the FXC compiler distributed as part of the Windows OS in favor of a new compiler called ‘DXC’ (‘directx compiler’), which exists as a public Microsoft-official fork of LLVM/Clang v3.7 Microsoft/DirectXShaderCompiler and prebuilt binaries you can download.

In this Microsoft fork of LLVM, changes are meticulously annotated via // HLSL Change Start and // HLSL Change End comments making it clear who owns what code:

What a DirectX driver eats for breakfast: DXBC or DXIL

Although HLSL is the language of choice for Direct3D programming, at the end of the day GPUs under the hood all have different compute architectures and requirements: the compiled binary form of a shader program that an Intel GPU needs is going to be different from what an NVIDIA GPU needs, same goes for AMD.

Microsoft’s role is to provide the nice game developer frontend APIs (like Direct3D, and the HLSL shanding language), while working with independent hardware vendors (IHV’s) like Intel/AMD/NVIDIA who write the drivers - bridging those nice frontend APIs to whatever is hopefully closest to hardware manufacturer’s instruction set architecture (ISA) under the hood. You can think of it like web browsers making sure JavaScript can run on both a Windows PC and a macOS Apple Silicon device, though graphics developers would spit at the suggested comparison.

DirectX versions 9-11 had driver manufacturers consuming what is called DXBC (DirectX Byte Code) - game developers would produce DXBC either using a CLI tool to compile their HLSL programs like fxc.exe, or at runtime using the d3dcompiler APIs, and then the driver’s job was to take that decently-optimized shader bytecode and turn it into the actual binary that the GPU would run. This bytecode was an undocumented, proprietary format really only shared between Microsoft and GPU driver manufacturers - excluding a few odd-ball Linux developers who cared to reverse engineer it for Proton.

With the advent of DirectX 12 and Shader Model 6.0, Microsoft aspirationally had intended to create their own standard IR called DXIR, but in 2021 they removed all language suggesting they might do this. The intent was for DXIR to be the ‘high level’, ‘unoptimized’ IR form which compilers (think: Rust) could target, and then the DXC compiler could lower DXIR into the optimized DXIL bytecode form, a new ’low level’ post-optimization IR format, before handing it off to graphics drivers to muck with as they please before it gets translated to run on the actual hardware.

Asked about DXIR documentation, a Microsoft employee had noted this in 2019:

Unfortunately, documentation on the lowering process [from DXIR to DXIL] is mostly non-existent. […]

Oh, and DXIR isn’t anything official, but just the first LLVM IR after CodeGen.

As you’ll soon see, this theme of ’there are no docs, just whatever our compiler actually does’ will become a common pattern.

DXIL

DXIL (pronunciation?) is the official format that DirectX 12 driver manufacturers consume today.

A game developer produces DXIL bytecode using the DXC compiler, which is a fork of LLVM/clang heavily modified to support HLSL compilation, and the DirectX APIs hand that DXIL over to the graphics driver which then converts the IR into their own intermediate languages, performing any secret sauce optimization passes on it, and ultimately boiling down to the actual machine code that will run on the GPU hardware.

Much like the old bytecode format DXBC which DXIL replaced, it is also an undocumented bytecode format, specifically it is LLVM’s version 3.7 post-codegen post-optimization-passes bitecode format. It is undocumented not because nobody wants to document it, but rather because the documentation is literally ‘whatever the Microsoft fork of LLVM v3.7 with all the HLSL changes we made, after CodeGen and optimization passes have occurred, actually emits as LLVM bitcode - plus a small custom container/wrapper file format on top.’

Correcting the Microsoft fork of LLVM

Microsoft themselves are well aware that a bunch of independent driver manufacturers relying on and expecting to consume a hyper-specific undocumented LLVM bitcode format specifically produced by their fork is, well, less than ideal - and also aware that their fork of LLVM is not super fun to maintain, either. Quoting another Microsoft employee (Sep 2023) who was asked about the potential of adding DirectX 9/10/11 support to the new/better DXC compiler, they stated:

DXC’s fork of LLVM removed and/or damaged much of the code generation layer and infrastructure [of LLVM]. Given that, supporting DXBC generation in DXC would be a massive task to fix and restore broken LLVM functionality. Due to the large scale of this issue and resource constraints on our team we’re not going to address this issue in [the new] DXC [compiler] ever.

We may support DXBC generation in Clang in the future (we mentioned that in the original proposal to LLVM). That work is unlikely to begin for a few years as our focus will be on supporting DXIL and SPIR-V generation first.

As noted above, in March of 2022, Microsoft had proposed and begun work on upstreaming HLSL compilation support directly into LLVM/clang proper - work that is still ongoing today - and involved adding back legacy LLVM v3.7 bitcode writing support to modern LLVM/clang versions:

By isolating as much of the DXIL-specific code as possible into a target we hope to minimize the cost on the community to maintain our legacy bitcode writing support.

i.e. the plan to get away fromn their fork is to upstream HLSL and DXIL support to LLVM/clang proper.

The challenge for gamedevs, WebGPU, etc.

Graphics abstraction layers which aim to provide a unified interface to modern graphics APIs like Metal, Direct3D 12, and Vulkan.. ultimately need to provide a unified shading language as well. If you look today, you’ll find most WebGPU implementations which do this have had a goal of ‘in the future we might be able to emit DXIL directly..’ but in practice, none actually do.

Instead, basically every WebGPU implementation today behaves as follows:

The WGSL textual language first gets translated to HLSL at runtime
HLSL is compiled into DXBC or DXIL using an HLSL compiler
The optimized DXBC/DXIL is handed to the graphics driver, which then gets converted to the various vendor-specific ILs before finally becoming machine code that runs on the GPU.

A quick detour: SPIR-V

Vulkan/SPIR-V does much the same as the above, in fact most drivers cannot assume SPIR-V is optimized at all - though some do, and this varies by mobile/desktop GPUs - and have more work to perform to get SPIR-V turned into a driver-compiled native binary.

Valve has Fossilize and maintains caches of each specific (GPU, driver version, etc.) pairing along with the actual driver-compiled binary for a SPIR-V blob, to enable downloading ‘pre-cached shaders’ from Valve servers ahead of playing games for this reason: so that you don’t spend all day waiting for your computer to go brrr compiling and optimizing SPIR-V shaders into actual native code your GPU understands.

In other words, DXIL is always post-optimization-passes LLVM bitcode, while SPIR-V can or cannot be an an optimized form, and GPU manufacturers write their drivers based on what SPIR-V looks like in the wild - which may or may not be a pre-optimized form. SPIR-V is closer to hardware than a textual shading language, but still very far from native machine code a GPU understands.

Only Apple’s Metal graphics API supports compiling directly to the actual target hardware’s native binary format (thanks to that iron fist they hold over their hardware, I guess.)

To use dxcompiler.dll or not?

Since WGSL->HLSL->DXIL is happening at runtime, WebGPU runtimes are faced with a challenge: do we use the new DXC HLSL compiler, or the old, officially deprecated FXC compiler which has worse performance and codegen quality? On the surface, this hardly sounds like a difficult choice!

However, despite this, many indie devs and game engines choose to use FXC by default. Bevy game engine’s documentation puts it really well:

The Fxc compiler (default) is old, slow and unmaintained. However, it doesn’t require any additional .dlls to be shipped with the application.

The Dxc compiler is new, fast and maintained. However, it requires both dxcompiler.dll and dxil.dll to be shipped with the application. These files can be downloaded from https://github.com/microsoft/DirectXShaderCompiler/releases.

As a result, much software defaults to the old, slow and unmaintained compiler. And it’s not just Bevy: wgpu Rust users, Dawn WebGPU users, etc. are all faced with this same question. It’s likely one of the reasons WebGPU does not support Shader Model 6.0+ functionality today - using the DXC compiler is not so pleasant: it is after all a large, clunky Microsoft fork of a C++ codebase from nearly a decade ago!

Well, why not just statically link against it?

You can’t.

Firstly, there is the issue that Microsoft’s fork of LLVM doesn’t support statically linking. On the surface, this appears just to be due to some cmake files assuming SHARED instead of STATIC when creating libraries, but if you dig into it - as I did - you’ll soon find it is much more involved than that.

Switching SHARED to STATIC everywhere in CMake files will appear to get you a build with ~15 different static libraries to link against (not pleasant compared to just one.) You might think using cmake OBJECT libraries could solve this, but with this you will quickly encounter an issue where although the cmake files are structured logically as dependants, they actually have implicit dependencies on eachother due to the HLSL changes Microsoft made. I am 80% sure you would need to rewrite every cmake file in the repository to support OBJECT libraries. I can say this, because I tried!

You might be thinking, linking against ~15 static libraries isn’t SO bad as long as the final executable is static, right?

Not so fast - many parts of DXC’s COM interface implementation is also explicitly designed to load itself as a DLL, i.e. to load dxcompiler.dll and dxil.dll as dynamic libraries and self-invoke methods.

OK, we just need to patch the implementation to not call LoadLibraryW then, basically, right?

Introducing dxil.dll - the proprietary code signing blob for DirectX shaders

If you’ve ever built DirectXShaderCompiler from source, you might notice something: dxil.dll doesn’t get built. Why? It’s distributed in every release on GitHub, both for Windows (x86/arm) and Linux (x86 only).

Strange, I thought the compiler was supposed to be open source? Well, it wouldn’t be the first time[0][1] I’ve encountered a Microsoft ‘open source’ repository that actually completely depends on some proprietary platform-specific code blobs behind the scenes.

Incidentally, I stumbled across the D3D12 Shader Cache API specification which mentions the existence of this proprietary code signing blob as a ‘good reason for invoking the shader compiler at runtime’:

D3D12 will only accept signed shaders. That means that if any patching or runtime optimizations are performed, such as constant folding, the shader must be re-validated and re-signed, which is non-trivial.

And in the recent ‘preview release’ for Shader Model 6.8 functionality, Microsoft notes how they appear to leverage this DLL to restrict new experimental shader functionality:

The DXIL signing library (dxil.dll/libdxil.so) is not provided with this preview release. DXIL generated with this compiler targeting Shader Model 6.8 is not final, cannot be validated, and is not supported for distribution or execution on machines not running in developer mode.

In other words: if you do not have dxil.dll, then your shaders will not be signed/validated. If your shaders are not signed/validated, then they cannot run on a Windows machine unless it is running in Developer Mode.

Platform support challenges

For a second, I’d like to go back to something I wrote at the start of this article:

For Mach engine […] we need to compile shader programs into something that Direct3D 12 can consume.

I’d like for us to be able to perform offline shader compilation, and skip out on distributing the heavy DXC dependency, when desired.

But Microsoft only distributes a copy of dxil.dll for Windows (x86/arm) and Linux (x86). There’s no Linux aarch64 binary. There’s no macOS binary. In other words, you can’t produce builds of your cross-platform game for Windows using offline shader compilation on a mac, or in your Arm Linux CI pipeline. You need a Windows or x86_64 Linux machine to run the proprietary blob.

Recap

To recap:

We cannot build DXC as a static library, because the decades-old Microsoft fork of LLVM v3.7 has a very messy build-system.
Even if we could, we cannot build DXC as a static library because of the proprietary code-signing blob.
We cannot compile DirectX HLSL shaders offline on a Mac, or build our cross-platform game in an arm Linux CI pipeline, because Microsoft doesn’t distribute copies of the proprietary code signing blob for those platforms.

Going deeper

Un#$@&%*! the build system

The first problem I wanted to address was how to actually build this codebase into a single static library.

After several days of attempting to fix the implicit dependencies that changing the cmake virtual libraries from DYNAMIC -> OBJECT surfaces, I gave up. Originally, my intent was to use their existing cmake build system (so as to not diverge from their codebase too much) and just swap the compiler with zig cc as the build toolchain for cross-compilation.

After it slowly and painfully became apparent that direction was not going to be any better than maintaining the entire buildsystem myself, I decided to just bite the bullet and rewrite the entire CMake build system they had, some ~10.5k lines of code, using build.zig instead. To make things simpler, I chose to build only the two parts we (and others) really care about as consumers of the code: the dxcompiler.dll library, and dxc.exe binary for offline compilation / testing. (we’ll deal with dxil.dll later.)

This resulted in somewhere around ~1k lines of build.zig logic, and in practice it’s less than that because much of it is just related to running git clone on the source repository, having the ability for Zig package consumers to use a prebuilt binary instead of building the large C++ library from source, and header/source generation (though we’re still not done with that, thanks to llvm-tablegen)

Un#$@&%*! the dynamic library dependency

As mentioned earlier, DXC is written with the expectation that dxcompiler.dll and dxil.dll exist. Reading the code, it almost appears as if the COM API implementation invokes the DLL, which then invokes itself dynamically depending on which is available.

Taking some advice from Microsoft, I got my hands dirty, forked their codebase and got to work on the actual C++ code. I began annotating my changes with cute // Mach change start and // Mach change end comments, to know who owns what code. All of this existing as a choice that I hope will come back to haunt my dreams in the future as much as Microsoft’s own choice to underemploy the HLSL team and fork LLVM 3.7 originally.

I was off to the races: simulating dllmain entrypoints, disabling the ability to print the compiler version info derived from the dlls, and emulating dynamic library function pointer loads.

Un#$@&%*! the proprietary code signing

All that was left was that pesky dxil.dll - what sort of magic might Microsoft be employing in that library to “sign shaders”? How can they prevent unsigned shaders from running on Windows machines that aren’t in developer mode? How are they able to distribute that binary on Linux, too?

I won’t comment on any of those questions, but will say that you’ll find dxil.dll is NOT a dependency of mach-dxcompiler in any form. You can compile an HLSL shader on a macOS machine using mach-dxcompiler, without the proprietary dxil.dll blob - and end up with a DXIL bytecode file that is byte-for-byte equal to one which runs it on a standard Windows box. Enjoy!

Results

We now have prebuilt, static binaries of the dxcompiler library, as well as the dxc CLI here, with zero dependency on the proprietary dxil.dll. At the time of writing, we have binaries building in our CI pipeline for:

macOS (the first ever in history), both Apple Silicon (aarch64) and Intel (x86_64).
Linux, including musl and glibc, as well as aarch64 (first ever in history) and x86_64.
Windows, x86_64 and aarch64, including for MinGW/GNU ABI (first ever in history?)

Additionally included is a small C API the library now exposes, as an alternative to the COM API traditionally required.

Zig game developers will find the repository also includes a Zig API, see src/main.zig tests for usage. By default prebuilt binaries are downloaded/used.

You can build from source yourself for any OS/arch with only zig and git, just make sure you have the right Zig version:

git clone https://github.com/hexops/mach-dxcompiler
cd mach-dxcompiler/
zig build -Dfrom_source -Dtarget=aarch64-macos
zig build -Dfrom_source -Dtarget=x86_64-windows-gnu
zig build -Dfrom_source -Dtarget=x86_64-linux-gnu

Caveats

It’s not all roses - there are some drawbacks:

Windows MSVC ABI binaries are currently not building due to a small bug in the C bindings - will fix it quickly if important for you, otherwise at our own pace.
Linux musl binaries are untested, they build fine and I’d be curious to know if they run fine!
With Mach engine, we plan to use Zig itself as our shading language, not HLSL, so I do not build SPIRV-output support, sorry! I have no plans to add it.
No plans to update this to support SM6.7 currently (released very recently), though perhaps in the future.
LLVM’s cmake build system is not trivial, there are some aspects yet-to-be-translated. See generated-include/ for specifics which come from the cmake build system still.
If you use this, you’ll be relying on myself to fix/address any issues. I am the only person working on this, and it exists solely to solve Mach’s own problems. If it works for you, great - but there may be a time we find a better path forward for us and it could get deprecated, so keep that in mind.

On a personal note

My name is Emi, I work a normal tech job, and after signing off from work at the end of the day I go online to build Mach engine. I’ve been dreaming of being able to build a game engine like this for a long time, and I’m finally doing it!

FOSS is in my roots, I believe we should own our tools, they should empower us-not be part of the ‘open source’ game which is all too prevelant today (even among ‘open source’ engines.) I need Mach to genuinely be software you can love.

My dream is one day to live a simple, modest, life earning a living building Mach for everyone and selling high-quality games. Please consider sponsoring my work if you believe in my vision. It means the world to me!

Thanks for reading

Check out machengine.org
Consider sponsoring development so we can do more of it!
Join the Mach Discord server

Mach v0.3 released - Zig game engine & graphics toolkit

Fri, 02 Feb 2024 00:00:00 +0000

Mach is a Zig game engine & graphics toolkit for building high-performance & modular games, visualizations, and desktop/mobile apps. Learn more

We are working towards Mach 1, and have just released v0.3 which includes 6 months of work - here are the highlights!

Coming soon: intro to 2D gamedev workshop

The first-ever intro to 2D gamedev workshop using Mach will be hosted at the Software You Can Love conference in Milan, Italy, May 14-17. The workshop will use Mach’s currently in-development higher level 2D graphics APIs.

If you’re interested in Zig or Mach, then check out the SYCL conference! It’s an amazing experience, a great opportunity to meet a ton of Zig community members, core team members, as well as enjoy some of the best food that Italy has to offer!

Community highlight: Pixi and Scoop’ems

@foxxne is an early adopter of Mach core, largely pushing it to its limits. They make use of Mach’s new experimental sysgpu graphics API (which we intend to be a successor/descendant of WebGPU), as well as other libraries like flecs and dear-imgui. They’re developing Pixi - a pixel art editor:

And have used it to make games like Scoop’ems:

@foxxne is making awesome tools and games in Zig, pushing things to their limits, I encourage watching how humble Colton is when speaking about their work. We are very excited to make Mach support foxxne’s projects better in the future, and enable others to build things like this too.

Please consider sponsoring their work on GitHub - you could be their second-ever sponsor!

Mach core

Mach core aims to provide just a window, input, and truly cross-platform graphics API.

We think of it as an alternative/competitor to the classic options of SDL+OpenGL, GLFW+Vulkan, etc. Today, it’s not quite there yet - it uses GLFW behind the scenes for desktop support, and WebGPU as its graphics API, but we’re actively working on making it a genuine competitor written in Zig.

In this release, it saw general bug fixes - as well as some libmach development - which aims to provide a C API to both Mach core and engine APIs.

sysgpu

In Mach v0.2, we announced an experiment - that we were working on a WebGPU implementation written in Zig, as an alternative to using Dawn (Google Chrome’s WebGPU implementation.) In the past 6 months, this experiment saw an immense amount of development and exceeded our expectations!

sysgpu today is a nearly fully-functional WebGPU native implementation (minus browser-level safety checks), thanks to two amazing contributors. It has functional D3D12, Vulkan, Metal, and OpenGL backends. It has it’s own WGSL shader compiler, and nearly all mach-core examples are runnable using it. We’ve even seen real applications (the Pixi pixel editor from foxxne, for example) begin to adopt it.

As we continued development of it over the past six months, we identified key design tradeoffs where we could differ from WebGPU’s API choices and gain a faster, more modern, featureful graphics API. As a result, we’ve come to view sysgpu as a leaner and meaner successor and descendant of WebGPU for native graphics, rather than just another implementation of it. As a result, it builds on the back of WebGPU’s design choices, but ultimately has its own distinct API and will not be ABI-compatible.

We have plans to alleviate some major pain points of WebGPU, specifically around pipeline creation / descriptor boilerplate, supporting push constants when available via a better API design (not as an extension), a more integrated/seamless approach to binding resources to shaders with type-correctness, and more.

We are also evaluating using Zig itself as the shading language, instead of WGSL, and are looking to enable fully offline shader compilation as an optional feature.

sysgpu is still under heavy development, particularly all of the ‘successor/descendant’ API design choices noted above have not been implemented yet. It is disabled in the v0.3 release by default, and after this release we plan to invest more aggressively in it - so expect more details and specifics to come soon.

sysaudio

As a bit of background, mach-sysaudio started out as Zig bindings to Andrew Kelley’s fantastic C library libsoundio, but ultimately it grew to stand on its own two feet - becoming a brand new library written in Zig from first-principles and the ground-up to achieve similar goals: providing just low-level audio input/output, nothing else. It saw a good deal of polish in this iteration:

SIMD sample conversion support - and sample conversion is now explicitly optional (so the default is to work with the driver’s active audio format.)
Major API design improvements
Fixed issues with microphone/input devices, specifically multi-channel devices on macOS with CoreAudio.
Fixed various issues with the WASAPI/Windows backend.

Audio synthesizer hack-project

As a quick hack project over the holidays, I leveraged Mach’s audio libraries, reading midi input from a piano keyboard, synthesizing audio using Zig code / Mach, and playing it back through digital piano speakers. Here’s a few vertical videos of me being silly & having fun with it (skip to 2:24 to hear how I think a Ziguana might sound!):

Nominated Zig versions

Mach has always needed a sweet-spot between stable Zig and nightly Zig, a better balance of latest-and-greatest features and bug fixes, but less of a moving target than nightly Zig. To address this, we formalized how we nominate Zig versions for use, enabling others to synchronize their Zig version with the one Mach supports more easily.

Mach engine as a standard library of modules

We began documenting how we view Mach engine as a standard library of modules for game development, and how we’ll enable you to use just the parts you wish. This is a small-but-important step in showcasing how the engine’s higher level APIs will be more modular than the monolithic big engines of today.

Entity component system

The Mach entity component system provides a key role in Mach’s modularity, in this iteration it saw numerous polish / bug fix improvements - the ability to actually query entities, a more clear/concise API, etc. It is still under heavy development, however.

mach.math

mach.math was introduced to the Mach standard library: a custom math library tailored towards our graphics API conventions, matrix representations, coordinate systems, etc.

Today it includes many of the basics: vectors, matrices, quaternions - though it is still missing some basic tablestakes. It also has ray-triangle intersection, and we intend to expand it to cover more general collision utilities later.

A new set of math docs were added to the website with some cute diagrams/visualizations.

mach.gfx.Sprite

mach.gfx.Sprite was introduced, which is the start of a 2D sprite-rendering module. It is largely usable, though we anticipate its API to change a fair amount and are looking to add animated sprite support among other key features.

It has been useful in letting us test basic rendering of hundreds of thousands of sprites, each as separate entities with their own transformation matrices calculated CPU-side, and get more of an end-to-end feel for how things are looking with our ECS:

mach.gfx.Text

Development of a basic text rendering module is underway, but not ready for use yet.

Status of ‘simple 2D game’ support

In short, we’re still working on it. More to come soon.

General project maintenance

Two new examples rgb-quad and textured-quad showing off super basic 2D rendering were added.
Began to formulate our hardware support plans, such as when we will target certain SIMD instruction sets.
All of our build.zig scripts went through a great deal of changes and improvements, as Zig’s build system and package manager matured greatly.
Various other issues were addressed.

A personal note

I work a normal tech job, and most days after I sign off from work I go online to build Mach, often like working two jobs. I’ve been doing this for a few years now, and dreaming of being able to build Mach for a decade before that.

FOSS is in my roots and I believe we should own our tools, they should empower us-not be part of the ‘open source’ game which is all too prevelant today (even among ‘open source’ engines.) Mach needs to be for people like you and me-it needs to genuinely be software you can love.

My dream is one day to live a simple, modest, future earning a living building Mach for you and creating high-quality games for everyone. Please consider sponsoring my work if you believe in this vision.

Thanks

Immense thank you to all those who helped make this release possible, to those who contribute regularly or in the past, and those who sponsor development. It means the world!

Join the Mach Discord server
Check out machengine.org
Consider sponsoring development so we can do more of it!

Announcing Mach nominated Zig versions

Sun, 07 Jan 2024 00:00:00 +0000

Today we’re announcing Mach nominated Zig versions, a sweet-spot between stable Zig and nightly Zig which offers a different balance of latest-and-greatest features and fixes, and less of a moving target.

If you are in the Zig community, you likely fall into one of two categories:

You target Zig nightly, a target which moves every day
You target Zig stable, which may be released once or twice per year.

The challenge of using Zig stable

In recent years, Zig stable has about 2 releases per year. There are great benefits to using stable Zig:

It is generally documented how to update/migrate your code to the new Zig version, rather than it being an ad-hoc process of discovery.
The release is at a point where the Zig core team feels it is solid and ready to go (Zig releases are done when they are done, not usually timed.)
These versions get nice numbers associated with them, package authors, distributions, etc. can easily target them.

But, for a language that is developed quite quickly, and has yet to reach v1.0 stability.. when you find that an awesome new feature like the package manager, incremental compilation, or something else you care about has just landed in Zig nightly.. are you going to wait 6 months or more to get that change?

The challenge of using Zig nightly

Zig nightly is an ever-moving target. Although it is often nearly as stable as stable releases, that is not neccessarily the case during large refactors - such as the migration to the self-hosted compiler.

There are benefits to using nightly, though! It means you are testing the latest version of Zig, and your project can exist in a sort of symbiotic relationship with the Zig project where you test new functionality, help provide feedback on it, discover new issues, and have a greater chance of getting them fixed/addressed while that code is on everyone’s mind.

Unlike stable Zig, you’re not integrating 6+ months of breaking changes into your codebase all at once (quite painful!) but rather doing so as the breaks happen. Since many others in the Zig community do target nightly Zig, there are often people around who can help you with upgrading your code.

One major downside, aside from death by a thousand paper cuts, is that everyone targets a different Zig nightly version. Often, it’s difficult to coordinate with others and keep your code compatible with theirs.

The challenge Mach has using Zig nightly

We use Zig nightly, but we only periodically update the version.

Updating Mach’s Zig version involves updating a dependency tree of over 40+ Zig repositories, first updating the Zig code itself and testing manually, then updating their CI pipelines, then going and updating anyone who depends on that repository - from the bottom of the tree to the top!

We’ve built some serious automation to help with this process, but it is still painful - and it is also the case that every time we update our Zig version, our users need to do the same: we’re not just updating Zig for us, we’re updating Zig for every user of Mach.

This process has worked pretty well for us, but the frequency of updates has always been ad-hoc, and when we do update it is a bit chaotic.

We need our ecosystem of 40+ packages to be compatible

If we haven’t updated our Zig version for a bit of time, then we end up with tens of outstanding pull requests by people who maybe don’t use all of Mach but rather just parts of it, and they can become (rightfully!) fairly frustrated that getting their pull-request merged takes us a while.

We can’t just merge a one-off pull request to one repository, we have to update all 40+ to be compatible with the same Zig version.

Coordinating outside Mach

Although Mach provides a lot of libraries, there are still many important aspects of gamedev we do not have yet. Some folks in the Mach community will pull in third-party Zig projects, like those from zig-gamedev, introducing another challenge in ensuring their code works with the same version.

Announcing Mach nominated Zig versions

Today we’re formalizing the process we’ve (generally) been following. This formalization will make it easier for others to understand what we’re doing and when, and also make it easier for other projects to align their Zig version with Mach’s if they desire.

Throughout the year (aiming for the 4th day of the month), we will pick the latest Zig nightly version at that time and nominate it for use:

When	What	🚀 Other notable event
January		🚀 Mach version release
February		👋 Anticipated influx of new Machanists / Ziguanas
March	⚡ Zig version nominated
April
May	⚡ Zig version nominated
June	⚡ Zig version nominated
July		🚀 Mach version release
August		👋 Anticipated influx of new Machanists / Ziguanas
September	⚡ Zig version nominated
October
November	⚡ Zig version nominated
December		👋 Anticipated influx of new Machanists / Ziguanas

These versions will be noted as e.g. ‘2024.1.0-mach’, and will correspond to a specific Zig nightly version from that month.

The exact versions, which nightly version they map to, and the whole process (which is more involved), is documented in full here.

At the time of writing this, you’ll see that 2024.1.0-mach is marked as ‘in progress’ - we will make sure that the Zig version we intend to nominate is at least compatible with all Mach projects before finalizing the nomination.

A sweet spot between nightly and stable

Mach’s nominated Zig versions provide a different set of tradeoffs, we believe it is a sweetspot between the two extremes of nightly and stable. You can benefit from the changes in Zig 2-3x faster than if you were using stable, and suffer less from the never-ending game of catch-up and incompatibilities between projects that nightly necessarily requires.

Other projects can target the same Zig version if they wish to be compatible with Mach Zig packages. For example, zig-gamedev is aiming to target the same versions. We encourage other gamedevs using Zig to do the same.

Projects that target nightly Zig can often be coincidentally compatible, too, since they possibly had a compatible Zig version around the time we nominated a Zig version for use.

Final thoughts

You can read more about the specifics of everything in the Mach documentation.

We’re currently working on nominating the first version, which will likely be finalized in the next week or so.

Thanks

Join the Mach Discord server (check #discuss for this article)
Checkout machengine.org
Consider sponsoring development so we can do more of it!

Mach v0.2 released - Zig game engine & graphics toolkit

Sat, 12 Aug 2023 00:00:00 +0000

Mach is a Zig game engine & graphics toolkit for building high-performance, truly cross-platform, robust & modular games, visualizations, and desktop/mobile GUI apps. Learn more

We’ve been developing Mach for ~2 years; this release includes over a year of work, thousands of commits, and fixes 300 issues.

On your machine in just ~60 seconds

With this Zig nightly version you can run the above demo on your machine in ~60 seconds:

git clone https://github.com/hexops/mach-core
cd mach-core/
zig build run-deferred-rendering

Zero system dependencies to slow you down; only zig is needed, we build and package the few relevant dependencies on our own. known issues

Engine and Core split

We completely split Mach engine and Mach core apart, so that you get to choose your journey and decide if you just want low-level window+input+GPU and nothing else, or prefer to use our higher level engine (which although not ready for use yet, will be very modular itself):

Mach core

Mach core aims to be a truly cross-platform way to get window+input+GPU, and nothing else. It supports Linux, Windows, and Mac today, with WebAssembly / browser support in active development, and mobile coming in the future.

It gives you the power of Vulkan, DirectX, Metal, and modern OpenGL in a single concise graphics API and shader language - by compiling Google Chrome’s WebGPU implementation natively using Zig’s build system.

Seamless multi-threading capabilities are provided, which means that your rendering and input handling are trivially decoupled from one another, you get butter-smooth window resizing, and your render loop and input handling can run at different frequencies. For example, a 60FPS render loop while your application handles keyboard & mouse events at a much faster dynamic rate (as fast as the OS can deliver them.)

You can think of Mach core as an alternative to the classic options of SDL, GLFW+OpenGL, etc.

There are 15+ examples in the showcase, and we’re planning a C API so it can be used from other languages as well.

Engine development has begun

Mach engine is not ready for use yet, but we’ve started breaking ground on higher-level engine APIs.

The v0.2 release focuses on deep changes and improvements to our infrastructure, primarily building out the Zig gamedev ecosystem and building foundational packages that we needed for Mach core, the engine, and a game we’re starting to build.

As a result, we’ve finally just broken ground on the engine side of things.

Breaking up our monorepo

Previously, all of Mach’s standalone packages were developed in a single giant monorepo. This was both intimidating for new contributors, and we wanted to better communicate how many standalone Zig gamedev packages we actually provide.

Today, we’re happy to report all standalone packages are now developed in separate repositories and available via the package manager!

Zig package manager

We migrated 100% to the self-hosted Zig compiler and the new experimental Zig package manager, every Git submodule has been banished!

We created pkg.machengine.org - a mirror for downloading Mach packages and Zig downloads as well.

Introducing Wrench the Machanist

Wrench is the Mach engine mascot (artwork contributed by Keyla Jones); and also our infrastructure automation tool, written in Go, to help us with various tasks:

Giving us an overview of our many repositories CI statuses and pull requests:

Sending us pull requests to automatically update our CI pipelines to the latest Zig version, and update our build.zig.zon dependencies - in a fully atomic way across all our repositories at once:

…and much more:

Checking our website and docs for broken links and sending us GitHub issues.
Performing automatic updates of involved dependencies, such as updating our fork of Google Chrome’s WebGPU implementation, which involves pull requests across a few different repositories, pushing branches, running out-of-band commands, and ultimately presenting us with helpful/pretty diffs so we can just do the human work.
A custom CI job runner system, running on a custom mini server with actual GPUs - so we can do screenshot-based testing of graphical applications in the future.

mach-gpu: rewritten for perfection

mach-gpu is the WebGPU interface for Zig, and last year we completely rewrote it, achieving:

Zero overhead, using comptime interfaces
100% API coverage
Default values for 100% of the API (which makes writing descriptors, and makes examples, look much simpler.)

Audio development

Contributor @alichraghi has been relentless in pushing our audio capabilities (and more) forward

mach-sysaudio started as Zig bindings to Andrew Kelley’s awesome libsoundio library, and ended up being a fully-fledged new library written in Zig to achieve similar goals:

Truly cross-platform, low-level, audio IO in Zig - playback and recording with backends for:
- Linux
  - PulseAudio
  - PipeWire
  - Jack
  - ALSA
- Windows: WASAPI
- macOS/iOS: CoreAudio
- WebAssembly: WebAudio

Then just recently we got mach-flac and mach-opus, which combined give you FLAC (lossless audio) and Opus (lossy audio) via the respective battle-hardeneed xiph.org libraries.

New website

We built a brand new website that will serve us well into the future, featuring:

Project roadmap
Documentation for Engine, Core, Packages, and more.
WebGPU documentation and learning material
Offline-viewing support (see link in the footer)
All around better design, landing page, etc.

Note: Mobile support isn’t great right now, we’ll fix it later.

Browser support: in development

Chrome has already shipped WebGPU, and others will follow soon. Mach support for WebAssembly is not yet ready, but is coming along nicely:

Input, audio, etc. is working already (piano demo, click in the frame and type with your A-Z keys.)
mach build is a new CLI command written in Zig which:
- Starts an HTTP development server
- Invokes zig build for you when you reload the page
- Generally provides a nice browser development experience

We currently do not have graphics support in the browser: we are currently doing a full rewrite of mach-sysjs to use a code generation approach to enable Zig and JavaScript to communicate across the WebAssembly boundary, while being able to pass complex types (strings, slices, etc.) instead of just pointers and integers as normal. Once this rewrite is finished, we will generate a gpu.Interface implementation which simply invokes the browser’s JavaScript WebGPU API with minimal overhead.

Dusk: Experimental pure-Zig WebGPU implementation

Dusk is a highly experimental WebGPU implementation in Zig, aiming to be blazingly fast, lean & mean.

@alichraghi has been quietly working on Dusk tirelessly and consistently over the past year, and we think it may begin to be usable in Mach v0.3. Although it is not usable today, it already features:

A full WGSL shader parser and compiler, based loosely on the Zig compiler, capable of emitting SPIRV.
A partial Vulkan implementation.

Dusk is a long-term bet / investment for us, we intend to always have the option of using Dawn (the Google Chrome WebGPU implementation), and we don’t expect Dusk will be the default very soon. Since both will implement the same gpu.Interface, it’ll just be another backend you can select from at build time.

Learn more about our goals with Dusk here, and feel free to join the #dusk channel in Discord or check out the repository if you’re interested in contributing some Vulkan, Metal, or Direct3D knowledge.

Model loading

mach-model3d provides Zig bindings to Model3D, a compact, featureful model format & alternative to glTF. We may replace this with our own model format in the future, but for now this enables us to load models from Blender in a decent, performant way.

Sprite / 2D examples

mach-core now has a sprite2d example which is ~400 lines and demonstrates loading sprites from a JSON file and sprite atlas, basic keyboard movement, etc.

We’ve already begun making a higher-level API for 2D graphics, as well - though not ready for use yet.

Community

Our Discord community grew to over 700+ members, though we aim to keep all valuable information in GitHub issues and on the new website.
We attended Software You Can Love in Milan, Italy - gave a talk, had a table full of Zig gamedevs, and gave out some cool stickers
We saw a number of new contributors, both one-off and ongoing.
Many coffee was drank, and much coding was done over the holidays.

A personal note

I work a normal tech job, and every day after I sign off from work I go online to build Mach, almost like working two jobs. I’ve been working on Mach double-time like this for over two years now, and dreaming of it for a decade before that.

Thanks

Both to everyone who has contributed and sponsored the project, as well as you for reading this far!

Join the Mach Discord server
Checkout machengine.org
Consider sponsoring development so we can do more of it!

Mach: providing an ecosystem of C libraries using the Zig package manager

Wed, 14 Jun 2023 00:00:00 +0000

Andrew Kelley gave a keynote speech at Software You Can Love 2023 in Vancouver last week (a recording will be available later), the outline was:

How to Build Software From Source

[…] then I’ll take things in a completely different direction, by showing you how to rip apart a project’s build system and replace it with the zig build system. This will make building things from source work effortlessly for more people and more platforms, as well as annoy a lot of boomers. It’s going to be super fun and spicy!

As we all know, Zig is a C/C++ compiler and its own build system

Zig is really three things:

Programming language
Build system (build.zig) replacing makefiles/cmake/ninja/etc
C/C++ compiler with an emphasis on cross-compilation

We’ve been leveraging all three in Mach engine for a while now. For example, we maintain a version of Google Chrome’s WebGPU implementation (Dawn) with its rather complex build system (code generation, python scripts, depot_tools, ninja, cmake, depot_tools, etc.) replaced with build.zig.

That let’s us say that if you have a recent Zig version you can get started with Mach in ~60s on Windows, Mac, and Linux:

git clone --recursive https://github.com/hexops/mach-examples
cd mach-examples/
zig build run-textured-cube

And instead of getting a bunch of dependency errors that you might need to apt-get install or whatever, you’ll just get something that works out of the box:

Zig has a new package manager (for C/C++ too!)

Those in the Zig community know that Zig has a new package manager, it’s built into the compiler. Effectively you describe your dependencies in a build.zig.zon file, and then the zig compiler is able to fetch them for you as part of zig build. You’re then able to link against/use dependencies in your build.zig file, which declaratively says how to build your project (except, using a real language instead of a DSL like cmake/etc use.)

It’s still very experimental, has little to no documentation yet - it’s not ready for widespread use. But one strong point is that it also aims to address the issue of building C/C++ projects, not just Zig ones. You can write a build.zig file in Zig, describing how to build your C/C++ project using Zig as the toolchain. Then for free you get quite solid cross-compilation (since Zig bundles clang, every glibc version, and more), plus now a dependency manager, as well as a declarative way to describe your build using the Zig language.

One example of this is in Andrew Kelley’s fork of ffmpeg, where he merely forked the ffmpeg repository, removed their build system & unnecessary files, and added a build.zig file. This allows you to clone the repository and zig build will fetch all the required dependencies and build ffmpeg for you. Fancy!

Mach engine

Mach engine is an upcoming game engine built in Zig, that we’re building with the aim of becoming competitive with Unity/Unreal/Godot - but with an emphasis on modularity:

Mach core: If you choose to use core, then it’s like an alternative to GLFW+OpenGL or SDL, you just get a window+input+WebGPU with minimal dependencies. Your application runs natively on Windows/Linux/Mac using their respective graphics APIs (DirectX12, Vulkan, Metal), you get cross-compilation and zero-fuss installation, and also web/mobile support in the future with the same codebase. Write once, run everywhere.
Mach engine: If you choose this option, you additionally get an entity component system - with a library of standard modules that you can ‘plug and play’ with for rendering/audio/etc.

Keeping our runtime C dependencies small

One way that we’re keeping runtime C dependencies (the ones your game/app would ship with!) a smaller, focused, set - is by building tooling: a mach CLI and fully-fledged GUI editor like other engines have. But how does that help reduce runtime dependencies? Well, at runtime you may need:

Harfbuzz: for Unicode text layout
GLFW (and some headers): for window management)
Basisu and PNG: for GPU supercompressed textures / lossless textures
Opus and FLAC: for lossy and lossless audio

Mach will ‘bless’ certain formats, being opinionated in what you ship with your game. You’re free to pull in other formats, if you like, but the default/easy path will be these ones. As a result, there’s a lot we won’t need at runtime:

Freetype
JPEG, TGA, or other image formats
MP3, ffmpeg, or other audio formats

We don’t need these because our tooling (the CLI and GUI editor) is going to make it easy to convert whatever format you want into the ‘blessed’ runtime formats. One major benefit of this is that we can nudge you to the right defaults, without you being an expert. For example, you probably want to be using texture compression formats that GPU hardware itself understands, instead of say shipping a JPEG that just gets expanded to an uncompressed texture, eating a bunch of GPU memory and harming your texture bandwidth.

Providing an ecosystem of C libraries

Similar to Andrew Kelley’s ffmpeg fork (although, with a few niceties to verify the supply chain) - Mach is now maintaining forks of various C libraries that we make use of. These aren’t Zig bindings to these libraries (which we have separately), but rather are just forks of the actual project with their build system replaced by build.zig.

A massive special thanks to @LordMZTE who has been tirelessly pushing us along here over the past month-ish, helping to inch us ever-closer to fully adopting the new package manager.

Forks we maintain

We have forks of these projects which switch their build systems to build.zig:

hexops/harfbuzz
hexops/freetype
hexops/brotli
hexops/glfw
hexops/basisu (basis_universal, supercompressed textures)

A note about supply chain verification

I personally care a lot about supply chain security - and more importantly, bugs. In general, I don’t ever want anyone to have to ‘wonder’ if our fork of a library has some strange patches applied to it or something.

As a result, in each of these forks we’ve taken the time to ensure you know the exact git diff command you can run to verify that our fork exactly matches the upstream version - with the only difference being removing the project’s old build system, and unnecessary files.

Header packages we’re maintaining

In addition to the above, we’re maintaining the following which aren’t strict forks (a repository for each would simply be too much for us to maintain), but rather are collections of common headers that you very often need together. These can help you build GLFW, SDL, and other such applications.

Some headers are generated with platform-specific tools (e.g. in the case of Wayland this is needed.) We always provide the exact steps we used to produce the headers from upstream repositories in an update-upstream.sh script, with the intent that you can fully reproduce what’s in these packages and have confidence it came from the upstream repository.

hexops/vulkan-headers
hexops/linux-audio-headers includes:
- ALSA
- Jack
- PipeWire
- PulseAudio
- SPA
hexops/x11-headers includes:
- x11
- xcb
- xkbcommon
- xcursor
- xrandr
- xfixes
- xrender
- xinerama
- xi
- xext
- xorgproto
- GLX
hexops/wayland-headers includes:
- xdg-shell
- xdg-decoration
- viewporter
- pointer-constraints-unstable-v1
- relative-pointer-unstable-v1
- idle-inhibit-unstable-v1

How do I use these?

Later we’ll provide more details specifically on how to use these. Effectively you just add them to a build.zig.zon file next to your build.zig and then call b.dependency("name") to retrieve each one. If you need more help than that, you might need to join our Discord because as mentioned previously the Zig package manager is pretty immature and has sharp edges today. There are lots of known issues & bugs that prevent even us from using it fully today.

But, it is coming along rather quickly! We wanted to let the broader Zig community know we’re maintaining these packages to help with collaboration.

Help us become sustainable

We’re working towards Mach v0.2, this article was one of the first steps in beginning to share the progress we’ve been making towards that behind the scenes over the past several months. We have some exciting things to share next, this was the ‘boring’ article that had to go first. :)

Consider sponsoring my work to help us become a sustainable OSS project and enable us to do more in the future.

Join the Mach Discord where we’re building the future of Zig game development in realtime!

Zig tips: v0.11 std.build API / package manager changes

Mon, 13 Feb 2023 00:00:00 +0000

We’ve just updated Mach engine to use the latest Zig nightly version, which includes a fair amount of improvements and breaking changes to the std.build API used in build.zig files, and figured now would be a good time to share the general changes you may need to make if you want to update your own code.

Package manager: incoming!

Zig is finally starting to see its package manager and build system shape up, some notable mentions:

std.http.Client and std.crypto.tls were added (#13980)
The package manager MVP landed almost a month ago and has seen steady improvements since (#14265)
Zig packages can now expose C headers are part of their public API (#14449)
Transitive dependencies are now handled better (#14392)
“zig build: The breakings will continue until morale improves.” (#14498)
Zig Object Notation (ZON, an alternative to JSON) was introduced (#14523)
The caching system is being moved from the compiler to the std lib to start using it in the bulid system (#14571)
Zig plans to run the build system in a sandboxed WASM environment (#14286)

You can get an overview of progress on the package manager on this GitHub project board

Mach isn’t yet using the new package manager: it’s improving rapidly, and we plan to make use of it soon, but things are still changing so we’ve held off for now. What we have done, though, is updated to the latest API and want to share those changes with you.

Release options have been renamed to optimization

Previously you would’ve used b.standardReleaseOptions() which would provide your zig build command with multiple options like zig build -Drelease-fast=true, zig build -Drelease-safe=true, etc.

It’s been renamed to b.standardOptimizeOption(.{}) and now exposes a single build option zig build -Doptimize=ReleaseFast, zig build -Doptimize=ReleaseSafe, etc. instead.

-pub fn build(b: *std.build.Builder) void {
- const mode = b.standardReleaseOptions();
- const target = b.standardTargetOptions(.{});
+pub fn build(b: *std.Build) void {
+ const optimize = b.standardOptimizeOption(.{});
+ const target = b.standardTargetOptions(.{});

-mode: std.builtin.Mode
+optimize: std.builtin.OptimizeMode

-step.build_mode
+step.optimize

Creating tests, libraries, and executables

Creating tests, libraries, and executables now takes a struct with options as the parameter instead of using a setter API:

-const exe = b.addExecutable("example", "src/main.zig");
-exe.setBuildMode(mode);
-exe.setTarget(target);
+const exe = b.addExecutable(.{
+ .name = "example",
+ .root_source_file = "src/main.zig",
+ .target = target,
+ .optimize = optimize,
+});

See more examples

Tests:

-const main_tests = b.addTestExe("glfw-tests", sdkPath("/src/main.zig"));
-main_tests.setBuildMode(mode);
-main_tests.setTarget(target);
+const main_tests = b.addTest(.{
+ .name = "glfw-tests",
+ .kind = .test_exe,
+ .root_source_file = .{ .path = sdkPath("/src/main.zig") },
+ .target = target,
+ .optimize = optimize,
+});

Shared libraries:

-const lib = b.addSharedLibrary("glfw", null, .unversioned)
-lib.setTarget(target);
-lib.setBuildMode(mode);
+b.addSharedLibrary(.{ .name = "glfw", .target = target, .optimize = optimize })

-const lib = b.addSharedLibrary("machcore", "src/platform/libmachcore.zig", .unversioned);
-lib.setTarget(target);
-lib.setBuildMode(mode);
+const lib = b.addSharedLibrary(.{
+ .name = "machcore",
+ .root_source_file = "src/platform/libmachcore.zig",
+ .target = target,
+ .optimize = optimize
+});

Static libraries:

-const lib = b.addStaticLibrary("basisu-transcoder", null);
-lib.setTarget(target);
-lib.setMode(mode);
+const lib = b.addStaticLibrary(.{
+ .name = "basisu-transcoder",
+ .target = target,
+ .optimize = optimize,
+});

Renamings

std.build.LibExeObjStep has been renamed to just std.build.CompileStep (beautiful!)
*std.build.Builder has been renamed to just *std.Build (nice, this is used extensively everywhere!)

Modules

Units of code you @import("foo") (previously known as packages) are now known as modules, and packages now refers to a piece of code you download/depend on using the Zig package manager. Libraries is reserved for referring to C-style libraries, .dlls, etc.

These units of code used to be declared as a std.build.Pkg struct:

pub const my_pkg = std.build.Pkg{
 .name = "earcut",
 .source = .{ .path = "src/main.zig" },
};

And added as a dependency using e.g. exe.addPackage(my_pkg);

Now, these are called modules and can be created in a few ways. One is using b.createModule:

const my_module = b.createModule(.{
 .source_file = .{ .path = "src/main.zig" },
 .dependencies = &.{
 .{ .name = "core", .module = core.module(b) },
 .{ .name = "ecs", .module = ecs.module(b) },
 .{ .name = "sysaudio", .module = sysaudio.module(b) },
 },
});

And then depend on that module using e.g. exe.addModule("earcut", my_module);

Notably, modules are created at runtime via the *std.Build now - so you may have some reworking to do if you previously depended on std.build.Pkg being a global constant you could rely on at comptime.

Another option, which may be preferred, is via addModule. It will make the module available to other packages which depend on this package.

You may also like to know that a pair of dependency name + the module can be represented as std.Build.ModuleDependency now.

We’ve just gone for an initial 1:1 translation in our code, but adoption of the package manager will likely mean structuring your code a bit differently than the above, and the package manager is still a work-in-progress.

Thanks for reading

As we work towards Mach v0.2, we’re getting more serious about what stability means for us. Our intent is to enable us to move quickly, while also helping you to update your code. We will be achieving this through articles like this which help you understand & update your code to the latest APIs. Hopefully this has helped you! You can find other zig: Tips here.

Be sure to join the Mach engine Discord where we’re building the future of Zig game development.

You can also sponsor my work if you like what I’m doing! :)

Debugging undefined behavior caught by Zig

Mon, 14 Nov 2022 00:00:00 +0000

Mach engine uses Zig as the C/C++ compiler for almost everything. Unlike other toolchains, Zig enables many more safety checks by default - such as clang’s undefined behavior sanitizer.

Using Zig, we’ve caught undefined behavior in established projects like GLFW[1], the DirectX Shader Compiler[2], and more. Undefined behavior is everywhere, often relatively innocuous, and hard to catch.

Professional C/C++ developers know to run UBSan fuzzers as part of their test suite, but even with that we’ve found e.g. Google Chrome’s fuzzers weren’t running normally at one point, and we caught undefined behavior in Chrome’s implementation of WebGPU as a result[3].

Zig having UBSan enabled by default is valuable, but it can also lead to tricky to debug errors that are a bit annoying to interpret today. And so this article is a walkthrough of how to debug such an error when it arises using Zig and LLDB.

The situation

We’re looking at using model3d in Mach. Model3D is an up-and-coming compact, featureful, universal model format that tries to address the shortcomings of existing formats (yes, including glTF - see their rationale.) It is a small, zero-dependency single-header C implementation which in Zig we can simply import.

As we’ve been testing it with various models, though, we found our Zig program just crashes when we use it:

% zig build test 2>&1|cat
1/1 test_0... The following command terminated unexpectedly:
cd /mach/libs/model3d && /mach/libs/model3d/zig-cache/o/26c4104a1643fed2068dfa9244dfe90e/model3d-tests /Users/emidoots/zig-macos-aarch64-0.11.0-dev.38+b40fc7018/zig
error: the following build command failed with exit code 1:
/mach/libs/model3d/zig-cache/o/679e494577315c1bcc3749ee7068ea2f/build /Users/emidoots/zig-macos-aarch64-0.11.0-dev.38+b40fc7018/zig /mach/libs/model3d /mach/libs/model3d/zig-cache /Users/emidoots/.cache/zig test

As you can see, we’re not getting much info here on why our tests crashed. This is a telltale sign of undefined behavior in Zig (and there’s an open issue to make this error messaging way more clear). When we compile our program with -Drelease-fast, which disables safety checks, we find it runs as expected - which confirms our suspicion about it being a safety check.

Debugging with LLDB

In the output above, we can grab the path to the executable model3d-tests. We can debug it using lldb (I think we should find a nicer way to invoke lldb via zig build test though!):

lldb -- /mach/libs/model3d/zig-cache/o/26c4104a1643fed2068dfa9244dfe90e/model3d-tests

Next we just enter the run command at the (lldb) prompt:

(lldb) run
Process 6830 launched: '/mach/libs/model3d/zig-cache/o/26c4104a1643fed2068dfa9244dfe90e/model3d-tests' (arm64)
Process 6830 stopped
* thread #1, queue = 'com.apple.main-thread', stop reason = EXC_BREAKPOINT (code=1, subcode=0x100033634)
frame #0: 0x0000000100033634 model3d-tests`m3d_load(data="\xc3.:>", readfilecb=0x0000000000000000, freecb=0x0000000000000000, mtllib=0x0000000000000000) at m3d.h:3356:56
3353 model->tmap[i].v = (M3D_FLOAT)(*((uint16_t*)(data+2))) / (M3D_FLOAT)65535.0;
3354 break;
3355 case 4:
-> 3356 model->tmap[i].u = (M3D_FLOAT)(*((float*)(data+0)));
3357 model->tmap[i].v = (M3D_FLOAT)(*((float*)(data+4)));
3358 break;
3359 case 8:

If you squint, you can see stop reason = EXC_BREAKPOINT in the output. This is the sign I was looking for: it tells me that UBSan likely inserted a breakpoint for undefined behavior, and we hit it. There’s some form of undefined behavior going on here!

Thanks to lldb, we also got an exact line number, and see the source code where the crash occurred. Sometimes you may not get this much, in which case you may need to run the lldb up command until you get to a line you can make sense of.

In this instance, we’re running code inside of a for loop - and so one question I have is: what iteration of that loop are we at? We can get this info easily using p i to inspect the i variable:

(lldb) p i
(unsigned int) $1 = 0

This shows us that the i variable is an unsigned 32-bit int with the value 0. It crashed on the first iteration.

(Note: we could also use frame variables to get a list of variables in the local frame (i.e. our function), but in this case it’s quite a few and so I didn’t find that useful.)

Next, I wanted to find out: what would the float at that address actually look like? Luckily, the LLDB p <expr> command actually interprets C-like expressions for us. It’s rather easy to write the C expression for that:

(lldb) p *(float*)(data+0)
(float) $2 = 0.181819007

Here we can see the float at the address of data is 0.181819007 - which is within the normalized range I’d expect for a UV coordinate. It seems correct. My next question was.. is the pointer address aligned? I know UBSan has a check for pointer alignment, so I looked at it:

(lldb) p data
(unsigned char *) $3 = 0x0000000101008251 "\xc3.:>"

Since we’re accessing a float at this address, we’d expect it to be aligned to 4 bytes. We can check this easily by plopping the address into Python and dividing by 4. If there’s a remainder, it’s not aligned:

>>> 0x0000000101008251 % 4
1

What are the consequences?

Many times, UBSan will catch undefined behavior that in practice isn’t really harmful on modern machines you might care about. I wasn’t sure about the consequences of unaligned pointer accesses like this, so I asked someone smarter than myself and filed an issue on model3d which led to some very interesting insights from @bztsrc.

Short answer: the bug is in UBSan

Long answer: it is true that in ancient times there were CPUs that couldn’t handle unaligned access. However that’s not the case with today mainstream processors: x86, ARM, RISC-V, etc. all handle unaligned access out-of-the-box. (Ok, for ARM it’s not out-of-the-box, you have to enable MMU which is surely done by the OS kernel otherwise virtual memory mapping would be impossible.)

[…] The reason for the unaligned access is pretty simple. In the binary bit-chunk that a compressed model file is, there’s obviously no guarantee that a value is aligned. Such compactness is absolutely needed for small file sizes, padding with zeros would insanely increase the required storage requirements.

[…] So the decision I had to make here was: keep UBSan happy but create crappy and slow code, or don’t care about UBSan and take advantage of modern CPU features. I’ve decided on the latter.

Which is a quite compelling argument for this just being noise for our purposes. :)

RISC-V: a notable exception

It’s worth noting (as was pointed out to me by someone much more knowledgable) that RISC-V cores lack hardware support for unaligned accesses[0][1] (‘if sifive doesn’t do this in hardware (unalignment) there’s no way any other risc-v cores do […due to sifive’s sheer popularity in the space]’), unalignment is done by trap handlers instead:

Officially, programs running in any mode but M-Mode on risc-v are allowed to do unaligned accesses but that doesn’t mean it can’t be pretty painful

[0] https://forums.sifive.com/t/ld-sd-alignment/5530/6

[1] https://patchwork.kernel.org/project/linux-riscv/patch/60c1f087-1e8b-8f22-7d25-86f5f3dcee3f@gmail.com/#24313195

I don’t have any RISC-V hardware to test on, so this doesn’t affect us at present, but I figured it worth noting.

Disabling the sanitizer

In our case, we can just disable the alignment sanitizer for this one function:

+__attribute__((no_sanitize("alignment")))
m3d_t *m3d_load(unsigned char *data, m3dread_t readfilecb, m3dfree_t freecb, m3d_t *mtllib)

And with this, our tests pass. And we continue to get all the other benefits and safety checks of UBSan elsewhere.

In this case, though, I opted to just disable alignment sanitization entirely when building model3d by adding the -fno-sanitize=alignment compiler flag to our build.zig to avoid this surprising us in other model3d functions.

Thanks for reading

If you’re writing Zig, C, or C++ code - then I hope this zig: Tip helps you! You can find other zig: Tips here.

Be sure to join the Mach engine Discord where we’re building the future of Zig game development.

You can also sponsor my work if you like what I’m doing! :)

Perfecting WebGPU/Dawn native graphics for Zig

Sun, 11 Sep 2022 00:00:00 +0000

We’ve just finished a complete rewrite of mach/gpu (WebGPU/Dawn bindings for Zig), with 700+ commits, ~7.4k LOC, and 100% API coverage.

WebGPU (not to be confused with WebGL) is a modern graphics API, acting as a unified API to the underlying Vulkan/Metal/DirectX APIs. Despite it’s name, it is also designed for use in native applications via its C API.

Dawn is the C++ implementation of WebGPU by Google, used in Chrome, planned to be shipped to millions of browsers in the not too distant future.

`mach/gpu`: WebGPU for Zig

6 months ago we released Mach v0.1 which enabled the creation of native applications using WebGPU graphics in Zig:

It all Just Works™ out of the box in under ~60s - all you need is zig, git, and curl:

git clone https://github.com/hexops/mach
cd mach/
zig build run-example-boids

(requires Zig v0.10+, see known issues.)

We do all the heavy-lifting behind the scenes for you: building Dawn using Zig as a C++ compiler, rewriting build scripts in Zig (so you don’t need ninja/cmake/etc), package up all required dependencies so you don’t need Google’s depot_tools, and more.

Because of this, cross-compilation to every major desktop OS is available at the flip of a switch:

$ zig build example-boids -Dtarget=x86_64-windows
$ zig build example-boids -Dtarget=x86_64-linux
$ zig build example-boids -Dtarget=x86_64-macos.12
$ zig build example-boids -Dtarget=aarch64-macos.12

But this is old news! We released this 6 months ago-so what’s new since?

Zig + WebGPU showcase (10+ examples)

The new Zig WebGPU demo showcase has 12+ examples you can try on your own machine to begin learning Zig and WebGPU quickly:

Mach core vs. Mach engine

Mach has a choose-your-journey development strategy, where you don’t even have to adopt the entire engine to benefit from it. All the WebGPU examples we provide are Mach core apps: they rely on Mach for window creation, user input, and setting up the WebGPU API - nothing else. Using Mach core, you write your own engine!

Why use this over, say, GLFW and WebGPU on your own? The benefit is that this will work on Desktop, WebAssembly (soon), Mobile (future), and consoles (long term.) You can write Mach core apps in Zig, or other languages via libmach (more on this later.) Think of Mach core as a competitor to SDL/GLFW.

In the future we’ll offer Mach engine apps, where you buy into our ECS, Unity/Unreal-like editor, and other composable building-blocks that make up the engine at your choosing. But this isn’t ready today.

Dawn/WebGPU on the Steam Deck

We believe Linux should be a first-class platform, and because of this we’ve found Mach all Just Works™ right out of the box on the Steam Deck (running natively as a Linux Vulkan application, no DirectX or Proton in the mix.):

A complete rewrite of `mach/gpu` to be lean & mean

When we wrote the initial WebGPU bindings for Zig 6+ months ago, our primary goal was just to get something working to where we could start building out examples: we always knew we’d need to revisit things later, especially as Browser support, the use of native extensions in Dawn (like bindless support in the future, etc.), overhead & other aspects became clear.

We’ve finally done that revisit in a month-long complete rewrite of mach/gpu from the ground up. This brings 700+ commits, zero-overhead bindings, Dawn native extensions, and much more. Here are the highlights.

Righting our wrongs: runtime interfaces

One goal of mach/gpu is to be able to intercept WebGPU API calls, so that we can provide superior debugging facilities in the future (imagine record-and-replay, step-by-step debugging of WebGPU API calls, etc.)

In the old mach/gpu, we achieved this by wrapping each WebGPU API object that had methods (like textures, render pass encoders, etc.) in a runtime interface similar to Zig’s std.mem.Allocator interface:

pub const Texture = struct {
 /// The type erased pointer to the Texture implementation
 /// Equal to c.WGPUTexture for NativeInstance.
 ptr: *anyopaque,
 vtable: *const VTable,
};

pub const VTable = struct {
 destroy: fn (ptr: *anyopaque) void,
 // ...
};

pub inline fn destroy(tex: Texture) void {
 tex.vtable.destroy(tex.ptr);
}

Our thought process was simply to follow any established patterns, learn what didn’t work about it by writing examples, and then revisiting the API later. Even six months ago, though, we knew there were issues with this approach.

The problem: In WebGPU, Descriptor data structures are often passed to methods: these fairly large data structures contain a wide range of options and graphics pipeline state to use, and often involve passing a list of WebGPU objects as a field - or nested field - as part of the Descriptor data structure. Because our Texture involves keeping a ptr (the interface implementation) and a vtable pointer (our implementation methods) it meant that a gpu.Texture was two pointers, while a C WGPUTexture was a single pointer - breaking ABI compatibility.

This meant that our Texture could not simply be passed to a C API expecting a WGPUTexture: instead, we needed to pass our .ptr field only. This had viral effects, though: every Descriptor struct which embedded a Texture needed to be copied/rewritten to convert our two-pointer Texture to a single-pointer WGPUTexture. Worse yet, some descriptors hold dynamic arrays of such objects, requiring us to copy an array to a temporary (and worst-case, heap-allocated), buffer just in order to call the actual WebGPU C API.

Needless to say, this was a cancer we felt we absolutely had to get rid of in the rewrite.

Comptime interfaces

While we want to get rid of runtime interfaces, maintain C ABI compatability, and be zero-overhead-we’d still like to be able to intercept WebGPU API calls if desired, so that we can provide superior debugging facilities in the future.

Zig’s std.mem.Allocator being a runtime interface makes sense because they have different use cases, no existing ABI to remain compatible with, and importantly there are cases where you would want to have multiple allocator implementations in the same program for different purposes.

With WebGPU, we have different constraints: it’s very unlikely to want multiple WebGPU implementations per program. We do need to maintain ABI compatibility. So to address this, we introduce a comptime interface.

Let’s look at the Texture.destroy method from earlier:

pub inline fn destroy(tex: Texture) void {
 tex.vtable.destroy(tex.ptr);
}

As you can see, this would’ve called the tex.vtable pointer, and passed the tex.ptr interface implementation pointer to it. It’s a classical runtime interface implementation. The key point here is that the data type can remain the same, while the implementation pointer could be replaced at runtime with a different one. On the other side of this invocation, tex.vtable.destroy would look like this:

pub fn destroy(ptr: *anyopaque) void {
 c.wgpuTextureDestroy(@ptrCast(c.WGPUTexture, ptr));
}

Now let’s look at how the comptime interface approach differs:

pub const Texture = opaque {
 pub inline fn destroy(texture: *Texture) void {
 Impl.textureDestroy(texture);
 }
 // ...
}

Firstly, we see that *gpu.Texture is merely an opaque pointer (a C void* if you like), just the same as before. Unlike before, however, there is no vtable pointer: there is only one pointer, it’s passed directly to the implementor via Impl.textureDestroy - and the implementation cannot be changed at runtime.

This solves the issue of ABI compatibility (we have only one pointer now), but we still need to let the user of the library - say from their main.zig file - decide which Implementation of the interface to use.

Traditionally, one might use generics for this (passing an Impl type parameter to each method for example), but we’d rather not pass that around everywhere: after all, we know it will be decided by one user of the API for the entire program, and requiring a type parameter here would have viral effects to every user of the WebGPU API (every API they expose would need that same type parameter.)

Luckily, in Zig there is a trick: from within our WebGPU API we can import the root file of the program (e.g. main.zig). Zig allows this since it lazily evaluates code, so there’s no dependency loop here. So in our mach/gpu package, we can define:

pub const Impl = blk: {
 const root = @import("root");
 if (!@hasDecl(root, "GPUInterface")) @compileError("expected to find `pub const GPUInterface = T;` in root file");
 _ = gpu.Interface(root.GPUInterface); // verify the type
 break :blk root.GPUInterface;
};

This effectively looks in the user’s main.zig (“root”) file for a declaration like:

pub const GPUInterface = gpu.dawn.Interface;

Once resolved, our Impl constant is known statically at compile time to be an exact interface implementation of the gpu.Interface: gpu.dawn.Interface in this case, which is just a struct type with functions in it calling the Dawn C API:

pub const Interface = struct {
 pub inline fn textureDestroy(texture: *gpu.Texture) void {
 procs.textureDestroy.?(@ptrCast(c.WGPUTexture, texture));
 }
 // ...
}

The trick here to ensuring that a type actually satisfies the gpu.Interface is that you write a type validator function, which checks if the struct passes to it has the desired methods with matching function signatures:

/// Verifies that a gpu.Interface implementation exposes the expected function declarations.
pub fn Interface(comptime T: type) type {
 assertDecl(T, "textureDestroy", fn (texture: *gpu.Texture) callconv(.Inline) void);
 // ...
 return T;
};

Best of all, since the interface implementation is completely static and known at comptime, we can enforce every method invocation is inline and we’re not adding any overhead.

libmach and gpu.Export

One recent development is libmach, which will provide at least a C ABI for the creation of Mach core applications from other languages (think a bit like SDL, but for WebGPU and it works on Desktop, Mobile, WebAssembly & more in the future.)

One thing we’d like to retain, though, is the ability to have such applications get the same nice WebGPU debugging experience in the future, while still using that language’s existing WebGPU bindings. This means instead of calling Dawn’s wgpuTextureDestroy for example, we’d need to call libmach’s wgpuTextureDestroy.

This is where gpu.Export comes in: it merely takes a gpu.Interface struct with all of the Zig functions that implement the WebGPU API, and exports the WebGPU C ABI for them:

/// Exports C ABI function declarations for the given gpu.Interface implementation.
pub fn Export(comptime T: type) type {
 _ = Interface(T); // verify implementation is a valid interface
 return struct {
 // WGPU_EXPORT void wgpuTextureDestroy(WGPUTexture texture);
 export fn wgpuTextureDestroy(texture: *gpu.Texture) void {
 T.textureDestroy(texture);
 }
 // ...
 };
}

From this, you might notice something important: We’ve maintained 100% C ABI compatability in the new mach/gpu rewrite. Every data structure is ABI compatible with Dawn’s webgpu.h header.

Zig flag sets

One nice property of Zig is it’s packed structs. For example, in C there is a WGPUColorWriteMaskFlags type which is a uint32_t where the first four bits represent a color write mask for red, green, blue, and alpha respectively. The remaining 28 bits are unused at present.

Interacting with WGPUColorWriteMaskFlags in C can be a bit cumbersome: you need to make sure you remember the right bit masking operations to set bits, check if they are set, and so on.

In Zig, we have packed struct in which bool is just one bit - and we have integers of any bit width we desire. We can use this to compose a 32-bit data structure compatible with the C ABI variant, but using nice bools to represent those first four bits:

pub const ColorWriteMaskFlags = packed struct {
 red: bool = false,
 green: bool = false,
 blue: bool = false,
 alpha: bool = false,

 _padding: u28 = 0,
};

This is nice because now one can simply check if (write_mask.red and write_mask.blue) for example, or simply pass it as a parameter to a function like ColorWriteMaskFlags{.red = true, .blue = true}.

Read more about how this works: “Packed structs in Zig make bit/flag sets trivial”

Dawn native extensions

One not-so-friendly aspect of webgpu.h (the C API for WebGPU) is that it allows for arbitrary extension of the API via so-called chaining. For example, let’s look at a descriptor struct used as the parameters to create a shader module from its text source code:

typedef struct WGPUShaderModuleDescriptor {
 WGPUChainedStruct const * nextInChain;
 char const * label; // nullable
} WGPUShaderModuleDescriptor;

Here you can obviously see there is a label for the shader module - but where does our shader source code go? It’s not clear. And what goes in that nextInChain field? It looks like this:

typedef struct WGPUChainedStruct {
 struct WGPUChainedStruct const * next;
 WGPUSType sType;
} WGPUChainedStruct;

Effectively, WebGPU implementations can take arbitrary data structures via this chaining process - as extensions to the WebGPU API for example - so long as the chained struct begins with these ABI-compatible fields.

For example-to construct a shader in Zig, you might write:

const next_in_chain = c.WGPUShaderModuleWGSLDescriptor{
 .chain = c.WGPUChainedStruct{
 .next = null, // nothing else to chain
 .sType = c.WGPUSType_ShaderModuleWGSLDescriptor, // so it knows what type we chained!
 },
 .source = my_shader_source_code_text,
};
const shader_module_descriptor = c.WGPUShaderModuleDescriptor{
 .nextInChain = @ptrCast(?*const c.WGPUChainedStruct, next_in_chain),
 .label = "my shader module",
};

That’s pretty nasty! Also take note of how nextInChain needs to be cast to the WGPUChainedStruct pointer type, only the sType field identifies it (the C type system can’t.)

More importantly: because nextInChain is an opaque type, you can’t really know what type of pointer is legal at all to give to the API in a nextInChain field. Oof!

Needless to say, we didn’t want to adopt this lack of type safety (and lack of documentation), so we worked with the Dawn developers at Google to add documentation about what structs are legal where, and then in Zig we used this information to replace next_in_chain fields with a union of pointers so it’s type safe (for all known structs) and self-documenting. Our example from before becomes just:

const shader_module_descriptor = gpu.ShaderModule.Descriptor{
 .next_in_chain = .{
 .wgsl_descriptor = &.{.source = my_shader_source_code_text},
 },
 .label = "my shader module",
};

It may not seem much more readable, but all of the type system info is there to protect you and that’s what counts. Of course, we also added a helper to create WGSL shader modules so this ends up being truly clean:

device.createShaderModuleWGSL("my shader module", my_shader_source_code_text);

Upstreamed patches to Dawn

Out of the box, Dawn needed a little love to be compiled with Zig as the C/C++ compiled - so we’ve contributed patches upstream for this:

Resolving some undefined behavior in Dawn caught by Zig using UBSAN by default. #87380
Improving constexpr compatibility for a DirectX constant, due to using MinGW DirectX headers. #87381
Correcting an invocation of _uuidof on Windows. #87309
Adding an option to disable use of (Windows 10+) Windows UI, as we don’t have headers for it. #87383

Obvious improvements

There were many other obvious improvements we won’t enumerate in detail here:

Achieving 100% API coverage, and coming up with processes/rules/conventions to ensure this all remains up-to-date and correct going forward as Dawn’s webgpu.h API changes.
Setting the right default values for every field in the entire API, which reduces verbosity of the API substantially.
Adding slice helpers where the C ABI uses pointers-and-lengths distinctly.
Adding type-safe helpers to callbacks which would have a void* userdata pointer in the C API.
Exposing every Dawn native extension, e.g. in anticipation of bindless support in the future.

Standalone repository

As with all standalone Mach libraries that reach a certain level of maturity, mach/gpu is now available in it’s own standalone repository with an example using it with GLFW: https://github.com/hexops/mach-gpu

What’s next: browser support, more examples

I’d say we’re well on our way to having a perfect WebGPU/Dawn API for Zig, but we do have a little ways to go. Things coming up include:

More examples
Adding browser support: this will be achieved in the near future by direct WebAssembly->JS calls (not via Emscripten.)
Adding higher-level helpers (always 100% optional, the C ABI is always available and present via gpu.Impl.foobar methods.)

We’re continuing to work towards the Mach v0.2 release otherwise (special thanks for all those contributing to Mach today!)

Thanks for reading

Join the Mach Discord server
Check out the mach/gpu example showcase
Help us port/write more WebGPU examples to Zig
Read up on WebGPU compute and rendering
Sponsor development if you like what we're doing!

Packed structs in Zig make bit/flag sets trivial

Mon, 29 Aug 2022 00:00:00 +0000

As we’ve been building Mach engine, we’ve been using a neat little pattern in Zig that enables writing flag sets more nicely in Zig than in other languages.

What is a flag set?

We’ve been rewriting mach/gpu (WebGPU bindings for Zig) from scratch recently, so let’s take a flag set from the WebGPU C API:

typedef uint32_t WGPUFlags;
typedef WGPUFlags WGPUColorWriteMaskFlags;

Effectively, WGPUColorWriteMaskFlags here is a 32-bit unsigned integer where you can set specific bits in it to represent whether or not to write certain colors:

typedef enum WGPUColorWriteMask {
 WGPUColorWriteMask_None = 0x00000000,
 WGPUColorWriteMask_Red = 0x00000001,
 WGPUColorWriteMask_Green = 0x00000002,
 WGPUColorWriteMask_Blue = 0x00000004,
 WGPUColorWriteMask_Alpha = 0x00000008,
 WGPUColorWriteMask_All = 0x0000000F,
 WGPUColorWriteMask_Force32 = 0x7FFFFFFF
} WGPUColorWriteMask;

Then to use it you’d use the various bit operations with those masks, e.g.:

WGPUColorWriteMaskFlags mask = WGPUColorWriteMask_Red | WGPUColorWriteMask_Green;
mask |= WGPUColorWriteMask_Blue; // set blue bit

This all works, people have been doing it for years in C, C++, Java, Rust, and more. In Zig, we can do better.

Zig packed structs

Zig has packed structs: these let us pack memory tightly, where a bool is actually a single bit (in most other languages, this is not true.) Zig also has arbitrary bit-width integers, like u28, u1 and so on.

We can write WGPUColorWriteMaskFlags from earlier in Zig using:

pub const ColorWriteMaskFlags = packed struct {
 red: bool = false,
 green: bool = false,
 blue: bool = false,
 alpha: bool = false,

 _padding: u28 = 0,
};

This is still just 32 bits of memory, and so can be passed to the same C APIs that expect a WGPUColorWriteMaskFlags - but interacting with it is much nicer:

var mask = ColorWriteMaskFlags{.red = true, .green = true};
mask.blue = true; // set blue bit

In C you would need to write code like this:

if (mask & WGPUColorWriteMask_Alpha) {
 // alpha is set..
}
if (mask & (WGPUColorWriteMask_Alpha|WGPUColorWriteMask_Blue)) {
 // alpha and blue are set..
}
if ((mask & WGPUColorWriteMask_Green) == 0) {
 // green not set
}

In Zig it’s just:

if (mask.alpha) {
 // alpha is set..
}
if (mask.alpha and mask.blue) {
 // alpha is set..
}
if (!mask.green) {
 // green not set
}

Comptime validation

Making sure that our ColorWriteMaskFlags ends up being the same size could be a bit tricky: what if we count the number of bool wrong? Or what if we accidently get the padding size wrong? Then it might not be the same size as a uint32 anymore.

Luckily, we can verify our expectations at comptime:

pub const ColorWriteMaskFlags = packed struct {
 red: bool = false,
 green: bool = false,
 blue: bool = false,
 alpha: bool = false,

 _padding: u28 = 0,

 comptime {
 std.debug.assert(@sizeOf(@This()) == @sizeOf(u32));
 std.debug.assert(@bitSizeOf(@This()) == @bitSizeOf(u32));
 }
}

The Zig compiler will take care of running the comptime code block here for us when building, and it will verify that the byte size of @This() (the type we’re inside of, the ColorWriteMaskFlags struct in this case) matches the @sizeOf(u32).

Similarly we could check the @bitSizeOf both types if we like.

Note that @sizeOf may include the size of padding for more complex types, while @bitSizeOf returns the number of bits it takes to store T in memory if the type were a field in a packed struct/union. For flag sets like this, it doesn’t matter and either will do. For more complex types, be sure to recall this.

Explicit backing integers for packed structs

It’s worth noting that in Zig 0.10 (shipping in Nov), the new self-hosted compiler has support for explicit backing integers for packed structs which will simplify this even further.

Instead of manually adding padding to make up 32 bits, one could simply write packed struct(u32):

pub const ColorWriteMaskFlags = packed struct(u32) {
 red: bool = false,
 green: bool = false,
 blue: bool = false,
 alpha: bool = false,
}

Thanks for reading

Be sure to join the new Mach engine Discord server where we’re building the future of Zig game development.

You can also sponsor my work if you like what I’m doing! :)

“But C has had bitfields since forever!”

Shortly after posting this article I was inundated with comments proclaiming “But C has had bitfields since forever!”

First, I’d like to say I was not aware of C bitfields at the time of writing - I simply had not ever come across usage of them. Secondly, I’d like to question: if C has bitfields, then why do seemingly all modern C APIs not use? Why do they all expose integer types instead?

And then I found the answer in the TC3 C specification:

As this user writes:

The in-memory representation of bit fields is implementation-defined. Therefore, if you’re calling into an external API that takes a uint32_t like in the example without an explicit remapping, you may or may not like the results.

In practice, everything you’re likely to come across will be little endian nowadays, and the ABI you’re using will most likely order your struct from top to bottom in memory, so they will look the same most of the time. However, it’s still technically not portable.

My intention behind this article wasn’t to say C is bad; but rather to say that I find Zig’s packed structs quite nice. I actually come from a background mostly in Go - which absolutely does not have bitfields, packed structs, or arbitrary bit-width integers. Having never come across them in C either, my claims against C bitfields today could be summarized as:

C’s bitfields are more implementation-defined than Zig’s.
C’s bitfields being so implementation-defined, tend not to be used in modern APIs - so the fact that Zig has come up with a variant which is used in practice in most APIs is very important.

In any case, I am not an expert in C bitfields! I just hate masking to check if bits are set, and the insane number of ways that exact same logic can be written - both correctly and incorrectly. We deserve nicer syntax to check if a bit field is set out of the box, Zig provides that and I am happier for it.

Please stop messaging me about how C has bitfields :)

Let's build an Entity Component System (part 2): databases

Sat, 28 May 2022 00:00:00 +0000

In this series we build the Mach engine Entity Component System from scratch in the Zig programming language.

In part one, we looked at how ECS intersects with data oriented design, starting without any foundational understanding of how ECS typically works and instead working from first-principles to arrive at what would probably be the most computationally efficient implementation.

In this ~24 page part two, we examine functionality gaps our first approach had, explore how databases relate to ECS, and begin writing our actual implementation in Zig! By the end, you'll have an archetypal ECS with the ability to add/remove entities and components. In part 3, we'll cover queries.

Check out the prior parts of this series if you haven't already!

The case for a general-purpose runtime ECS

In part 1 we proposed an architecture which would have had you end up with something like this where you declare an entity archetype at comptime, as a struct type:

const Player = struct {
 name: []const u8, // a string / byte slice
 location: Vec3,
 velocity: Vec3,
 health: u8,
 team: Team,
};

In this code, our struct fields name, location, etc. are said to be our entity components for the Player archetype. To create a Player entity, we would simply create a value of this type. We proposed using std.MultiArrayList to store lists of Player entities for efficient CPU cache utilization:

var players: MultiArrayList(Player) = .{}; // all players

This approach is minimal, simplistic, and has an extremely efficient memory layout. Anyone who has used a production-worthy ECS, though, will tell you: that lacks flexibility.

Here are a few reasons why a general-purpose runtime ECS is more flexible.

Operating on components, not entities

Imagine you’d like to have your physics system operate on every entity with velocity and location components. We know that as long as we have those two values, we can do some maths and update the location of an entity to where it should be. But wait, which entities?

var players: MultiArrayList(Player) = .{}; // all players
var monsters: MultiArrayList(Monster) = .{}; // all monsters
var cameras: MultiArrayList(Camera) = .{}; // all cameras
var lights: MultiArrayList(Light) = .{}; // all lights

We could look at these types as a programmer and find which ones have velocity and location fields, probably just players and monsters do. But now, how do we write our physics code to operate on both lists of Player and Monster entities? We just need the velocity and location fields - we don’t care if it’s a player or monster!

Rapid, iterative, game design

If we wish to add a weapon: Weapon component to our Player entity, all good: we update the Player struct to have that field. If we want to add a weapon at runtime, we make it an optional weapon: ?Weapon so it can be null.

Let's say we're working on a whacky new cooking simulator game: you've got a kitchen stove, ingredients, utensils, etc. as entities. Some code checks if an entity touches the stovetop and, if it has a cookable: void component, then it gets cooked. If we're trying to build a 100% science-based cooking simulator, well, then we could probably plan ahead and "know" that Ingredient entities should have the cookable component while Utensil entities should not. But often, there's immense joy in strange mechanics: What if utensils were cookable?!

Maybe even a game server has made this decision, or a scripting language. We didn't anticipate this at compile time! It'd be great if we could quickly try it out at the flip of a switch, though, while the game is running. And especially without having to track down every codepath handling cookable ingredients to now handle cookable utensils!

Runtime components? Just as fast

We want runtime components in Mach engine for the reasons above, all of which boil down to rapid, iterative game design. Integration of our ECS with a GUI level editor, etc. all require deep levels of runtime introspection of the data in our ECS.

One may assume that runtime ECS will just naturally be slower than a comptime ECS.

It’s important to note that just because we’re defining components at runtime, it doesn’t mean we cannot take special care to follow data oriented design and structure our memory in a way that is very efficient for CPU cache.

Thinking in terms of databases

With more complex aspects of an ECS, there are just tradeoffs, tradeoffs, tradeoffs everywhere!

Querying
- “find all entities that have Physics and Location components”
- “find all entities within 5 units distance from (x, y, z)”
- “find me all player entities whose Name component starts with ‘ziggy’”
- …
Indexing queries (how to make complex queries fast?)
Dense vs. sparse storage
- “almost every player has a Weapon component”
- “only a few players have a Weapon component, most don’t”

Could there ever be a perfect way to represent ECS data in memory to handle all possible ways someone might want to use it? You might see this as a drawback - there cannot be a perfect ECS! “Maybe that means you shouldn’t use one at all” you might think

But, if we begin to think about an ECS as nothing more than an in-memory database for game entities, it’s incredibly tempting to draw analogies with traditional databases:

Pushing the database analogy further

Multi-threaded queries / writes

A physics system which wishes to calculate physics for any entity with location and velocity components (columns on any table) ideally can run in parallel with other systems which wish to query and mutate entities.

Such a physics system could interact with the ECS through a “database connection” or “database handle” which synchronizes access (say through table locks, column locks, row locks, etc.) to ensure conflict-free parallel execution with other systems.

Additionally, finding entities with location and velocity components is as simple as asking: which tables have those columns? Every entity in such a table is guaranteed to have those components, we don’t need to check each entity to see if it has those components.

Indexing queries

The natural row-by-row order of database tables is great, but we could have indexes to optimize specific query usage patterns without fundamentally changing our architecture:

Spatial index: Maybe there are 1 million entities spread across a huge area, we only want to find those within 10 meters from our player. A spatial index could utilize an octree to optimize such queries.
Graph relation index: If you anticipate walking up/down a graph of entities (think a GUI / scene graph) a lot, then it’s important to have a fast way to lookup a given entity’s parent/children - a graph relation index could efficiently keep track of such relations.
Generic probability index: Sometimes you’ll need to “find all entities where component X has a value Y”, a probability index could maintain fastfilters to statistically answer “these entities likely have value Y (though a few might not)” extremely quickly.
Generic function index: An escape hatch - maybe you want to find all entities where arbitraryFunction(entity) returns true, a generic index could keep track of when entities (rows) are changed and only invoke arbitraryFunction when changes occur.

Other ECS implementations

After thinking about this anology quite far, writing our implementation around it, etc. I was quite happy to find that I wasn’t the only one who thought of this: The Rust Bevy authors also describe ECS as a data structure in this way and after writing my implementation I got in touch with them to discuss tradeoffs, get advice, etc. (many thanks!)

While this is a helpful analogy to have in the back of your head, we won’t take it too far - it’s not completely perfect. For example, the database equivalent of ‘sparse storage’ might be “every row of our “players” table has a foreign key (the row ID of another table with less rows)”, but in reality we wouldn’t want our ECS sparse storage to pay the cost of storing that ID for every table row: only rows of entities where we want such a component value. Instead, sparse storage in an ECS is more like a mapping of row ID -> component_value.

Writing our ECS implementation in Zig

At this point we’ve made a large amount of the architecture decisions for our ECS: we understand how ECS relates to data oriented design, databases, and the tradeoffs we’ll make with our implementation. From this point on, this series will be much more code-heavy!

Representing a “world” of entities

The first thing we need is a way to represent our “database of tables”, the tables that will contain our entities, usage may look something like:

var world = Entities.init(allocator); // create a world
defer world.deinit(); // free the world

We can define this as a struct:

pub const Entities = struct {
 allocator: Allocator,

 pub fn init(allocator: Allocator) Entities {
 return .{
 .allocator = allocator,
 };
 }

 pub fn deinit(entities: *Entities) void {
 _ = entities;
 // TODO: release anything we allocate
 }
};

So if Entities is our “database”, we need a way to represent our “tables” (or “archetypes”) that will store our actual entities component data:

 pub const Entities = struct {
 allocator: Allocator,

+ /// A mapping of archetype hash to their storage.
+ ///
+ /// Database equivalent: table name -> tables representing entities.
+ archetypes: std.AutoArrayHashMapUnmanaged(u64, ArchetypeStorage) = .{},

 pub fn init(allocator: Allocator) Entities {
 return .{
 .allocator = allocator,
 };
 }

 pub fn deinit(entities: *Entities) void {
- _ = entities;
- // TODO: release anything we allocate
+ var iter = entities.archetypes.iterator();
+ while (iter.next()) |entry| {
+ entry.value_ptr.deinit();
+ }
+ entities.archetypes.deinit(entities.allocator);
 }
 };

The Entities.archetypes field is a hashmap of archetype hashes (more on these later) to the archetype storage, where our actual entities component values will be stored.

std.AutoArrayHashMapUnmanaged(u64, ArchetypeStorage) is a bit of a mouth full! If you’re keen to understand more about Zig hashmaps, I wrote a quick article explaining Zig hashmaps you should check out, but all you need to know is this: it’s a hashmap of u64 keys to ArchetypeStorage struct values.

The deinit function we’ve added just iterates over each value in the hashmap and calls deinit on it so it has a chance to free it’s allocated memory before we free the entire hashmap.

`ArrayHashMap` as an alternative to sparse sets

Importantly, we use an ArrayHashMap here not a regular hash map: an ArrayHashMap is actually just backed by an ordered array behind the scenes, and because of this it’s optimized for iteration over the hashmap values rather than hashmap lookups, since consecutive values are very likely to be in CPU cache.

Critically, we can directly index into the ordered backing array: if we know the index of a table we’d like to lookup, that’s a simple O(1) index operation and not a hashmap lookup - we’ll take great advantage of this later as an alternative to ‘sparse sets’ you may read about in other ECS implementations.

Why our archetype table names are hashes: entities move between tables

You may have noticed we use u64 values to name our archetype storage tables: why not strings? In a traditional database, these would be strings:

In our ECS, though, there’s a trick: tables will not be user-defined, they’ll be automatically created and destroyed as needed for you. We’ll just put entities into the table that has all the needed columns, and so our players and monsters tables above would actually just be one big table (since they have identical components) with a name like has_sword__and__health__and__location:

Let’s say we want to give a player entity a new component, like a rotation, then we’ll just create a new table with has_sword, health, location, rotation columns and move just that one entity over to the new table.

And so the names for our archetype tables are actually just a hash of all the component names/types! This means that when we add that rotation component to another player entity, we can merely hash all the component names the entity will now have to quickly check: does a table for storing this archetype of entity already exist, or do we need to create a new one?

Creating our first archetype table

When we first create an entity, it’s not going to have any components. We need a way to represent entities that do not have any components - for this, we’ll create a special “void archetype”, an empty table where entities will start out:

+pub const void_archetype_hash = std.math.maxInt(u64);

 pub const Entities = struct {
 ...

- pub fn init(allocator: Allocator) !Entities {
- return .{
- .allocator = allocator,
- };
+ pub fn init(allocator: Allocator) !Entities {
+ var entities = Entities{ .allocator = allocator };
+
+ try entities.archetypes.put(allocator, void_archetype_hash, ArchetypeStorage{
+ .allocator = allocator,
+ .components = .{},
+ .hash = void_archetype_hash,
+ });
+
+ return entities;
 }
 };

This puts a single item into our entities.archetypes hashmap: void_archetype_hash as the key, which will be a special key for entities without any components, and the value is a new ArchetypeStorage{...} table, which looks like this:

pub const ArchetypeStorage = struct {
 allocator: Allocator,

 /// The hash of every component name in this archetype, i.e. the name of this archetype.
 hash: u64,

 /// A string hashmap of component_name -> type-erased *ComponentStorage(Component)
 components: std.StringArrayHashMapUnmanaged(ErasedComponentStorage),

 pub fn deinit(storage: *ArchetypeStorage) void {
 for (storage.components.values()) |erased| {
 erased.deinit(erased.ptr, storage.allocator);
 }
 storage.components.deinit(storage.allocator);
 }
};

ArchetypeStorage is representing all entities that have the exact same set of component types (think back to our “monsters and players go in one table” diagram above.) The database equivalent of ArchetypeStorage is a table, where rows are entities and columns are components (dense storage.)

It’s aware of it’s own hash (table name), and maintains it’s own hashmap components which maps component names (strings) to the actual place in memory where we store the components’ values. Here, this is ErasedComponentStorage, a type-erased pointer to *ComponentStorage(Component), which brings us to..

Storing components in memory

Within our ArchetypeStorage database tables, we need to actually store the values for components somewhere. And how we represent these in memory is critical. You may recall from Andrew Kelley’s “A Practical Guide to Applying Data-Oriented Design” talk that if we use std.MultiArrayList it would store our data in a way that is more efficient for CPU cache, leading to much greater performance. We get to take advantage of that here by storing component values as a struct-of-arrays instead of array-of-structs which, as the talk describes, helps to reduce the in-memory size of our data and ensure more of them are in CPU cache.

A component will be a relatively simple, small value - such as:

const Location = struct {
 x: f32 = 0,
 y: f32 = 0,
 z: f32 = 0,
};

On an entity, there may be many of these. Thanks to the database model we have and tables being laid out, we get to store all Location component values in contiguous memory which is great for CPU caches:

/// Represents the storage for a single type of component within a single type of entity.
///
/// Database equivalent: a column within a table.
pub fn ComponentStorage(comptime Component: type) type {
 return struct {
 /// A reference to the total number of entities with the same type as is being stored here.
 total_rows: *usize,

 /// The actual densely stored component data.
 data: std.ArrayListUnmanaged(Component) = .{},

 const Self = @This();

 pub fn deinit(storage: *Self, allocator: Allocator) void {
 storage.data.deinit(allocator);
 }
 };
}

So when we have a table of entities that have a Location component, our ArchetypeStorage.components hashmap will have a "location" entry for example that points to a *ComponentStorage(Location) densely storing all location values for every entity in the entire table.

Type-erased component storage

You might recall our components hashmap in our table is ErasedComponentStorage, not *ComponentStorage(Component):

pub const ArchetypeStorage = struct {
 ...

 components: std.StringArrayHashMapUnmanaged(ErasedComponentStorage),
};

What gives? Well, the problem is that we need to store multiple component types in this hashmap. For example, if this table represents player entities we may need two entries:

"weapon" -> *ComponentStorage(Weapon)
"location" -> *ComponentStorage(Location)

Here Weapon and Location are generic type parameters. Our ArchetypeStorage.components hashmap can only point to one type of value, though! So we must first turn our *ComponentStorage(Weapon) into a type-erased pointer *anyopaque (equal to C’s void*). Of course, this can make working with the values quite difficult because then we don’t know what data type they were supposed to have! To aid with this, we introduce a ErasedComponentStorage type:

/// A type-erased representation of ComponentStorage(T) (where T is unknown).
pub const ErasedComponentStorage = struct {
 ptr: *anyopaque,

 // Casts this `ErasedComponentStorage` into `*ComponentStorage(Component)` with the given type
 // (unsafe).
 pub fn cast(ptr: *anyopaque, comptime Component: type) *ComponentStorage(Component) {
 var aligned = @alignCast(@alignOf(*ComponentStorage(Component)), ptr);
 return @ptrCast(*ComponentStorage(Component), aligned);
 }
};

This is useful as it allows us to store all of the typed ComponentStorage(T) as values in a hashmap despite having different T types, and allows us to still interact with them in consistent ways even though we don’t remember what the underlying type is. For example, add the requirement that ErasedComponentStorage knows how to deinitialize itself:

 pub const ErasedComponentStorage = struct {
 ptr: *anyopaque,
+ deinit: fn (erased: *anyopaque, allocator: Allocator) void,

 ...
 };

When we go to actually create an ErasedComponentStorage value, we know the type, and so we can set up a function that does the deinitialization for us:

pub const Entities = struct {
 ...

 pub fn initErasedStorage(entities: *const Entities, total_rows: *usize, comptime Component: type) !ErasedComponentStorage {
 var new_ptr = try entities.allocator.create(ComponentStorage(Component));
 new_ptr.* = ComponentStorage(Component){ .total_rows = total_rows };

 return ErasedComponentStorage{
 .ptr = new_ptr,
 .deinit = (struct {
 pub fn deinit(erased: *anyopaque, allocator: Allocator) void {
 var ptr = ErasedComponentStorage.cast(erased, Component);
 ptr.deinit(allocator);
 allocator.destroy(ptr);
 }
 }).deinit,
 };
 }

 ...

Here initErasedStorage is a way for us to say:

Hey! I anticipate storing total_rows of Component values in a table, please allocate that for me and give me a ErasedComponentStorage value in return.

The first two lines create a pointer where we can store our ComponentStorage(Component) struct value itself (not the items inside of it), and initialize new_ptr.* with a value:

var new_ptr = try entities.allocator.create(ComponentStorage(Component));
new_ptr.* = ComponentStorage(Component){ .total_rows = total_rows };

Then we create the ErasedComponentStorage value, giving it the pointer *ComponentStorage(Component) (and erasing that type in the process) .ptr = new_ptr,, and create our deinit helper:

.deinit = (struct {
 pub fn deinit(erased: *anyopaque, allocator: Allocator) void {
 var ptr = ErasedComponentStorage.cast(erased, Component);
 ptr.deinit(allocator);
 allocator.destroy(ptr);
 }
}).deinit,

.deinit = is setting the deinit field of ErasedComponentStorage to a value. The (struct { ... }).deinit, part is just creating an anonymous struct with a deinit function in it so we can pick it back out, this is some syntactual cruft Zig currently requires for writing function expressions.

You’ll see that what the function does is pretty simple, though: It takes that erased: *anyopaque pointer, casts it back to the typed value *ComponentStorage(Component) (since within this function we know what the Component type is) and then calls ptr.deinit(allocator) which is just a standard method on the ComponentStorage struct so it has a chance to free any memory it allocated, before ultimately we ask the allocator allocator.destroy(ptr) to destroy the pointer where we’re storing that struct value ComponentStorage(Component) we allocated earlier.

Now if we have an *ErasedComponentStorage value, we can call it’s .deinit function and it knows how to cast back to the appropriate pointer type before freeing everything. We’ll reuse this pattern to do other generic operations on component storage later.

Managing entity IDs / pointers

At this point we’ve got:

Entities (our database)
ArchetypeStorage (a table)
ErasedComponentStorage and ComponentStorage - the columns in a table

It’s time to actually represent entities! We’ll do so with just an ID:

/// An entity ID uniquely identifies an entity globally within an Entities set.
pub const EntityID = u64;

We need a way to know which table an entity is stored in, and which row in that table it’s component values are located at. You may be tempted to think that EntityID could just be that information, but remember than when we add or remove a component from an entity it will move between ArchetypeStorage tables! When that happens, it’s nice if other user code referencing that EntityID can stay oblivious to that - so we’ll store a mapping of entity IDs to Pointer values:

 pub const Entities = struct {
 ...
+ counter: EntityID = 0,
+
+ /// A mapping of entity IDs (array indices) to where an entity's component values are actually
+ /// stored.
+ entities: std.AutoHashMapUnmanaged(EntityID, Pointer) = .{},

+ /// Points to where an entity is stored, specifically in which archetype table and in which row
+ /// of that table.
+ pub const Pointer = struct {
+ archetype_index: u16,
+ row_index: u32,
+ };

 pub fn deinit(entities: *Entities) void {
+ entities.entities.deinit(entities.allocator);
 ...
 }
 ...
 };

Remember how I said earlier it was important that the mapping of table names -> tables (Entities.archetypes) is an array hash map, not a regular hash map? That’s because we can index directly into it! Say we’re given an arbitrary EntityID and want to find it’s component values, first we would find out which table/row the entity points to via a hash map lookup:

const ptr: Pointer = entities.entities.get(entity_id).?;

Now we know exactly which table and row it’s stored in, and can lookup the table, or component values, with simple O(1) array access operations. e.g. to get the archetype table the entity is stored in:

var archetype = entities.archetypes.entries.get(ptr.archetype_index);

Here, entities.archetypes.entries is our AutoArrayHashMapUnmanaged mapping table names to their ArchetypeStorage - but we access the array inside the hash map directly instead of using a hash map lookup.

Creating an entity

To create new entities, we’ll use an Entities.new method that returns a new entity ID by incrementing a global counter in our database:

 pub const Entities = struct {
 ...

+ /// Returns a new entity.
+ pub fn new(entities: *Entities) !EntityID {
+ const new_id = entities.counter;
+ entities.counter += 1;
+ return new_id;
+ }
 };

Initially, an entity will have no components, and thus we’ll put it into that special “void” archetype mentioned earlier (this just gives us a guarantee that an entity is always residing in an archetype, even if it has no components - a property that will come in handy later):

 /// Returns a new entity.
 pub fn new(entities: *Entities) !EntityID {
 const new_id = entities.counter;
 entities.counter += 1;

+ var void_archetype = entities.archetypes.getPtr(void_archetype_hash).?;
+ const new_row = try void_archetype.new(new_id);

 return new_id;
 }

void_archetype here is, of course, ArchetypeStorage (a database table). We’re invoking void_archetype.new(new_id) to reserve a row in the void archetype table, which will be done using this:

 pub const ArchetypeStorage = struct {
 ...
+ entity_ids: std.ArrayListUnmanaged(EntityID) = .{},

 pub fn deinit(storage: *ArchetypeStorage) void {
 for (storage.components.values()) |erased| {
 erased.deinit(erased.ptr, storage.allocator);
 }
+ storage.entity_ids.deinit(storage.allocator);
 storage.components.deinit(storage.allocator);
 }

+ pub fn new(storage: *ArchetypeStorage, entity: EntityID) !u32 {
+ // Return a new row index
+ const new_row_index = storage.entity_ids.items.len;
+ try storage.entity_ids.append(storage.allocator, entity);
+ return @intCast(u32, new_row_index);
+ }
 };

Now ArchetypeStorage maintains a mapping of rows in the table (entity_ids indices) to the entity ID. This will come in handy later - the only important thing to note here is that this is reserving a new row in the table where the entity can live, but it’s not actually allocating the storage for that entity’s component values yet.

Back over to Entities (the database), we need to record which archetype (table), and row number in that table, that the entity ID actually points to (remember-entity IDs are merely pointers to a specific table and row):

 /// Returns a new entity.
 pub fn new(entities: *Entities) !EntityID {
 const new_id = entities.counter;
 entities.counter += 1;

 var void_archetype = entities.archetypes.getPtr(void_archetype_hash).?;
 const new_row = try void_archetype.new(new_id);
+ const void_pointer = Pointer{
+ .archetype_index = 0, // void archetype is guaranteed to be first index
+ .row_index = new_row,
+ };
+
+ entities.entities.put(entities.allocator, new_id, void_pointer) catch |err| {
+ void_archetype.undoNew();
+ return err;
+ };

 return new_id;
 }

Remember how void_archetype.new from earlier reserves a row in the table? Well, what happens if we reserved that row, but then fail to record the pointer (entities.entities.put OOMs)? In this case, our table has reserved a row for the entity to-be, but we don’t have enough memory to record which table/row the entity ID points to. So we need to undo that reservation to ensure our table doesn’t have an unused (reserved) row. undoNew does exactly that:

 pub const ArchetypeStorage = struct {
 ...

 pub fn new(storage: *ArchetypeStorage, entity: EntityID) !u32 {
 // Return a new row index
 const new_row_index = storage.entity_ids.items.len;
 try storage.entity_ids.append(storage.allocator, entity);
 return @intCast(u32, new_row_index);
 }

+ pub fn undoNew(storage: *ArchetypeStorage) void {
+ _ = storage.entity_ids.pop();
+ }
 };

Since the call to new merely appended a value to entity_ids, we only need to pop the last value off in order to undo the call to new and effectively unreserve the row we last reserved.

At this point, the following test works:

test "ecs" {
 const allocator = testing.allocator;

 var world = try Entities.init(allocator);
 defer world.deinit();

 const player = try world.new();
 _ = player;
}

(A copy of the full code at this point is available here and you can run it using zig test ecs.zig)

Working with component storage

As we work with component storage (table columns, where component values for a single type T are stored contiguously in memory) we’re going to need some helper functions. The first one is to swap remove a value from a column:

 pub fn ComponentStorage(comptime Component: type) type {
 return struct {
 ...
 data: std.ArrayListUnmanaged(Component) = .{},

+ pub fn remove(storage: *Self, row_index: u32) void {
+ if (storage.data.items.len > row_index) {
+ _ = storage.data.swapRemove(row_index);
+ }
+ }

Note that ComponentStorage memory is lazily allocated, so we only remove if the table column does in fact have storage allocated.

Next up is a simple copy function, a specific row’s value from src to dst:

+ pub inline fn copy(dst: *Self, allocator: Allocator, src_row: u32, dst_row: u32, src: *Self) !void {
+ try dst.set(allocator, dst_row, src.get(src_row));
+ }

And a helper to get the actual component value from a column:

+ pub inline fn get(storage: Self, row_index: u32) Component {
+ return storage.data.items[row_index];
+ }

Here we do not need to check the length of storage.data.items, because we assert the column must have that row.

Working with type-ErasedComponentStorage

As we discussed earlier, we often won’t have a typed ComponentStorage(T) and instead will have ErasedComponentStorage. We need a few more helpers to operate on the columns of a table, this time without knowing the underlying data type.

Cloning ComponentStorage types

The first helper we need is the ability to create a new value of type ComponentStorage(T) when we don’t know the actual type of T:

 /// A type-erased representation of ComponentStorage(T) (where T is unknown).
 pub const ErasedComponentStorage = struct {
 ptr: *anyopaque,
 ...
+ cloneType: fn (erased: ErasedComponentStorage, total_entities: *usize, allocator: Allocator, retval: *ErasedComponentStorage) error{OutOfMemory}!void,

The goal of this helper is purely to clone the type, let’s see how it is implemented when when initializing erased component storage:

 pub const Entities = struct {
 ...

 pub fn initErasedStorage(entities: *const Entities, total_rows: *usize, comptime Component: type) !ErasedComponentStorage {
 ...

 return ErasedComponentStorage{
 ...
+ .cloneType = (struct {
+ pub fn cloneType(erased: ErasedComponentStorage, _total_rows: *usize, allocator: Allocator, retval: *ErasedComponentStorage) !void {
+ var new_clone = try allocator.create(ComponentStorage(Component));
+ new_clone.* = ComponentStorage(Component){ .total_rows = _total_rows };
+ var tmp = erased;
+ tmp.ptr = new_clone;
+ retval.* = tmp;
+ }
+ }).cloneType,
 };
 }

Notably this doesn’t copy the actual values in the component storage, it’s just creating a new ComponentStorage(T) for us - just the struct value, as if we’d written ComponentStorage(T){.total_rows = total_rows}! This doesn’t allocate storage for the rows, it just says we anticipate there will be that many.

Copying component values between tables

The second helper we’ll need is a way to copy a component value from one ComponentStorage(T) column row to another of the same type T, again when we don’t know the underlying type and just have two ErasedComponentStorage we need to copy a value between:

 /// A type-erased representation of ComponentStorage(T) (where T is unknown).
 pub const ErasedComponentStorage = struct {
 ptr: *anyopaque,
 ...
+ copy: fn (dst_erased: *anyopaque, allocator: Allocator, src_row: u32, dst_row: u32, src_erased: *anyopaque) error{OutOfMemory}!void,

The implementation is simple:

 pub const Entities = struct {
 ...

 pub fn initErasedStorage(entities: *const Entities, total_rows: *usize, comptime Component: type) !ErasedComponentStorage {
 ...
 return ErasedComponentStorage{
 ...
+ .copy = (struct {
+ pub fn copy(dst_erased: *anyopaque, allocator: Allocator, src_row: u32, dst_row: u32, src_erased: *anyopaque) !void {
+ var dst = ErasedComponentStorage.cast(dst_erased, Component);
+ var src = ErasedComponentStorage.cast(src_erased, Component);
+ return dst.copy(allocator, src_row, dst_row, src);
+ }
+ }).copy,

Removing a row from a column

Removing a single component value from a column in a table looks as you’d expect:

 /// A type-erased representation of ComponentStorage(T) (where T is unknown).
 pub const ErasedComponentStorage = struct {
 ptr: *anyopaque,
 ...
+ remove: fn (erased: *anyopaque, row: u32) void,

The implementation is simple:

 pub const Entities = struct {
 ...

 pub fn initErasedStorage(entities: *const Entities, total_rows: *usize, comptime Component: type) !ErasedComponentStorage {
 ...
+ return ErasedComponentStorage{
+ ...
+ .copy = (struct {
+ pub fn copy(dst_erased: *anyopaque, allocator: Allocator, src_row: u32, dst_row: u32, src_erased: *anyopaque) !void {
+ var dst = ErasedComponentStorage.cast(dst_erased, Component);
+ var src = ErasedComponentStorage.cast(src_erased, Component);
+ return dst.copy(allocator, src_row, dst_row, src);
+ }
+ }).copy,

 pub const Entities = struct {
 ...

 pub fn initErasedStorage(entities: *const Entities, total_rows: *usize, comptime Component: type) !ErasedComponentStorage {
 ...
 return ErasedComponentStorage{
 ...
+ .remove = (struct {
+ pub fn remove(erased: *anyopaque, row: u32) void {
+ var ptr = ErasedComponentStorage.cast(erased, Component);
+ ptr.remove(row);
+ }
+ }).remove,

Removing an entire row from a table

When we want to remove an entire row (all column values), we need to invoke remove on each column:

 pub const ArchetypeStorage = struct {
 ...
+ pub fn remove(storage: *ArchetypeStorage, row_index: u32) !void {
+ _ = storage.entity_ids.swapRemove(row_index);
+ for (storage.components.values()) |component_storage| {
+ component_storage.remove(component_storage.ptr, row_index);
+ }
 }

This also swap removes the row from the table entity_ids mapping of row indices -> entity ID.

Adding components

Now on to the fun part: adding components to our entity. Suppose we want to do this:

 test "ecs" {
 ...

+ const Location = struct {
+ x: f32 = 0,
+ y: f32 = 0,
+ z: f32 = 0,
+ };

+ try world.setComponent(player, "Name", "jane"); // add Name component
+ try world.setComponent(player, "Location", Location{}); // add Location component
+ try world.setComponent(player, "Name", "joe"); // update Name component
 }

When we call setComponent, we may be updating the value of an existing component OR adding a new component to the entity. If the latter, we need to move the entity to a new archetype table (which may or may not exist!) - so the first thing we need to do is figure out: what archetype table is this entity currently in, and where does it need to be?

 pub const Entities = struct {
 ...

+ pub inline fn archetypeByID(entities: *Entities, entity: EntityID) *ArchetypeStorage {
+ const ptr = entities.entities.get(entity).?;
+ return &entities.archetypes.values()[ptr.archetype_index];
+ }

+ pub fn setComponent(entities: *Entities, entity: EntityID, name: []const u8, component: anytype) !void {
+ var archetype = entities.archetypeByID(entity);

+ const old_hash = archetype.hash;

+ var have_already = archetype.components.contains(name);
+ const new_hash = if (have_already) old_hash else old_hash ^ std.hash_map.hashString(name);
+ };

archetypeByID takes an entity ID and gives us an actual memory pointer to the *ArchetypeStorage table where the entity is stored. It does this by looking up the entity ID (entities.entities.get(entity).?) so we know which archetype_index it is stored in, and then simply returns a pointer to that table.

Then setComponent first finds out which archetype table the entity is stored in now, and determines if that table already has the component we’re adding/updating the entity with. Recall how earlier we mentioned why our archetype table names are hashes - the hash here is simply a hash of every component name stored by the archetype table.

Creating new archetype tables

As noted earlier, when adding a new component to an entity (say going from components (Location, Name) -> (Location, Name, Weapon)) it will move from the old table to the new one. If the new one doesn’t exist yet we need to create it. We know this because new_hash is the name of the table, encompassing all the component types it stores:

 pub const Entities = struct {
 ...
 pub fn setComponent(entities: *Entities, entity: EntityID, name: []const u8, component: anytype) !void {
 ...

+ var archetype_entry = try entities.archetypes.getOrPut(entities.allocator, new_hash);
+ if (!archetype_entry.found_existing) {
+ archetype_entry.value_ptr.* = ArchetypeStorage{
+ .allocator = entities.allocator,
+ .components = .{},
+ .hash = 0,
+ };
+ var new_archetype = archetype_entry.value_ptr;

Merely creating a new ArchetypeStorage table is not enough, though - we need to create storage columns in the table to store all of the existing components found on the entity (Location, Name) - by iterating the components in the old table, which we don’t know the actual type of (they’re ErasedComponentStorage, not ComponentStorage(Location)), so we use a cloneType helper (which we’ll define later):

+ var column_iter = archetype.components.iterator();
+ while (column_iter.next()) |entry| {
+ var erased: ErasedComponentStorage = undefined;
+ entry.value_ptr.cloneType(entry.value_ptr.*, &new_archetype.entity_ids.items.len, entities.allocator, &erased) catch ...;
+ new_archetype.components.put(entities.allocator, entry.key_ptr.*, erased) catch ...;
+ }

And finally, create storage / a new column for the new component we’re adding to the entity (Weapon):

+ // Create storage/column for the new component.
+ const erased = entities.initErasedStorage(&new_archetype.entity_ids.items.len, @TypeOf(component)) catch ...;
+ new_archetype.components.put(entities.allocator, name, erased) catch ...;
+
+ new_archetype.calculateHash();
+ }
+ }
...

You may have noticed we wrote catch ... in the snippets above, these are simply written as:

catch |err| {
 assert(entities.archetypes.swapRemove(new_hash));
 return err;
}

The reason for this is simple: If we fail to clone the storage columns, or add the new storage column, then we failed to create the archetype storage table! In this case, we need to clean up after ourselves so as to not leave the database in a bad state - by removing the entry we added to entities.archetypes earlier.

Finally, we implement ArchetypeStorage.calculateHash:

 pub const ArchetypeStorage = struct {
 ...
 hash: u64,

+ pub fn calculateHash(storage: *ArchetypeStorage) void {
+ storage.hash = 0;
+ var iter = storage.components.iterator();
+ while (iter.next()) |entry| {
+ const component_name = entry.key_ptr.*;
+ storage.hash ^= std.hash_map.hashString(component_name);
+ }
+ }

This simply walks over each storage.components entry (the columns in the table), and hashes the component type names.

Making `Entities.setComponent` update component values

At this point our setComponent method finds the archetype_entry table that needs to be updated, and has created it if necessary:

 pub const Entities = struct {
 ...
 pub fn setComponent(entities: *Entities, entity: EntityID, name: []const u8, component: anytype) !void {
 ...
 var archetype_entry = try entities.archetypes.getOrPut(entities.allocator, new_hash);
 if (!archetype_entry.found_existing) {
 // ... creates new archetype table
 }

Now it’s time to actually update the table, putting our component values into it:

+ var current_archetype_storage = archetype_entry.value_ptr;
+
+ if (new_hash == old_hash) {
+ const ptr = entities.entities.get(entity).?;
+ try current_archetype_storage.set(ptr.row_index, name, component);
+ return;
+ }

Here, current_archetype_storage is going to be either the new storage table (if the entity moved from an old table to a new table), or the prior storage table (if we’re just updating the value of a component that was already on the entity.) We then compare new_hash == old_hash and, if equal, that implies we’re just updating the value of the existing component on the entity.

Now, if we’re moving the entity to a new table, things are a bit more involved. First, we need to copy all component values for our entity from the old archetype storage table to the destination storage table. We do this by creating a new row in the destination table, iterating each component value in the old row, and copying it over:

+ const new_row = try current_archetype_storage.new(entity);
+ const old_ptr = entities.entities.get(entity).?;
+
+ var column_iter = archetype.components.iterator();
+ while (column_iter.next()) |entry| {
+ var old_component_storage = entry.value_ptr;
+ var new_component_storage = current_archetype_storage.components.get(entry.key_ptr.*).?;
+ new_component_storage.copy(new_component_storage.ptr, entities.allocator, new_row, old_ptr.row_index, old_component_storage.ptr) catch |err| {
+ current_archetype_storage.undoNew();
+ return err;
+ };
+ }

We also need to update the table’s mapping of entity_ids (row indices -> entity ID):

+ current_archetype_storage.entity_ids.items[new_row] = entity;

And since we only copied over the old components -> the new table row, we don’t yet have the new component in the new table row - it’s undefined memory at present. We update it:

+ current_archetype_storage.set(new_row, name, component) catch |err| {
+ current_archetype_storage.undoNew();
+ return err;
+ };

At this point, our entity would be in the new table! The new table has a new row with all of our component values, too! But the old table row still exists: we need to remove it.

Removing the old table row, updating pointers

We’ll use a swap removal (swapping the row that may be in the middle of the table somewhere, with the last row in the table, and finally decreasing the size of the table by one.)

+ var swapped_entity_id = archetype.entity_ids.items[archetype.entity_ids.items.len - 1];
+ archetype.remove(old_ptr.row_index) catch |err| {
+ current_archetype_storage.undoNew();
+ return err;
+ };

Notably, archetype.remove swap removes old_ptr.row_index from the table. But in doing so, our global mapping of entities entity ID -> entity ptr has become invalid! So we correct it:

+ try entities.entities.put(entities.allocator, swapped_entity_id, old_ptr);

Last but not least, the entity we were using setComponent on has moved to a new archetype table, and a new row. We update it’s pointer in the global entities map:

+ try entities.entities.put(entities.allocator, entity, Pointer{
+ .archetype_index = @intCast(u16, archetype_entry.index),
+ .row_index = new_row,
+ });
+ return;

Setting the value of a component in a table

Earlier in Entities.setComponent we had invoked ArchetypeStorage.set:

 if (new_hash == old_hash) {
 const ptr = entities.entities.get(entity).?;
 try current_archetype_storage.set(ptr.row_index, name, component);
 return;
 }

This function will just find the ErasedComponentStorage (column storage) for name, cast it to the type of component so we have ComponentStorage(T) and update row_index to have the value component:

 pub const ArchetypeStorage = struct {
 ...

+ pub fn set(storage: *ArchetypeStorage, row_index: u32, name: []const u8, component: anytype) !void {
+ var component_storage_erased = storage.components.get(name).?;
+ var component_storage = ErasedComponentStorage.cast(component_storage_erased.ptr, @TypeOf(component));
+ try component_storage.set(storage.allocator, row_index, component);
+ }
};

Over in ComponentStorage, we implement the set method - which is quite simple - if the data array isn’t large enough (we haven’t actually allocated storage for the row yet), then we allocate it to undefined memory - and finally we set row_index to the component value:

 pub fn ComponentStorage(comptime Component: type) type {
 return struct {
 total_rows: *usize,
 data: std.ArrayListUnmanaged(Component) = .{},

 const Self = @This();

+ pub fn set(storage: *Self, allocator: Allocator, row_index: u32, component: Component) !void {
+ if (storage.data.items.len <= row_index) try storage.data.appendNTimes(allocator, undefined, storage.data.items.len + 1 - row_index);
+ storage.data.items[row_index] = component;
+ }
 };
}

Finally, we can create entities and add/update components on them!

Entities.setComponent is fully implemented! These lines from our test earlier now work:

 try world.setComponent(player, "Name", "jane"); // add Name component
 try world.setComponent(player, "Location", Location{}); // add Location component
 try world.setComponent(player, "Name", "joe"); // update Name component

A copy of the full code at this point is available here and you can run it using zig test ecs.zig as before.

Getting component values

Getting component values is pretty simple:

 pub const Entities = struct {
 ...
+ pub fn getComponent(entities: *Entities, entity: EntityID, name: []const u8, comptime Component: type) ?Component {
+ var archetype = entities.archetypeByID(entity);
+
+ var component_storage_erased = archetype.components.get(name) orelse return null;
+
+ const ptr = entities.entities.get(entity).?;
+ var component_storage = ErasedComponentStorage.cast(component_storage_erased.ptr, Component);
+ return component_storage.get(ptr.row_index);
+ }

This finds the archetype table the entity is stored in, then finds the components column the named component is stored in, and finally casts the ErasedComponentStorage -> ComponentStorage(Component) so we can get the row value. Notably, this means both the name of the component and the provided type must be correct, or else undefined behavior could occur. This is a fatal flaw in our ECS implementation which we will fix!

Removing components, entities

Removing components is similar to adding them (because the entity needs to move between ArchetypeStorage tables.) Removing entities is similar as well. The code is lengthy, and nearly identical, so we won’t cover it here.

Conclusions

By this point you have a relatively solid archetypal ECS. The full source code for this article is available here.

Notably, it is lacking the following which we’ll cover in part 3:

Querying of actual entites, iterators over components for a single entity, etc.
Type-safety, as noted earlier if you pass the wrong name / component type it will result in undefined behavior!
… and more

mach/ecs is available on GitHub, slightly ahead of this series and changing rapidly. Once it becomes stable, it will also be available as a standalone Zig library anyone can use in their own engine/game.

Join our Matrix chat room for ECS discussion & to help us reach Mach 1.0.

Support my work

If you like my work on Mach engine, zigmonthly.org, etc. you can sponsor me on GitHub.

Mach v0.1 - cross-platform Zig graphics in ~60 seconds

Sat, 26 Mar 2022 00:00:00 +0000

Five months ago we announced some of our vision for Mach & the future of graphics with Zig. Today we’ve reached Mach v0.1 with over 1,100 commits.

Cross-platform graphics in 60 seconds

If you have Zig v0.10 you can get started with cross-platform graphics in under 60 seconds, try it for yourself:

git clone https://github.com/hexops/mach
cd mach/gpu
zig build run-example

(not working? see known issues)

All you need is zig, git, and curl.

One key point we’re solving with Mach is the developer experience. We’re tired of people wasting hours and sometimes days getting the right versions of dependencies on their system in order to build projects!

Our glfw bindings build GLFW 100% from source using Zig. We even go so far as to build the DirectX Shader Compiler from source via Zig’s build system.

For the few inescapable system dependencies, such as Metal.framework or libx11, we package them up ourselves and our build system knows how to fetch them as needed.

Effortless cross-compilation

Because of our aggressive approach to solving dependencies, you get effortless cross-compilation to any OS literally at the flip of a switch:

$ zig build -Dtarget=x86_64-windows
$ zig build -Dtarget=x86_64-linux
$ zig build -Dtarget=x86_64-macos.12
$ zig build -Dtarget=aarch64-macos.12

The binaries you end up with are virtually static.

For Linux, Zig lets you target any glibc version, and musl too, so no more worrying if that binary will run on other machines.

Unified graphics API (Metal, Vulkan, DirectX 12)

Backed by Metal, Vulkan, DirectX 12 & OpenGL (as a fallback), you get a truly cross-platform graphics API for Windows, Linux, and macOS (Browser and Mobile coming in the future)

Unified shader language & compute shaders

There’s no need to write shaders for each graphics backend, instead you write shaders in a single modern language (WGSL):

With compute shaders, you have the ability to leverage computation on the GPU outside of graphical applications (machine learning, physics calculations, etc.) using a straightforward & approachable API that works with every GPU vendor.

Behind the scenes

Mach leverages WebGPU, a successor to WebGL. WebGPU is an up and coming graphics API being built by Mozilla, Google, Apple, Microsoft, Intel and others.

Natively, Mach uses Zig as a C/C++ compiler to build Google Chrome’s native WebGPU implementation and we use Zig’s build system so you don’t even have to deal with cmake/ninja/gn/etc.

Our infrastructure produces binary releases so you don’t even have to wait the handful of minutes it would take to compile by default. From-source builds are literally at your fingertips, though, just add -Ddawn-from-source=true to your zig build command.

`mach/gpu`: the GPU interface for Zig

mach/gpu is our Zig interface to WebGPU and comes in at just over 250 commits.

It provides a gpu.Interface, similar to how Zig provides a std.mem.Allocator interface, it’s backed by any implementation:

A NativeInstance like Dawn (Google Chrome’s WebGPU implementation.)
(future) A browser implementation when targeting WebAssembly.

Imagine future implementations: maybe a pure-Zig implementation? Maybe a debugger implementation that wraps an existing one and streams all API calls to disk so you can replay them later and step through graphics API calls. Lots of possibilities here!

It’s a comprehensive interface, covering the 176 methods, 73 data structures, and 43 enum types that WebGPU exposes today - but there’s still much to do around documentation, fixing bugs, and ensuring we match the browser API nicely. but the foundation is all there!

Zig ≈ greatness

Zig provides some excellent runtime safety and catches many of the mistakes people make in C/C++ (memory leaks, integer overflow, index out of bounds, etc.)

In fact, we’ve caught several instances of undefined behavior in GLFW, and even illegal integer coercion in the DirectX Shader Compiler - all just by compiling C/C++ code using Zig.

The reason we’re really ecstatic about Zig, though, are what it promises us in the future:

Blazing fast compilation, compiling and running a program faster than Python can interpret it - impressive!
Hot code swapping, how cool would it be to edit variables etc. as your game is running?

Entity Component System

We’re building an Entity Component System in a blog series and inspired by:

Data Oriented Design
Database design
Advice from the authors of Bevy (a very popular ECS)

It’s still early stages, we’ve got some ways to go! But we’re on our third major revision and beginning to face the interesting problems. Keep an eye out for updates on that blog series coming soon.

Sounds great! What’s the catch?

Mach (and Zig) are still very early stages! APIs are going to change and break. Mach is missing a lot!

Documentation..
Examples..
Demos..
Browser and Mobile support..
..Literally everything else that makes a game engine

If you’re looking for cross-platform graphics in Zig, Mach is for you! Otherwise, you’ll probably need to wait a bit.

Getting started

Check out the GitHub and in particular this example code.

There’s a ton of material out there about WebGPU already, check out this excellent and comprehensive introductory article and these awesome samples. It should be easy to map any of these to the Mach gpu.Interface since it’s the same API, just Ziggified!

Join our Matrix chat room to get help, discuss, etc.

What’s next?

My lightning talk in which I’ll be making the case for Mach engine and conveying the vision for where we go from here will be presented at the first-ever European Zig meetup in Milan, Italy on Apr 9-10.

If like me you are unable to attend in person, the short video will be available afterwards!

Help us reach Mach v1.0

Consider sponsoring me on GitHub if you like my work, so I can do more of it!

Join our Matrix chat room to discuss ideas - collaboration very welcome!

Thanks for coming along in our journey!

Zig hashmaps explained

Sat, 29 Jan 2022 00:00:00 +0000

If you just got started with Zig, you might quickly want to use a hashmap. Zig provides good defaults, with a lot of customization options.

Here I will try to guide you into choosing the right hashmap type.

60-second explainer

You probably want:

var my_hash_map = std.StringHashMap(V).init(allocator);

Or if you do not have string keys, you can use an Auto hashmap instead:

var my_hash_map = std.AutoHashMap(K, V).init(allocator);

Where K and V are your key and value data types, respectively. e.g. []const u8 for a string.

You can then use these APIs:

Insert a value

try my_hash_map.put(key, value);

Insert a value, assert entry does not already exist

try my_hash_map.putNoClobber(key, value);

Note putNoClobber may be renamed to something like putAssumeNoEntry in the near future: ziglang/zig#10736

Get a value

var value = my_hash_map.get(key);
if (value) |v| {
 // got value "v"
} else {
 // doesn't exist
}

Get a value, insert if not exist

var v = try my_hash_map.getOrPut(key);
if (!v.found_existing) {
 // We inserted an entry, specify the new value
 // This is a conditional in case creating the new value is expensive
 v.value_ptr.* = "my value";
}

var value = v.value_ptr.*; // use the value

You can find more APIs by going here and using your browser’s builtin search for pub fn.

About key data types

Zig hash map types start with the data type of the key:

std.StringHashMap - uses a good default hashing function for string keys
std.AutoHashMap - uses a good default hashing function for most data types
std.HashMap - the “bring your own hashing function” option

Note: AutoHashMap does not support slices, such as []const u8 string slices, because that is a pointer to an array and it is ambiguous whether or not you intend to hash the array elements or the pointer itself. You can use the generic std.HashMap for any slice type, you just have to provide your own hash functions.

Hashmaps are also sets

A set in Zig is just a hashmap with a void value:

var my_hash_map = std.AutoHashMap(K, void).init(allocator);

try my_hash_map.put(key, {}); // `{}` is a value of type `void`

Advanced usages

If you’re just getting started with Zig, don’t worry too much about the below. Just know that you have options available should you need to reduce memory usage or optimize your use of hashmaps in the future.

Managed vs. unmanaged hashmaps

You can add Unmanaged to the end of a Zig hashmap data type, e.g. std.StringHashMapUnmanaged in order to get the unmanaged version.

This merely doesn’t carry an allocator internally, instead you must pass the allocator into every method of the hashmap. While only a few bytes, this can be a useful optimization if you’re storing many hashmaps for example.

Managed:

var my_hash_map = std.StringHashMap(V).init(allocator);

Unmanaged:

var my_hash_map = std.StringHashMapUnmanaged(V){};

Array hash maps

Zig actually provides two hashmap implementations in the standard library

std.HashMap, perfect for every-day use cases:

Optimized for lookup times primarily
Optimized for insertion/removal times secondarily

std.ArrayHashMap, useful in some situations:

Iterating over the hashmap is an order of magnitude faster (a contiguous array)
Insertion order is preserved.
You can index into the underlying data like an array if you like
Deletions can be performed one of two ways, mirroring the ArrayList API:
- swapRemove: swaps the target element with the last element in the list to remove it
- orderedRemove: removes target element by shifting all elements forward, maintaining current ordering

Hashmap context

If you choose to use std.HashMap or std.ArrayHashMap directly (without the String or Auto prefix), then you’ll find it wants a context parameter and max load percentage:

var my_hash_map = std.HashMap(K, V, std.hash_map.AutoContext(K), std.hash_map.default_max_load_percentage);

The context parameter lets you embed some of your own data within the hash map type. This can be useful for reducing the amount of memory that a hash map takes up when doing a string table.

Pick your hashmap

Regular implementation:

Key type	Managed?	How to initialize
`String`	yes	`std.StringHashMap(V).init(allocator)`
`Auto`	yes	`std.AutoHashMap(K, V).init(allocator)`
`String`	`Unmanaged`	`std.StringHashMapUnmanaged(V){}`
`Auto`	`Unmanaged`	`std.AutoHashMapUnmanaged(K, V){}`

ArrayHashMap implementation:

Key type	Managed?	How to initialize
`String`	yes	`std.StringArrayHashMap(V).init(allocator)`
`Auto`	yes	`std.AutoArrayHashMap(K, V).init(allocator)`
`String`	`Unmanaged`	`std.StringArrayHashMapUnmanaged(V){}`
`Auto`	`Unmanaged`	`std.AutoArrayHashMapUnmanaged(K, V){}`

Learn more

The source code is very readable:

Help improve this page

I wrote this article quickly because I needed to explain my choice of hashmaps in the “Let’s build an Entity Component System from scratch” series and there was no better source of this info. I’m sure there are things that can be improved.

Feel free to send a PR!

Let's build an Entity Component System from scratch (part 1)

Sun, 16 Jan 2022 00:00:00 +0000

In this multi-part series we’ll build the Entity Component System used in Mach engine in the Zig programming language from first principles (asking what an ECS is and walking through what problems it solves) all the way to writing an implementation in a low-level programming language. The only thing you need to follow along is some programming experience and a desire to learn.

In this article, we’ll mostly go over the problem space, data oriented design, the things we need our ECS to solve, etc. In the next article, implementation will begin.

Motivation
My approach to complex software architecture
What really is an entity component system, anyway?
What problems does an ECS solve?
Start with data oriented design
What would data oriented design look like? (code starts here!)
Sparse data storage
Archetype storage
- Comptime archetype storage
- Runtime archetype storage
Designing our ECS
Next up: starting our ECS implementation

Motivation

I’ve used and written more traditional OOP scene graphs in the past. These are often the core engine architecture used to represent everything in game worlds: they’re used in Unity historically (which is now migrating to ECS due to popular demand) and even in other modern engines such as Godot.

For Mach engine, however, we’re adopting an ECS as our core architecture. ECS has gained great momentum in recent years for its composition and performance benefits.

My approach to complex software architecture

What user problems does the proposed architecture (scene graphs, ECS, React-like frameworks, etc.) solve?
How does the proposed architecture typically solve such problems?

The key point here is that, personally, I find it useful to intentionally avoid looking directly at code for the implementations themselves.

I’ve used this approach to to great success before: the nice thing about this is that the end result really fits the language, using patterns and features specific to the language - it doesn’t just end up feeling like a port of some other language’s implementation.

I’ve researched a bit about ECS in general, and have chatted with people familiar with ECS, but haven’t read any other’s code. No doubt, initially, I’ll get some aspects wrong! As this series of articles progresses over the coming months, though, you’ll see how this can be a winning tactic as we learn together!

What really is an entity component system, anyway?

I’ve found the Rust project Bevy ECS to have a great succinct explanation, which I further simplify here:

Entities: a unique integer
Components: structs of plain old data
Systems: normal functions

When you hear this, things may start to sounds a whole lot simpler! Those are the core concepts of an ECS.

There is one other concept of an ECS that I think is particularly important:

Archetype: A chosen set of components that an entity of a certain type will have.

What problems does an ECS solve?

I’ve identified two problems it solves.

First and foremost is making it easy for game developers to architect their code compared to them doing it manually. If it’s easier for someone to structure their code themselves, manually, then such a system is not useful at all! Of course, as complexity and the scale of software increases then a consistent system is far more useful than a bunch of ad-hoc systems.

The second problem ECS solves, I believe, is making your software architecture efficient without you really having to think too much about it. You don’t have to think about how to structure all your code & data for logic first, and then for performance, but rather get good performance by nature of following patterns.

Start with data oriented design

ECS overlaps with data oriented design in many ways (although it’s roots are much earlier). There are many talks about data oriented design including Mike Acton’s at CppCon, and my personal favorite “A Practical Guide to Applying Data-Oriented Design” by Andrew Kelley. You don’t have to watch either, I’ll cover the important concepts we use here. But I highly suggest every developer watch Andrew Kelley’s talk above. It’s eye opening no matter what kind of programming you are doing.

Let’s work forwards, not backwards: We’re not starting by building an ECS, we’re starting by building a proper data oriented design for CPU cache and memory efficiency, and then we’re working towards “how do we make that easier for people to do by default?” and looking to existing ECS architectures for inspiration.

What would data oriented design look like? (code starts here!)

A simple first approach would be something like this:

const Player = struct {
 name: []const u8, // a string / byte slice
 location: Vec3,
 velocity: Vec3,
 health: u8,
 team: Team,
 alive: bool,
};

const Cat = struct {
 name: []const u8, // a string / byte slice
 location: Vec3,
};

const Monster = struct {
 location: Vec3,
 health: u8,
};

// All the players, cats, monsters in our game world.
var players: ArrayList(Player) = .{};
var cats: ArrayList(Cat) = .{};
var monsters: ArrayList(Monster) = .{};

// The index of a player in the players array, a cat in the cats array, etc.!
const Entity = u32;

Now we can refer to players, cats, or monsters by using an entity ID (their index in the array), which we call an entity. We could also write functions (called systems) which iterate over these arrays and e.g. compute physics for players.

However, we can improve this quite a bit!

Sparse data storage

Comptime sparse data

It’s likely that most players will be alive in our game, only a few will be dead at a time - but yet we’re paying the cost of storing which players are dead for every living player (via the Player.alive struct field)!

We can eliminate paying the cost of alive: bool per player by removing the field entirely, and having what I call compile time sparse data instead:

var alive_players: ArrayList(Player) = .{};
var dead_players: ArrayList(Player) = .{};

This not only reduces the amount of memory each Player entity takes up because we no longer store an alive: bool per player, but also it:

Improves performance by ensuring more players fit into L1/L2/L3 cache.
Reduces the amount of players we must skip (and reduces potential cache misses) because in some cases we might only be interested in alive players and have to skip over dead ones when iterating.

This introduces some complexity for us to deal with, though:

Now if a player goes from dead->alive, or alive->dead, we need logic to remove it from the old array and put it in the new one.
When we move a player from one array to another, the Entity ID we use to refer to that player (the array index) has changed! So if someone is storing a player Entity ID in order to have reference to it somewhere, we’d need to have logic to update that.

Now we start to see one thing our ECS needs to make simpler!

I call this type of data comptime sparse data.

Runtime sparse data

In an ideal world, we’re able to pre-declare all sparse data at compile time like we did above:

var alive_players: ArrayList(Player) = .{};
var dead_players: ArrayList(Player) = .{};

But sometimes, this just isn’t possible:

Maybe players in your game can give other players a customer nickname to display above their head. Again, for most players this won’t be set - but for some players it will be! Ideally we don’t have to pay the cost of storing a nickname string pointer for every player in the game without one
Maybe a handful of players out of thousands are given the speciality of having a custom weapon, they get to choose it’s type, a custom name for it, and even the damage it should do! Where should we store that information?
…

In this case, we could use a hash map:

const PlayerNickname = []const u8; // a string

const Weapon = struct {
 custom_name: []const u8, // a string
 type: WeaponType,
 damage: u8,
};

var players: ArrayList(Player) = .{}; // all players
var players_with_nicknames = AutoHashMap(Entity, PlayerNickname).init(allocator);
var players_with_weapons = AutoHashMap(Entity, Weapon).init(allocator);

Now we’ve got a mapping of player Entity IDs -> their nicknames and weapons. We only pay the cost of storing this information for players that do actually have these specialties - not for every player.

I call this type of data runtime sparse data.

Improving performance

Consider our player storage as it stands right now:

const Player = struct {
 name: []const u8, // a string / byte slice
 location: Vec3,
 velocity: Vec3,
 health: u8,
 team: Team,
};

var players: ArrayList(Player) = .{}; // all players

Because of the way structs get laid out in memory with padding, our players array above would end up having a larger memory footprint than needed. So we actually benefit from using a separate array for every type of data (thanks, Unity!):

var player_names: ArrayList([]const u8) = .{};
var player_locations: ArrayList(Vec3) = .{};
var player_velocities: ArrayList(Vec3) = .{};
var player_healths: ArrayList(u8) = .{};
var player_teams: ArrayList(Team) = .{};

Luckily, we don’t actually have to enumerate all our fields out like this: Zig has a nice MultiArrayList type which does this for us, we need change only one line:

const Player = struct {
 name: []const u8, // a string / byte slice
 location: Vec3,
 velocity: Vec3,
 health: u8,
 team: Team,
};

-var players: ArrayList(Player) = .{}; // all players
+var players: MultiArrayList(Player) = .{}; // all players

Not only does this use less memory, it also improves CPU cache efficiency a ton, especially when iterating over a lot of players to do work with them. If you’re curious why, then you should watch Andrew Kelley’s “A Practical Guide to Applying Data-Oriented Design” talk!

Archetype storage

Comptime archetype storage

Up until now, we’ve assumed we have pre-defined archetypes (“player”, “cat”, “monster”):

// All the players, cats, monsters in our game world.
var players: ArrayList(Player) = .{};
var cats: ArrayList(Cat) = .{};
var monsters: ArrayList(Monster) = .{};

This is ideal: we don’t need to ask the computer to do any work to find out where players, cats, or monsters are stored - we just know at compile time because they’re in that variable. When someone uses our ECS, we could have them write a compile time function like:

World(.{
 Player,
 Cat,
 Monster,
})

And that’s great because it means our ECS “world” can be aware ahead of time exactly which archetypes it needs to store. It could write out those var players: ArrayList... variables for us.

I call this comptime archetype storage.

Runtime archetype storage

However, real games are much more complex: we might not really know at the time we’re declaring the World all the different archetypes we plan on storing. Code gets messy. In some cases, maybe we even need to define some archetypes of a common type at runtime. For example, if we wanted to allow configuring red and blue here (or the number of teams) via a configuration file on disk or via a GUI:

var red_team_players: ArrayList(Player) = .{};
var blue_team_players: ArrayList(Player) = .{};

Of course our Player could have a team field in it to represent the team, but there may be cases where storing a separate list of entities like this is needed without pre-declaring it. If we want to do that, we could use a hashmap:

var runtime_archetypes = AutoHashMap([]const u8, *anyopaque).init(allocator);

In this model, we could store the archtype string name as the hashmap key (for example, the @typeName(Player) if we wanted, or maybe a custom name like red, blue, etc.). The value of the hashmap would need to be different types: an ArrayList(Player), an ArrayList(Monster), etc. and so we would store a type-erased *anyopaque (like a C void*) pointer. When we get a value out, we’ll need to “know” what type of ArrayList to cast the pointer back to. It won’t store that info for us.

I call this runtime archetype storage.

Designing our ECS

We now start to see some of the things our ECS architecture should solve:

Typed entity storage (how you interact with a list of players, monsters, etc.)
Sparse data: both comptime and runtime
Archetype storage: both comptime and runtime

Additionally, these are the design principles I’ve come up with:

Clean-room implementation (I’ve not read any other ECS implementation code), just working from first-principles as an engineer
Solve the problems ECS solves, in a way that is natural to Zig and leverages Zig comptime.
Fast. Optimal for CPU caches, multi-threaded, leverage comptime as much as is reasonable.
Simple. Small API footprint, should be natural and fun - not like you’re writing boilerplate.
Enable other libraries to provide tracing, editors, visualizers, profilers, etc.

From this, you can easily gather that storing entities is actually only a small (but critical) portion of this system. In the next article we will get into the details of implementing this in code, and go on to explore more challenging topics like multi-threading, systems, and scheduling in future articles.

Next up: starting our ECS implementation

As this series develops, all the code is being developed in the Mach repository’s ecs subfolder on GitHub. The articles will lag slightly behind.

As more articles come out, you can find them here. Join us in developing it, give us advice, etc. on Matrix chat.

If you like what I’m doing, you can sponsor me on GitHub.

Perfecting GLFW for Zig, and finding lurking undefined behavior that went unnoticed for 6+ years

Sun, 31 Oct 2021 00:00:00 +0000

Today, I am announcing mach-glfw: Ziggified GLFW bindings with 100% API coverage, zero-fuss installation, cross compilation, and more.

Building Mach for everyone

If Mach engine only benefits people interested in using that engine, and not the broader Zig (and even gamedev) community I would consider that a total failure.

Whether you’re interested in using all of Mach, just some of it with your own engine / project, or just the tools/ideas we develop in the future (with Unity, Unreal, etc.), I truly aim to produce something that benefits you.

Mach is in super early stages, I’ve spent the last four months perfecting a Zig interface to GLFW, and making no-fuss installation and cross-compilation a reality. Today, you can benefit from that work too.

Building GLFW for every platform

Just zig and git, that’s the idea. The GLFW C code is compiled with zig, and the build.zig file automatically uses git to clone (a very minimal set of) system dependencies for you (X11 libraries, etc.)

No installing apt packages. No dealing with missing header errors. It should just work out-of-the-box, and for every platform:

Today, this works for GLFW itself. Cross-compilation of OpenGL and Vulkan apps is not yet fully functional. We’re working on it, though.

Perfecting GLFW for Zig

Aside from platform support, mach-glfw now has:

100% API coverage of GLFW v3.3.4. Every function, type, constant, etc. has been wrapped in a ziggified API.
130+ tests, with CI testing Linux, Windows, Mac (x86 and M1/ARM) and cross-compilation between them.

You might be asking: why Zig bindings, when Zig can interface directly with C? Ziggified bindings to GLFW get us:

Errors as zig errors instead of via a callback function.
Enums: always know what value a GLFW function can accept as everything is strictly typed. And use the nice Zig syntax to access enums, like window.getKey(.escape) instead of c.glfwGetKey(window, c.GLFW_KEY_ESCAPE)
Slices instead of C pointers and lengths.
packed structs represent bit masks, so you can use if (joystick.down and joystick.right) instead of & | etc. bitwise operators.
true and false instead of c.GLFW_TRUE and c.GLFW_FALSE.
Generics: use window.hint instead of glfwWindowHint, glfwWindowHintString, etc.
Methods, e.g. my_window.hint(...) instead of glfwWindowHint(my_window, ...)

Explicit error handling solves a real problem

GLFW traditionally passes errors to the user via a callback. This can make errors easy to ignore, as well as difficult to correlate and handle effectively at the time of the function invocation.

We translated a a Vulkan example to mach-glfw, which you can try for yourself today:

After porting it, we found that the example was crashing with a NoWindowContext error. Strange?

As it turns out, we had found a small bug in the vulkan-zig example code, it was calling glfwSwapBuffers which is not needed for Vulkan. The error went unnoticed because it’s easy to miss errors with GLFW’s error callback handling style. But with mach-glfw, it was an explicit error you have to handle e.g. via try glfw.swapBuffers() - we literally couldn’t miss it.

Finding lurking undefined behavior in 6+ year old GLFW code

One particularly frustrating issue was tracking down why the last part of the GLFW API we needed to wrap for 100% coverage, the glfwSetWindowIcon function, was crashing:

Test [76/135] Window.test "setIcon"... Illegal instruction at address 0x2cee09
upstream/glfw/src/x11_window.c:0:0: 0x2cee09 in _glfwPlatformSetWindowIcon (/mach/glfw/upstream/glfw/src/x11_window.c)
upstream/glfw/src/window.c:511:5: 0x2de484 in glfwSetWindowIcon (/mach/glfw/upstream/glfw/src/window.c)
_glfwPlatformSetWindowIcon(window, count, images);
^
/mach/glfw/src/Window.zig:508:28: 0x23a083 in Window.test "setIcon" (test)
c.glfwSetWindowIcon(self.handle, @intCast(c_int, im.len), &tmp[0]);
^
/usr/local/bin/lib/std/special/test_runner.zig:77:28: 0x25a0d1 in std.special.main (test)
} else test_fn.func();
^
/usr/local/bin/lib/std/start.zig:517:22: 0x2896bc in std.start.callMain (test)
root.main();
^
/usr/local/bin/lib/std/start.zig:469:12: 0x25c117 in std.start.callMainWithArgs (test)
return @call(.{ .modifier = .always_inline }, callMain, .{});
^
/usr/local/bin/lib/std/start.zig:434:12: 0x25bec2 in std.start.main (test)
return @call(.{ .modifier = .always_inline }, callMainWithArgs, .{ @intCast(usize, c_argc), c_argv, envp });
^
???:?:?: 0x7f4b7c3280b2 in ??? (???)

That’s odd? Illegal instruction at address 0x2cee09 - are we corrupting the stack somehow? Is this a Zig compiler bug?

Running in lldb didn’t help with shining any light on the problem, either:

After poking around at the stack, checking all pointers and lengths were valid, etc. I was at a loss. The mach-glfw code sure seemed valid, and yet, this crash. I managed to track the crash down to the first iteration of a loop in GLFW’s x11_window.c:

void _glfwSetWindowIconX11(_GLFWwindow* window, int count, const GLFWimage* images)
{
 if (count)
 {
 int longCount = 0;

 for (int i = 0; i < count; i++)
 longCount += 2 + images[i].width * images[i].height;

 long* icon = _glfw_calloc(longCount, sizeof(long));
 long* target = icon;

 for (int i = 0; i < count; i++)
 {
 *target++ = images[i].width;
 *target++ = images[i].height;

 for (int j = 0; j < images[i].width * images[i].height; j++)
 {
 // illegal instruction on first iteration?
 *target++ = (images[i].pixels[j * 4 + 0] << 16) |
 (images[i].pixels[j * 4 + 1] << 8) |
 (images[i].pixels[j * 4 + 2] << 0) |
 (images[i].pixels[j * 4 + 3] << 24);
 }
 }
...

Reaching my limits

At this point, I feel confident in saying:

The Zig code is correct, the pointers are valid, the lengths are correct, everything’s right.
The GLFW code is pretty popular, and it’s been around for 6 years. Seems unlikely it’s a bug in GLFW?

Luckily, my brother (and reverse engineer) @Andoryuuta was available to help debug, so I pulled him in. Stepping through instructions, we could see clearly that after a bit shift we were stepping into the abyss:

* thread #1, name = 'test', stop reason = instruction step over
frame #0: 0x00000000002c6f84 test`_glfwPlatformSetWindowIcon(window=0x00000000004e53d0, count=1, images=0x00007fffec0b3000) at x11_window.c:2156:58
2153 *target++ = (images[i].pixels[j * 4 + 0] << 16) |
2154 (images[i].pixels[j * 4 + 1] << 8) |
2155 (images[i].pixels[j * 4 + 2] << 0) |
-> 2156 (images[i].pixels[j * 4 + 3] << 24);
2157 printf("DID WE GET HERE???x\n");
2158 }
2159 }
(lldb)
Process 6516 stopped
* thread #1, name = 'test', stop reason = instruction step over
frame #0: 0x00000000002c6c21 test`_glfwPlatformSetWindowIcon(window=0x00000000004e53d0, count=1, images=0x00007fffec0b3000) at x11_window.c:0
1 //========================================================================
2 // GLFW 3.3 X11 - www.glfw.org
3 //------------------------------------------------------------------------
4 // Copyright (c) 2002-2006 Marcus Geelnard
5 // Copyright (c) 2006-2019 Camilla Löwy <elmindreda@glfw.org>
6 //
7 // This software is provided 'as-is', without any express or implied
(lldb)
Process 6516 stopped

Inspecting the binary in IDA Pro we were able to see that we were jumping into an __asm { ud1 } section (ud1 standing for “undefined instruction 1”):

It turns out that clang’s UBSan inserts these instructions as traps for when the compiler thinks there is undefined behavior occurring, such as if a pointer addition leads to an overflow. This is super interesting, but unfortunately doesn’t always give a compiler error. We got lucky and found someone else who ran into this through Google:

I believe LLVM explicitly generates a ud2 x86 instruction because “it determined” there’s undefined behavior in the C code. So first I wonder which flags you’re passing it through zig (i.e. how strict are you being with the settings?) — Abner (@AbnerCoimbre)

And indeed, compiling via zig build test -Drelease-fast (which turns off UBsan) made the crash go away. So where’s the undefined behavior?

If we squint at the code and assume all pointers, counts, and indices are correct, you might be able to spot it:

void _glfwSetWindowIconX11(_GLFWwindow* window, int count, const GLFWimage* images)
{
 ...
 long* target = icon;

 for (int i = 0; i < count; i++)
 {
 ...
 for (int j = 0; j < images[i].width * images[i].height; j++)
 {
 // illegal instruction on first iteration?
 *target++ = (images[i].pixels[j * 4 + 0] << 16) |
 (images[i].pixels[j * 4 + 1] << 8) |
 (images[i].pixels[j * 4 + 2] << 0) |
 (images[i].pixels[j * 4 + 3] << 24);
 }
 }
...

What is happening here is that:

images[i].pixels[j * 4 + 0] is returning an unsigned char (8 bits)
~~It is then being shifted left by << 16 bits. !!! That’s further than an 8-bit number can be shifted left by, so that’s UB~~
- EDIT: Actually, it turns out that’s not exactly right, it’s the << 24 that’s the cause of the UB, thanks @Maato for pointing this out and explaining in better detail than I could.

Suddenly, it all makes sense. And if we load an equal snippet of code into Godbolt we can see what is happening when we compile without UBSan / the -fsanitize=undefined flag:

Without UBsan, clang merely uses the 32-bit EAX register as an optimization. It loads the 8-bit number into the 32-bit register, and then performs the left shift. Although the shift exceeds 8 bits, it does not get truncated to zero - instead it is effectively as if the number was converted to a long (32 bits) prior to the left-shift operation.

This explains why nobody has caught this UB in GLFW yet, too: it works by accident! Just because the compiler likes to use 32-bit registers in this context.

And this change benefits all the languages out there using GLFW: glfw/glfw#1986

Defaults are critical

This code, and undefined behavior, has been in GLFW for over 6 years according to git blame.

Anybody using GLFW could have enabled UBSan in their C compiler. Anybody could have run into this same crash and debugged it in the last 6 years. But they didn’t.

In mach-glfw, we compile all of GLFW’s C code with Zig (which is also a fully functional C and C++ compiler), with UBSan enabled by default.

Only because Zig has good defaults, because it places so much emphasis on things being right out of the box, and because there is such an emphasis on having safety checks for undefined behavior - were we able to catch this undefined behavior that went unnoticed in GLFW for the last 6 years.

Thanks for reading

All key Mach engine developments will be posted here.

Follow Mach engine on GitHub, and if you like what I’m doing please consider sponsoring my work.

Mach Engine: The future of graphics (with Zig)

Sun, 17 Oct 2021 00:00:00 +0000

In the coming months, we’ll begin to have truly cross-platform low-level graphics, with the ability to cross compile GPU-accelerated applications written in Zig from any OS and deploy to desktop, mobile, and (in the future) web.

Mach engine

I’ve been working on Mach Engine for about 4 months now, although it as a project is many years in the making, and I believe in the next 4-6 months we’ll have completion of the first key milestone: truly cross platform graphics and seamless cross compilation.

Vision

Today, I share only the first milestone: Mach engine core. I’ve been working on this for around 1 year now, and we’re close (maybe 4-6 months away) from completion:

Zero fuss installation & cross compilation

Only zig and git are needed to build from any OS and produce binaries for every OS. You do not need any system dependencies, C libraries, SDKs (Xcode, etc.), C compilers or anything else.

We’re able to achieve this thanks to two things:

Zig has fantastic cross-compilation support, including its own custom linker zld written by Jakub Konka which is capable of supporting MacOS cross compilation.
Mach doing the heavy lifting of packaging the required system SDK libraries and C sources for e.g. GLFW so our Zig build scripts can simply git clone them for you as needed for the target OS you’re building for, completely automagically.

Truly cross-platform graphics API

DirectX 12, Metal, Vulkan & OpenGL

Imagine a low-level, little to no overhead graphics API that unifies DirectX, Metal, Vulkan, and OpenGL (if no others are available):

This isn’t anything new: all modern engines provide this, Godot has been working towards this for years (and still is), and there exist abstraction layers for Vulkan over most of these APIs as well.

Vendor support

An API is only as good as the momentum behind it. What modern API can target the largest array of platforms with the most vendor backing?

Microsoft sees DirectX as the future, not Vulkan. (DirectX 13 is coming by the end of 2022.)
Apple sees Metal as the future, not Vulkan. OpenGL and OpenCL are deprecated, and private legal arguments with Khoronos make it unlikely we’ll ever see OpenGL or Vulkan on Apple hardware ever again.
Google, with their Fuschia OS appears to be primarily into Vulkan from a system-level POV.
NVIDIA, AMD, and Intel generally support as many graphics APIs as possible, they want to sell hardware.

One API that Apple, Microsoft, and Google can all agree on

Outside the bounds of traditional graphics APIs there exists an attempt to provide a unified API across all platforms, WebGPU (not to be confused with the much older WebGL).

Mozilla, Google, Apple, and Microsoft all got together to build an abstraction layer over the modern graphics APIs - finding the common ground between Direct3D 12, Metal, and Vulkan - plus a safe way to expose that functionality in browsers.

The name WebGPU might lead you to believe that this is only for browsers, and that it may not be low-level or fast - but this really couldn’t be further from the truth.

Apple & Google’s role is what makes WebGPU unique, and why we chose it

What is new about WebGPU in my view is the vendors playing key roles in its development, and the fact that it grew outside the Khronos Group.

Although abstraction layers over modern graphics APIs are nothing new - as Apple, Google, and Microsoft continue to get more into manufacturing their own hardware (it’s clear this is a strategic move for them) we should ask ourselves how this will change the landscape, and WebGPU is the first cross-vendor API to be produced by this new ecosystem.

WebGPU extended thoughts

Is WebGPU "native enough"? Yes

For browsers, WebGPU will require sandboxing and validation layers. But in native uses, this can all be turned off, and the WebGPU developers are clearly thinking about this use case:

Google's implementation of WebGPU, Dawn, can be configured to effectively turn off all browser sandboxing / validation that could harm performance due to its client/server architecture.
Mozilla / gfx-rs Rust engineers have published articles such as "The point of WebGPU on native".

As for the quality of implementations, we could compare the amount of resources going into e.g. Google's WebGPU implementation vs. the amount of resources going into Unity/Unreal/MoltenVK/other graphics abstraction layers - but I suspect they're about equal.

Will WebGPU be implemented on GPUs natively? Maybe someday

Not anytime soon. We get some insight into this via @kvark, a WebGPU developer:

[...] We are not in Khronos, and therefore we have limited participation from IHVs (only Intel and Apple are active). WebGPU was never designed to be implemented by the drivers. I mean, it would totally be rad, in the context of how usable WebGPU can be on native, but it couldn't be the requirement from the start.

But as WebGPU usage grows or even becomes prodominate due to it being the most powerful API in browsers, and as Microsoft, Google, and Apple continue to develop their own hardware - I think it's not unreasonable to think that it's possible some day WebGPU will be an even more direct 1:1 mapping between a cross-platform API and low-level APIs, more direct than Vulkan abstraction layers such as MoltenVK (which is required to get Vulkan working on top of MacOS's Metal API) - with the potential that some vendor starts asking "what would a GPU native WebGPU implementation look like?"

Momentum of WebGPU vs. Vulkan

To quote Dzmitry Malyshau / kvark, a Mozilla engineer working on gfx-rs and WebGPU:

At some point, it comes down to the amount of momentum behind the API. In case of WebGPU, we have strong support from Intel and Apple, which are hardware vendors, as well as Google, who can influence mobile hardware vendors. We are making the specification and have resources to appropriately test it and develop the necessary workarounds. It's the quantity to quality transition that sometimes just needs to cross a certain threshold in order to succeed.

According to some, Nvidia and AMD tend to develop new features with Microsoft as part of DirectX. Only then are they "ported" back to Vulkan and OpenGL. I think that says a lot.

What progress has been made so far on Mach Engine?

Today, we have cross-compilation of GLFW on all desktop OSs working out of the box with nothing more than zig and git:

This involved:

Packaging MacOS SDKs and Linux system X11/Wayland libraries into SDKs, and creating Zig build scripts that could merely git clone them and utilize them for cross-compilation.
Purchasing Apple M1 hardware to test on, and for GitHub Actions as it doesn’t support it.
Normalizing symlinks in Mac/Linux SDKs everywhere so that Windows users don’t have a hard time with Git symlink management.
Contributing a small fix to the Zig linker

All this to say, we’re really taking a holistic approach to achieve this.

What’s next? WebGPU

I’m happy to report that a fair amount of progress on this front has been made.

Here is Google’s WebGPU implementation, Dawn, compiled using zig:

This includes:

A ~500 line port of the hello_triangle example from Dawn to Zig
A ~1200 line build.zig file which compiles all the Dawn sources using Zig, without using Google’s ninja/etc development tools.
A hack to workaround a bug in Zig where ObjC++ .mm files are not yet recognized.
C shims for the dawn_native C++ API and utility APIs, which are required in order to bind Dawn to an actual GLFW window.

There are a few weeks of work to do before this can be merged and will be usable by others, please stay tuned for that.

After that will be development of idiomatic Zig bindings to the WebGPU C API which is shared between implementations such as Dawn and the Rust’s gfx-rs/wgpu-native implementation (we could theoretically switch between them at startup in the future, but we’ll probably stick with Dawn as it does not require a separate Rust toolchain and it would prevent out-of-the-box cross compilation.)

When will there be games, examples, etc.?

It’ll be a while because I am focusing purely on the groundwork first. It’s unlikely you’ll see anything with real demo value before later next year.

I’m sure that will be disheartening to hear - and may make you to think there’s nothing of substance here. I totally understand that view, but I hope you’ll stay tuned because I’m in this for the long haul and it’s not my first rodeo (I previously spent 4 years writing a game engine in Go, and have worked at a devtools startup for 7 years, with my biggest lesson from of those experiences being the importance of demos and examples.

Follow along

Major developments will be posted here.

You can also follow the project at github.com/hexops/mach.

If you like what I’m doing, you can sponsor me on GitHub.

Thanks for reading!

Unicode data file compression: achieving 40-70% reduction over gzip alone

Sat, 03 Jul 2021 00:00:00 +0000

A little story about how writing a domain-specific compression algorithm in a few days can sometimes yield big benefits, why it’s sometimes worth giving it a shot, and how to tell when you should try. Note: this is about Unicode spec data files, not general purpose text compression.

Background
Problem
Investigation
Zig implementation
Results? Better than gzip/brotli; and even better with them!
- Why test with gzip/brotli but not others?
- How complex is the implementation?
Notable mention
Conclusion

Background

Two weeks ago, I began using Ziglyph (“Unicode processing with Zig, and a UTF-8 string type: Zigstr.”) - an awesome library by @jecolon, for grapheme cluster sorting in Zorex, an omnipotent regexp engine.

I don’t personally have any prior experience working with the lower level details of Unicode, or compression algorithms for that matter.

Problem

As I stumbled into the wondrous world that is Unicode text sorting (see also my article: Unicode sorting is hard & why browsers added special emoji matching to regexp) and began using Ziglyph, I came across an issue: the standard Unicode collation algorithm, which Ziglyph implements, depends on some large Unicode data tables for normalization and sort keys - even gzipped these were fairly large:

hexops-mac:zorex emidoots$ du -sh asset/*
308K asset/uca-allkeys.txt.gz
260K asset/ucd-UnicodeData.txt.gz

These file sizes may seem small, but one of my goals is to make Zorex a real competitor to e.g. a browser’s native regexp engine. That’s challenging because WebAssembly bundle sizes matter a lot in that context, and using the browser’s regexp implementation is virtually free.

Investigation

I set out to try and reduce the size of these data files. First I opened an issue and asked if anyone else had thoughts around reducing the size of this data. The author of Ziglyph @jecolon is awesome and readily had some ideas and was able to reduce the two files substantially by removing unnecessary data (such as comments, etc.)

Curious how much further we could go, I kept squinting at the data files (warning: large):

Binary encoding?

My first thoughts were that a binary encoding would likely reduce the size a lot. I pulled in some help from Hobbyist reverse engineer @Andoryuuta and he got started on a binary encoding for UnicodeData.txt based on the spec. With that, he was able to reduce the original 1.9M allkeys.txt file down to 250K (125K gzipped) - quite a win.

Differential encoding/compression?

My secondary thought was that, scrolling through these data files it was obvious most entries were derived from prior entries. Many entries were long runs of data where the next entry had the same value, plus a small increment. For example, at the start of the allkeys.txt file:

0000 ; [.0000.0000.0000] # NULL (in ISO 6429)
0001 ; [.0000.0000.0000] # START OF HEADING (in ISO 6429)
0002 ; [.0000.0000.0000] # START OF TEXT (in ISO 6429)
0003 ; [.0000.0000.0000] # END OF TEXT (in ISO 6429)
0004 ; [.0000.0000.0000] # END OF TRANSMISSION (in ISO 6429)
0005 ; [.0000.0000.0000] # ENQUIRY (in ISO 6429)
0006 ; [.0000.0000.0000] # ACKNOWLEDGE (in ISO 6429)
0007 ; [.0000.0000.0000] # BELL (in ISO 6429)
0008 ; [.0000.0000.0000] # BACKSPACE (in ISO 6429)
000E ; [.0000.0000.0000] # SHIFT OUT (in ISO 6429)
000F ; [.0000.0000.0000] # SHIFT IN (in ISO 6429)

Of course, not all sections are so sequential. Many sections are a bit more arbitrary:

FF9A ; [.4304.0020.0012] # HALFWIDTH KATAKANA LETTER RE
32F9 ; [.4304.0020.0013] # CIRCLED KATAKANA RE
3355 ; [.4304.0020.001C][.42FB.0020.001C] # SQUARE REMU
3356 ; [.4304.0020.001C][.430A.0020.001C][.42EE.0020.001C][.42E3.0020.001C][.0000.0037.001C][.430A.0020.001C] # SQUARE RENTOGEN
308D ; [.4305.0020.000E] # HIRAGANA LETTER RO
31FF ; [.4305.0020.000F] # KATAKANA LETTER SMALL RO
30ED ; [.4305.0020.0011] # KATAKANA LETTER RO

Still, there are obvious patterns one can see in the way these values change.

Go implementation

I did a quick hacky Go implementation of differential encoding on these files to see how well that would work. The results were pretty good, and already beat just gzip -9 compression of the files:

File	Original	Original + `gzip -9`	My compression	My compression + `gzip -9`
Decompositions.txt	72K	28K	48K	12K
allkeys-minimal.txt	500K	148K	204K	36K

However, because I chose to do these experiments in Go I found a number of inefficiencies:

There were a lot of locations where I encoded things as 8-bit unsigned integers (Go’s smallest value type) instead of a more optimal 4-bit unsigned integer. I could’ve done bit shifting, but it would’ve been annoying.
There were also many places where I encoded Unicode codepoints as 32-bit unsigned integers, rather than a more optimal 21-bit unsigned integer (because valid Unicode codepoints do not exceed that range.)

For a real implementation, I switched over to Zig.

Zig implementation

Actually, two things made working on this in Zig much easier than in Go:

Zig has variable bit-width integers: I could just write u4 and u21 values instead of needing to handle bit packing within larger size integers myself. That was nice.
In the Zig standard library it provides:

With these two features, it became incredibly easy to write the most optimal bit-packed encoding of the data.

In fact, the basic uncompressed binary format was only a few lines to encode:

pub fn compressTo(self: *DecompFile, writer: anytype) !void {
 var buf_writer = std.io.bufferedWriter(writer);
 var out = std.io.bitWriter(.Little, buf_writer.writer());
 try out.writeBits(@intCast(u16, self.entries.items.len), 16);
 while (self.next()) |entry| {
 try out.writeBits(entry.key_len, 3);
 _ = try out.write(entry.key[0..entry.key_len]);
 try out.writeBits(@enumToInt(entry.value.form), @bitSizeOf(Form));
 try out.writeBits(entry.value.len, 5);
 for (entry.value.seq[0..entry.value.len]) |s| try out.writeBits(s, 21);
 }
 try out.flushBits();
 try buf_writer.flush();
}

Differential encoding state machine

To handle the compression, I started out really simple. First I encoded just a binary version of the data with no compression. The most important thing was to get to a point where I could start testing some theories about what would compress the data really well, and validate that it was in fact being losslessly compressed/decompressed without issues via tests:

test "compression_is_lossless" {
 const allocator = testing.allocator;

 // Compress UnicodeData.txt -> Decompositions.bin
 var file = try parseFile(allocator, "src/data/ucd/UnicodeData.txt");
 defer file.deinit();
 try file.compressToFile("src/data/ucd/Decompositions.bin");

 // Reset the raw file iterator.
 file.iter = 0;

 // Decompress the file.
 var decompressed = try decompressFile(allocator, "src/data/ucd/Decompositions.bin");
 defer decompressed.deinit();
 while (file.next()) |expected| {
 var actual = decompressed.next().?;
 try testing.expectEqual(expected, actual);
 }
}

A stream of op codes

I settled on a really simple idea: these data files all have basically just a variable number of integers per line. And if I kept “registers” representing the current value for each integer, I could determine the difference between the past line and the subsequent one to produce a difference. If I encoded that difference as a stream of opcodes with associative data, then to decompress the file I could simply “replay” those operations based on the opcodes and then iteratively come up with more finely-specified, specific opcodes to handle specific types of data.

I started out simple, really just with two opcodes:

// A UDDC opcode for a decomposition file.
const Opcode = enum(u4) {
 // Sets all the register values with no compression.
 set,

 // Denotes the end of the opcode stream. This is so that we don't need to encode the total
 // number of opcodes in the stream up front (note also the file is bit packed: there may be
 // a few remaining zero bits at the end as padding so we need an EOF opcode rather than say
 // catching the actual file read EOF.)
 eof,
};

Using these two opcodes, I was able to effectively encode the entire file. The set opcode had some associative data which effectively expressed an entire raw, uncompressed entry in the file (one line.) This increased the file size since it was effectively just adding 4 bits (the opcode) as additional overhead.

Iteratively finding the most lucrative opcodes

To find the most lucrative (i.e. compressed) opcodes, I printed the data I would associate with an opcode (like set) and then looked for repetitions. Sometimes manually, and sometimes by e.g. piping data to a combination of sort|uniq -c|sort -r to find common patterns.

Since I was printing differences between e.g. the current value and previous value, it was really easy to find common patterns that appeared in the file very frequently, such as specific fields incrementing by specific amounts with one field being arbitrary:

 // increments key[3] += 1; sets value.seq[0]; emits an entry.
 // 1685 instances
 increment_key_3_and_set_value_seq_0_and_emit,

Once I had narrowed down to a larger group of opcodes that more specifically represented the data, I was able to print the number of bits required to store the change in specific fields (like value.seq[0]) and add even more specific opcodes to use variable bit widths:

 // increments key[3] += 1; sets value.seq[0]; emits an entry.
 // 1685 instances
 increment_key_3_and_set_value_seq_0_2bit_and_emit, // 978 instances, 2323 byte reduction
 increment_key_3_and_set_value_seq_0_8bit_and_emit, // 269 instances, 437 byte reduction
 increment_key_3_and_set_value_seq_0_21bit_and_emit, // 438 instances

It being a stream of opcodes was quite nice, because it allowed me to determine how much space was being consumed by a given opcode in sum and target further reducing the size of opcodes that took up the most space. It also made it really easy to find opcodes that I though might help, but in practice turned out to not be that frequent. Just print them, pipe to sort|uniq -c|sort -r to count them - and remove the lowest hanging fruit.

A stream of opcodes for a state machine: a natural progression from a binary format?

I chose an opcode stream for a reason: so that I could encode some complex logic in the form of a state machine. This came in handy for the allkeys.txt file in specific, as it allowed me to introduce incrementors into the mix which would increment register values by a chosen amount each iteration (value “emission”).

The final opcodes for the allkeys.txt file ended up being:

// A UDDC opcode for an allkeys file.
const Opcode = enum(u3) {
 // Sets an incrementor for the key register, incrementing the key by this much on each emission.
 // 10690 instances, 13,480.5 bytes
 inc_key,

 // Sets an incrementor for the value register, incrementing the value by this much on each emission.
 // 7668 instances, 62,970 bytes
 inc_value,

 // Emits a single value.
 // 31001 instances, 15,500.5 bytes
 emit_1,
 emit_2,
 emit_4,
 emit_8,
 emit_32,

 // Denotes the end of the opcode stream. This is so that we don't need to encode the total
 // number of opcodes in the stream up front (note also the file is bit packed: there may be
 // a few remaining zero bits at the end as padding so we need an EOF opcode rather than say
 // catching the actual file read EOF.)
 eof,
};

This meant I could determine the difference in the key and value fields (what those actually are isn’t important, just that they are all minor incremental differences on the prior entry in the file) - set an incrementor to do some work on each emission, such as say increment the key array by [0, 1, 5] each emission, and then say “now emit_32 values!”.

Suddenly, instead of encoding 32 key entries (32 * 3 key values * 21 bits) I am just setting an incrementor (3 key values * 21 bits) and a single opcode to emit 32 values (3 bits).

Overall, this gave me a very nice, natural-feeling progression from a “raw binary format” to something a bit more specific - a bit more compressed.

Results? Better than gzip/brotli; and even better with them!

For lack of better words, I’ll call my compression algorithm here Unicode Data Differential Compression, since it’s differential and specifically for the Unicode data table files - or UDDC for short.

The two files went from the original 568K (with gzip) down to just 61K (with UDDC+gzip). With this, we are able to equal or match both gzip -9 and brotli -9 on their own, AND when combined with gzip or brotli we are able to reduce by 40-70%:

File	Before (bytes)	After (bytes)	Change
`Decompositions.bin`	48,242	19,072	-60.5% (-29,170 bytes)
`Decompositions.bin.br`	24,411	14,783	-39.4% (-9,628 bytes)
`Decompositions.bin.gz`	30,931	15,670	-49.34% (15,261 bytes)

`allkeys.bin`	373,719	100,907	-73.0% (-272,812 bytes)
`allkeys.bin.br`	108,982	44,860	-58.8% (-64,122 bytes)
`allkeys.bin.gz`	163,237	46,996	-71.2% (-116,241 bytes)

Before represents binary format without UDDC compression.
After represents binary format with UDDC compression.
.br represents brotli -9 <file> compression
.gz represents gzip -9 <file> compression

Why test with gzip/brotli but not others?

I chose to compare against gzip/brotli specifically because you get those effectively for free in WebAssembly: browsers already know how to decompress those and ship with gzip/brotli decompressors - so you can use them for free without shipping any additional code.

How complex is the implementation?

The final implementation for both files is only a few hundred lines (excluding blank lines, comments, and tests):

AllKeysFile.zig: 298 lines
DecompFile.zig 336 lines

I have not measured produced machine code size yet, but suspect it is relatively negligible compared to the gains.

Notable mention

I should mention that the Unicode spec, as @jecolon pointed out to me, does suggest ways to reduce sort key lengths and implement Run-length Compression:

I wasn’t able to locate an implementation of this (I’d be curious to compare results!) but suspect that, as the run-length compression does not fit the data as tightly, it would not compress quite as well (although would handle any major changes to the type of data in the files without requiring compression algorithm changes better.)

Also of note is that their algorithm only seems to be mentioned in the context of allkeys.txt / the Unicode Collation Algorithm, not in the context of normalization/decompositions from UnicodeData.txt.

Conclusion

Ask questions, stay curious, don’t be afraid to experiment even if it’s outside of your domain of expertise. You might surprise yourself and find something interesting, challenging, and worthwhile.

Unicode sorting is hard & why browsers added special emoji matching to regexp

Sun, 27 Jun 2021 00:00:00 +0000

As I work on Zorex, an omnipotent regexp engine I have stumbled into a world of tales about why Unicode text sorting is so annoying in the modern day. Let’s talk about that.

Why ASCII sorting is not enough
Twitter’s emoji problem - or when Unicode locale-aware sorting Really Matters™
Browsers added special emoji matching to regexp
Language comparison
- JavaScript Collator sorting is not guaranteed across browsers
Go sort.Strings is not locale aware
- Rust Vec sorting is not locale aware
- Swift’s default is not locale aware, but unicode support is notable
Zig’s ziglyph package
Why is localized text sorting hard?
WebAssembly may make things worse?

Why ASCII sorting is not enough

Perhaps you are sorting strings in JavaScript like this:

const words = ['Bears', 'Beetle', 'kiss', 'Similar', 'Apples'];
words.sort();
// [ "Apples", "Bears", "Beetle", "Similar", "kiss" ]

And that works pretty well, until someone translates it to German:

const words = ['Bären', 'Käfer', 'küssen', 'Ähnlich', 'Äpfel'];
words.sort();
// [ "Bären", "Käfer", "küssen", "Ähnlich", "Äpfel" ]

The preferred alphabetical sorting would be [ "Ähnlich", "Äpfel", "Bären", "Käfer", "küssen" ] - Array.sort doesn’t do that.

That is because it is sorting lexicographically by byte values in the string, and not taking into account locales.

Twitter’s emoji problem - or when Unicode locale-aware sorting Really Matters™

Twitter is no stranger to issues with emojis, but have you ever thought about how they check if a hashtag contains only legal characters and emojis? Regexp, of course!

You might think one could just use a regexp unicode character class, like [\u{1f300}-\u{1f5ff}] - but that only covers a single codepoint! Emojis and other text rely on combining multiple Unicode codepoints to compose grapheme clusters - and often what we see as a single visible character on our screen.

The full regexp needed to match all emojis with codepoints would be:

(?:\ud83e\uddd1\ud83c\udffb\u200d\u2764\ufe0f\u200d\ud83d\udc8b\u200d\ud83e\uddd1\ud83c\udffc|\ud83e\uddd1\ud83c\udffb\u200d\u2764\ufe0f\u200d\ud83d
[102,816 characters omitted]

For your sake, I’ve omitted the other 102,816 characters of that regexp. You can view it here: https://regex101.com/r/2ia4m2/7

Browsers added special emoji matching to regexp

Luckily for Twitter and others, ECMAScript’s TC39 proposal a few years back extended the regexp engine to support Unicode property escapes for emojis and a few other Unicode properties so you can write e.g.:

\p{Emoji_Presentation}

Without packing several thousand bytes of Unicode data tables or regexp into your JS bundle.

Language comparison

As Daniel Lemire said: sorting strings is stupidly hard.

JavaScript Collator sorting is not guaranteed across browsers

You may have found browser’s String.prototype.localCompare or Intl.Collator and they DO fix the issue[1]:

const words = ['Bären', 'Käfer', 'küssen', 'Ähnlich', 'Äpfel'];
words.sort(Intl.Collator().compare);
// [ "Ähnlich", "Äpfel", "Bären", "Käfer", "küssen" ]

(note, however, you may wish to use Intl.Collator('de').compare instead to sort according to German language customs)

However, beware that if you look at the ECMA spec for this you will find:

It is recommended that the CompareStrings abstract operation be implemented following Unicode Technical Standard 10, Unicode Collation Algorithm […]

Applications should not assume that the behaviour of the CompareStrings abstract operation for Collator instances with the same resolved options will remain the same for different versions of the same implementation.

Although many browsers may produce similar sorting results - not all will. For one thing, not all locales are available across browsers.

Further, different browsers may choose to sort things differently. For example IE 11 sorting “co-op” after “coop” while other browsers do the opposite.[2]

Go sort.Strings is not locale aware

It may be interesting to note that Go’s sort.Strings operates on byte comparisons, and has the same issue as JavaScript’s Array.prototype.sort:

words := []string{"Bären", "Käfer", "küssen", "Ähnlich", "Äpfel"}
sort.Strings(words)
// [Bären Käfer küssen Ähnlich Äpfel]

One can easily perform unicode code point (rune) sorting in Go, which would fix the above example - but note that rune sorting is not locale-aware, and importantly that a Go rune is not the same as a visible character and would not take into account grapheme clusters.

For proper Unicode locale-aware sorting in Go, you need to use the Unicode Collation Algorithm via golang.org/x/text/collate but be sure to also apply normalization to your text first via golang.org/x/text/unicode/norm

Rust Vec sorting is not locale aware

A Rust Vec of strings implements sorting[3] lexicographically by their byte values, consistent with Go’s sort.Strings and JavaScripts Array.prototype.sort:

let mut vec = Vec::new();
vec.push("Bären");
vec.push("Käfer");
vec.push("küssen");
vec.push("Ähnlich");
vec.sort("Äpfel");
println!("{:?}", vec);
// ["Bären", "Käfer", "küssen", "Ähnlich", "Äpfel"]

Locale-aware sorting in Rust is provided by ICU4C bindings by Google, google/rust_icu (note however, there have been a number of vulnerabilities in the ICU4C library) and there is ongoing work to implement internationalization in pure Rust as a safer alternative: unicode-org/icu4x.

Swift’s default is not locale aware, but unicode support is notable

Swift remains consistent with other languages in sorting strings lexicographically by byte value:

var words = ["Bären", "Käfer", "küssen", "Ähnlich", "Äpfel"]
words.sort()
// ["Bären", "Käfer", "küssen", "Ähnlich", "Äpfel"]

However, it is notable that Swift includes locale sensitive sorting out of the box[4]:

var words = ["Bären", "Käfer", "küssen", "Ähnlich", "Äpfel"]
let sorted = words.sorted { (lhs: String, rhs: String) -> Bool in
 return lhs.localizedStandardCompare(rhs) == .orderedAscending
}
// ["Ähnlich", "Äpfel", "Bären", "Käfer", "küssen"]

It also seems quite notable just how very unicode-aware the Swift documentation is on their String type. Other languages could learn a thing or two here in educating developers.

Zig’s ziglyph package

Zig’s standard library is still quite under development, however it seems likely that major unicode functionality will be outside the stdlib.

Luckily, @jecolon in the Zig community is working on an excellent package for this: ziglyph.

I mention this because I’m a fan of the language and have recently begun contributing to that package; but otherwise Zig isn’t any different than other languages listed here aside from there being no real “default” way to sort strings from what I know.

Why is localized text sorting hard?

I believe there are a combination of factors at play:

Most languages leave Unicode locale-aware text sorting as an afterthought.
Most developers don’t care enough to use Unicode, let alone implement locale-aware text sorting. Internationalization is always “that thing we’ll do if somebody complains” or an afterthought.
It’s hard. It wasn’t until recently that we got semi-decent support for it across browsers, and what is there still leaves a lot to be desired.
Many are still running into dated software, like NodeJS versions from ~2019 ish that didn’t have full ICU support on by default.

WebAssembly may make things worse?

As a closing thought, I just want to hint at why I think WebAssembly will make things worse before they get better.

Whether your application is in Go and has it’s own Unicode Collation Algorithm (UCA) implementation, or Rust and uses bindings to the popular ICU4C library - one thing is going to remain true: it requires large data files to work.

The UCA algorithm depends on two quite large data table files to work:

UnicodeData.txt for normalization, a step required before sorting can take place.
allkeys.txt for weighting certain text above others.
And more, if you want truly locale-aware sorting and not just “the default” the UCA algorithm gives you.

Together, these files can add up to over a half a megabyte.

While WASM languages could shell out to JavaScript browser APIs for collation, I suspect they won’t due to the lack of guarantees around those APIs.

A more likely scenario is languages continuing to leave locale-aware sorting as an optional, opt-in feature - that also makes your application larger.

I think this a worthwhile problem to solve, so I am working on compression algorithms for these files specifically in Zig to reduce them to only a few tens of kilobytes.

My game development journey & why I'm increasing my contribution to Zig to $200/mo

Sat, 10 Apr 2021 00:00:00 +0000

Today, I increased my monthly donation to Zig to $200 a month. Before Zig, I have not contributed financially to any open source project.

Before I can explain why I am so extremely excited about the Zig programming language and its community, I need to explain where I come from.

I grew up playing Linux games like Mania Drive
It wasn’t long before I found that the Mania Drive game engine was open-source.
I was so infatuated with this game engine, I convinced my dad’s coworkers to pay me to build them a virtual meeting world
But the game kept crashing at random, and I had no idea why
Panda3D: Disney’s Python/C++ game engine
The Panda3D game engine opened new doors for me
I began to prevail
But my limited knowledge hit me again
Learning C++
Learning Go, writing my own game engine
My game engine appeared on Hacker News (2014)
Joining Sourcegraph
Six and a half years later, I’m still at Sourcegraph.
But I’m still a game developer at heart
C was easier for me as a beginner than C++
Unity is the new Flash
Why do we encourage building, but not understanding?
One language to write your game and engine in
Looking for the one language to rule them all
- Could Rust be it?
- Could the V language be it?
- Could I build it?
Discovering Zig
- Learning Zig
- Working in it
- The community is incredible
- My commitment to Zig

I grew up playing Linux games like Mania Drive

Mania Drive was an open-source clone of the popular Trackmania series. Me and my siblings in our early teens easily spent hundreds, if not thousands, of hours in Mania Drive.

In retrospect it has quite bad graphics, physics, game-play mechanics, etc. But it was customizable! There was a simple tile-based level editor. We would spend days building the most confusing, crazy, impossible maps to beat so we could challenge each other. We would play it all night.

Obsession over this game led to even more modding: the discovery of Blender meant we could create even more custom maps than in the limited tile-based map editor. Although the Blender UI was pretty rough back then:

It wasn’t long before I found that the Mania Drive game engine was open-source.

Raydium, the C game engine behind Mania Drive, is still around today - one of the beauties of open source software! At the time, the things about it that just blew my mind were:

It supported scripting through PHP! I had used PHP a lot with LAMP stacks, so the idea that I could script the engine in PHP was mind blowing to now 14-year old me.
2 years later, I got an iPod touch and the Raydium developers had just posted a demo video showing the engine running on the iPhone. 16 year old me thought this was literally the coolest thing ever, albeit immensely disappointed I did not have a Mac to build it for my iPod:

I was so infatuated with this game engine, I convinced my dad’s coworkers to pay me to build them a virtual meeting world

My dad was running one of his many startups at the time - it had some momentum behind it, basically a platform like Ebay but for selling services instead of goods. Several of his work friends were funding it with significant amounts of their own money.

Unfortunately for them, they spent most of their focus on business operations than actually getting a product out the door. Lucky for me, however, this meant they had came across Sun’s Project Wonderland - the delightfully terrible 3D virtual workplace of the future (or so Sun thought, before they had to sell to Oracle.) It was terrible, barely a good demo:

It required something like 32 CPUs and 64G of memory to run the server for just 8 players - no small feat back in 2010! The client was laggy, there were virtual whiteboards you could draw on but everything was slow. Even its VOIP feature was glitchy - although quite novel at the time. It was all around a terrible experience.

16-year-old me convinced my dad and his coworkers to instead pay me to build them a better version: one using Raydium, C - and PHP.

It wasn’t long before I had some amateur copy of Project Wonderland - ironically better than Sun’s in many aspects - and surely worse in others. It even had a client auto-updater built with wxWidgets and Python (it just shelled out to an svn client to download the latest copy of the game, hah!)

But the game kept crashing at random, and I had no idea why

The truth was literally a 16-year old script kiddy copying and pasting C code from various demos of Raydium, without a care in the world for freeing memory or avoiding stack corruption.

// Don't remove this print statement. Game will crash!

It was around this time that I began to really get into Python: it was simple, something I could really wrap my brain around, and it was powerful. I stumbled into cython and wrote OpenGL bindings - this time with more appreciation for memory management.

Panda3D: Disney’s Python/C++ game engine

Panda3D was the game engine Disney used to create Toontown Online and Pirates of the Carribean Online:

It was written in C++, with automatic binding generation for Python. In fact, many portions of the engine were written in just Python and not usable from C++ at all. They revamped their website site recently, so I guess it’s still around.

The Panda3D game engine opened new doors for me

Discovering Panda3D opened new doors for me. At around 16-17 years old now, I was able to really get my first real taste of game development: I could write games in this – in Python – and they wouldn’t crash in ways that I couldn’t understand.

Pretty soon, I had actual games in the works. I was starting to learn about why draw order matters - and how I had no understanding of mip-mapping:

I began to prevail

At this point I had several, actually working games - I was proud of what I was working on, had multiplayer functionality hooked up to a MySQL database even.

But my limited knowledge hit me again

For my game, I wanted nothing more than for my friends to be able to chat with me using a chat box. The problem was, Panda3D’s Python GUI library, DirectGUI, was just too slow at rendering text.

I tried everything I could, and even got to the point where I was asking on the forums if it was possible to draw a TextNode with multi-threading:

Calls to TextNode.generate() are very expensive.

Is there a way for Panda to run all TextNode.generate() calls in a seperate thread? I’ve attempted doing it on my own using direct.stdpy.threading.Thread, only it causes dead locks, I would guess this is to my own lack of knowledge.

could anyone help me?

I didn’t get a response. I couldn’t solve the issue. “I can’t add a chat box to my games” became a problem I could not solve.

Learning C++

I was at a point where I had rewritten most of Panda3D’s UI components myself in Python (mind you, theirs are written in Python - you cannot use them from C++. I don’t know why I did this.)

But I still needed a way to render text. I needed a way to make TextNode.generate() faster. Little did I know at the time that it was generating geometry from freetype and creating one draw call per text drawn - which is super slow, and did not help my naive usage of its API!

The harsh reality was that I didn’t have anybody to teach me. I spent months trying to learn C++, but it is a beast (and “Disney game engine C++” is, of course, a flavor of C++ not found in books.) It wasn’t something I could handle as 16-year-old kid without any real knowledge of low level languages.

In trying to learn C++, something became painfully clear to me:

Having part of my application written in Python and part of it written in C++, two very different languages, was only great until I realized I had to dive into this large C++ code base and had no knowledge of it.

I gave up.

Learning Go, writing my own game engine

When Google announced Go, I heard about it very early on. At this time, they were still advertising it as a low-level systems language, an alternative to C, a better C. But more forgiving, because it had a garbage collector.

Coming from a predominantly Python background at the time, this sounded incredible to me: I could write a game engine in this and understand my code end to end and make sure there is no single piece that I do not understand.

I spent the next 4 years of my life, almost 100% full-time working on Azul3D, a game engine in Go - and spent only minimal time attending online community college on the weekends.

There was so much that I learned during this time, about software engineering, game engines, audio, input, math, image and audio codecs, blender plugins, file formats, physics, and working with other people (some cool things like a NES emulator came out of that)

I learned an immense amount, but I had nothing to show for it aside from a funny looking website and some quite poor screenshots (to the dismay of every person I told.)

My game engine appeared on Hacker News (2014)

Someone posted it on Hacker News, which was both exciting but also extremely depressing for me at the time. I took the feedback as statements that what I was doing was wrong, rather than as feedback about how to improve:

The web site looks cool, but it sets off a whole bunch of red flags for me.

the go programming language is not very suitable for games at all.

if you truly need a performant graphics engine, it’s going to be either C++, C or Rust anyway.

Azul3D is for programmers and doesn’t provide GUI-editors.

So, you write your levels using a text editor? That’s not for programmers, that’s for people who hate themselves.

No screenshots of the game at all?

Garbage collector FAQ isn’t necessarily reassuring, since it seems to say “go through the same hoops other GC gaming platforms push you through”. Obligatory Rust gaming comment goes here.

in Rust you have code without GC, but the compiler makes sure that everything is freed.

I learned so much from this interaction:

Being transparent about project status is important.
I shouldn’t have “hidden” screenshots of the project. I was worried people would judge what the engine is capable of based on bad programmer artwork: instead, they judged it for having none.
I should’ve talked about the interesting parts more:
- Did you know there is a D* lite pathfinding algorithm that was used in one of the Mars rovers, is super simple, and handles dynamic terrains? Much nicer than A* and other variants.
- What my vision for a game engine deeply integrated with Blender, and developer-first, would look like in practice.
I frankly shouldn’t have cared so much. I thought what I was doing was awesome, and I let others’ viewpoints affect my own view of my work negatively.

Joining Sourcegraph

It was around this time that I was basically deciding: what would I do for a living?

Luckily, someone in the Go community (whom I’d never talked to before) reached out to me and asked “hey, what are you doing?” - I told them I was in school, and left out the part where I was a college student living with parents, scraping by, and likely going grocery-store-part-time-job seeking soon.

I didn’t come from a background that would lead me to believe I could make a living programming in Go, to the contrary my parents often warned me I couldn’t and that I would need to go into Cisco network infrastructure instead.

I was told in blunt terms, I could scrape by doing what I love - or make a killing doing something I hate. My parents were mechanical engineers at aerospace companies (I’ll let you guess which path they took.)

Bill’s short ~20 minute conversation with me, quite literally changed my life in ways I couldn’t have imagined. I often think about where I would be today had he not reached out to me, and I never quite knew how to reach back out and say thank you in a way that was as meaningful to him as what he did was for me.

Six and a half years later, I’m still at Sourcegraph.

I’ve learned so much about startups, being a good engineer, management, business operations, cloud infrastructure, teamwork, communication, and so much more in the last six years I’ve spent at Soucegraph. There are so many stories I have, and so many great people I have had the opportunity to work with because of it.

We grew from awkward little startup without a clear product, a tiny team, an uncertain future - into a sprawling metropolis with massive amounts of happy users, customers, $50m i series C funding, and have grown the team to over a hundred people all over the world. I have played a key role in that, and continue to this day.

A passion for making games as a kid, turned into a passion for making developer tools all around better. I still have much to do here.

But I’m still a game developer at heart

If there’s one thing I return to regularly, consistently, and frequently despite working a demanding job at a startup - it’s game development. And you’re going to hear a lot more about that soon.

Since March of last year, I began basically working two jobs: every day after I sign off from work at Sourcegraph, I spend around 8 hours working on game development.

I am more determined than ever before, and success or fail - I will try.

C was easier for me as a beginner than C++

Hacking together games in Raydium’s C API taught me that C is hard, but also showed me in retrospect that if I had just a little bit more guidance, If C was just slightly easier, if I only knew the tricks of how to debug C programs: I would have been immensely successful in working with it.

With Panda3D, writing some decent games in its Python API only to later find I needed to dive into this magical box of a complex C++ core made me believe that:

C++ is less beginner friendly than C. One major reason for this is due to the different C++ dialects: you’re not going to understand Panda3D C++, or Unreal C++ - by going and reading books about the language or taking a class. They create their own dialects through the language. Today with different C++ versions, even the textbooks and classes you find will be using different dialects.
There are not good tutorials or explanations online about how game engines work and why. I regularly find that very experienced software engineers and even people who work in Unity or Unreal regularly, simply do not have a decent grasp of how game engines work. “What do you mean polygon count is not very important?!” are among the most basic questions that arise, with modern game engines abstracting away so many bits that your average developer merely says:

“Game engines are just magical ultra-complex things I could never even begin to understand! Only the professional AAA studios and god programmers like Jonathan Blow should even try to do that!”

I do not subscribe to this belief - and believe that most game developers have been robbed of the proper end-to-end understanding of game engines they deserve.

Unity is the new Flash

You, dear reader, do not understand just how far the bar for game development has been lowered.

Putting together a game in Unity is so beyond ridiculously easy today with Unity that it is incredible, the game engine is truly the new Adobe Flash equivalent.

You could pick up that engine today, and have a silly little game you yourself put together the next.

Of course, with Unity, comes large problems for serious game developers:

There are so many people hacking together Unity games that the quality of the information out there is quite bad.
The quality of what is on the Unity asset store is quite bad.
Unity encourages hacking things together to get a quick demo running - and it shows. Game developers hide the fact that they use Unity, because it has such a negative connotation with players that Unity == low quality.

Why do we encourage building, but not understanding?

Game engines today are the epitome of large complex code-bases:

The people and companies working on them value features over quality.
When there is a major issue, there are few people with an understanding of it to be found.
Teaching people how to write good software is hard - and that’s our customer base (I imagine Unity/Unreal say) - far easier to give them something akin to a scripting language. It’s good even if our users don’t understand how all of this works.

Teaching is hard, but if done right is invaluable. There is a reason NeHe Productions’ OpenGL tutorials are still revered today: they are incremental, and teach in the form of building blocks on top of what you previously learned.

There’s a reason many AAA studios simply throw out everything and start from scratch when working on their next title.

We encourage building new things, but not understanding existing things.

One language to write your game and engine in

Scripting languages for game engines stem from multiple desires - the most common being some variant of:

C++ is hard, but we need it for performance.
My level designers can’t write C++ code!
I cannot understand C++, but do know C#/Python/Java/etc.

A lot of people have a terrible experience from school where they were taught C or C++, had absolutely no understanding of what was going on - and were told “This is programming!”

I believe that in general, writing your game in a different language than the engine (Unity’s C#/C++ core model, Panda3D’s Python/C++ core model, and yes - perhaps even Unreal’s Blueprints/C++ core model - which I admit is the better of the three)

Pictured: The Unreal character controller blueprint for a game called Diacrisis.

Whether you have good code or bad code, good blueprints or bad blueprints - the truth is that having one part of your application in a completely different language creates a significant barrier to learning. I believe that is a bad thing, and the long-term costs outweigh the benefits.

Looking for the one language to rule them all

Could Rust be it?

Initially, I spent a substantial amount of time considering Rust as that language. It’s offer of memory safety guarantees is extremely compelling to me.

I even convinced us to adopt Rust at Sourcegraph in some form, our syntax highlighter is a little Rust HTTP server that was basically write-and-forget. We haven’t maintained it at all, and it’s held up pretty well for over 5 years.

But maintaining it has been brutal. We mostly have Go developers there, and despite a strong desire from many of them to learn Rust really none of them have been able to successfully dive into the codebase and get started.

Rust’s learning curve is steep. Steeper than C++ in my view, and definitely steeper than C (despite its many, massive flaws.)

I spent upwards of 6 months on-and-off trying to become proficient at writing Rust code, and I never really became productive: regularly stumbling across complex issues in downstream dependencies (often used by everyone, but maintained by no one in the rust-lang-nursery.)

I love the idea of Rust. I love what it promises. And I kept going back to it on-and-off for over 6 months because I truly wanted to be able to be productive in it.

It didn’t work. “I’m just not smart enough to use this language” I often thought. And I fear this will be the takeaway of many who hear the promise of the language, only to discover another “I took a C++ class in school and it was terrible” experience, leading so many more developers to conclude “I’m not good enough for low-level programming, I should learn JavaScript instead”.

Could the V language be it?

UPDATE: The V language author reached out and it would seem my memory was faulty about what happened here, this was due to a misunderstanding almost 100% on my side and I have falsely mis-characterized the V community here as being less friendly then they were in practice, and I am deeply sorry for that.

I believe my criticisms below about the controversy surrounding the project and the secretive nature when it launched are still valid, and were ultimately major factors in why I chose to not further consider it.

At the same time, I want to point out that V does not look the same as when it launched - and anybody who like me left due to those issues may do well to reconsider it today as the project and details surrounding it appear to have changed substantially.

What this section originally said was:

When I heard about the V programming language, it seemed right on the spot.

I immediately jumped into the community to chat with the author, despite the controversy surrounding it - and tried to get more info about it, how he was thinking of the language, etc.

I asked if there were plans to support raw multi-line string literals, like Go. I was struck by a firm ‘No. Go doesn’t have raw string literals either." - it was the unfriendly community I came across, the controversy surrounding it, and the secretive nature of the project (“I have this, but I’m not going to share it yet”) that made me lose faith in its promise.

This wasn’t a language whose community I could join and contribute to.

Could I build it?

When the COVID-19 pandemic first hit, I thought to myself:

If Go isn’t it, Rust isn’t it, the V language isn’t it - could I build it? Could I create the “better C” I am looking for? What would it look like?

4 months later, I had a pretty good picture. I had an early stages compiler for the language in Go using LLVM, and knew what I wanted in a “better C”. There was a long road ahead, but I had a picture of it. Until..

cat spills coffee on $2800 laptop, frying SSD with ~4 months of uncommitted work on EBNF parser generators yeah.. no, that’s.. that’s okay, I wanted to rewrite all of that code. Yeah. This is fine.

— Emi (@emidoots), May 27, 2020

Obviously, I was an idiot and should’ve just git pushd my code - or backed up my laptop - but nonetheless this was a setback.

Discovering Zig

I continued to look for this mythical “better C” - and one name that kept arising in my sphere was Zig.

I didn’t pay much attention to it, until I shared it with my brother for the 3rd time:

“…I already shared this with you?”

“I am really excited about this. It’s literally the language I was trying to build before I think”

Learning Zig

In trying to learn Zig, there were two things that struck me:

I could be productive in Zig right away. Transitioning from Go at work to Zig after-hours every day was easy.
The community was so friendly, inviting, and helpful in answering my questions.
I continuously saw a theme of “this is a decentralized community, there is no ‘official’ thing we’ll ever push onto you, we want everyone to contribute and truly be a part of this”

Zig became the first open-source project I had ever contributed to financially.

And if none of the above convinces you, let me tell you the following: @ziglang is the first language I have felt strongly I should try and contribute to, and the ONLY open source project I have ever donated to. No other has been so compelling

— Emi (@emidoots), October 23, 2020

Working in it

Thus far, I’ve worked on two things in Zig:

I continue to work in Zig daily, with no plans to stop - mark my words, this is an amazing language to work in.

The community is incredible

Over time, I watched and read more content from the Zig developers. It’s been beautiful to see:

Them constantly, proactively advocate against zealotry of the language.
Them constantly advocate for new members of the community to actually help others.

Not only that, but I began to notice the Zig foundation actually directly paying open source developers through donations:

ZSF is a small organization and makes efficient use of monetary resources. The plan is to keep it that way, but we do want to turn our unpaid volunteers into paid maintainers to help merge pull requests and make swifter progress towards 1.0. The whole point of ZSF being non-profit is to benefit people. We’re trying to get open source maintainers paid for their time.

(from https://ziglang.org/zsf)

This is such a beautiful thing to see happening, and I hope that other open source communities take lessons from Zig here. The execution here is so important, and so far the Zig community’s execution has been incredible here.

My commitment to Zig

For me, Zig ticks all the boxes of a programming language that could fundamentally upend the way that video games are built for the better.

I want to see it succeed - and make it succeed at exactly that. Today, I raise my monthly contribution on GitHub sponsors to $200/mo. I would encourage anyone reading this to go and find ways to contribute (financially or not) to a vision you believe in.

In addition to the above, I am committed to building the following in Zig:

A game engine for the future
Better developer tools (not just for game developers)
Several real video games, which I believe can be competitive with what AAA studios offer today.

Thanks for reading my journey, and I hope you’ll consider following it in the future.

Zig, Parser Combinators - and Why They're Awesome

Wed, 10 Mar 2021 00:00:00 +0000

In this article we will be exploring what parser combinators are, what runtime parser generation is - why they’re useful, and then walking through a Zig implementation of them.

What are parser combinators?
Why are parser combinators useful?
Going deeper: runtime parser generation
A note about traditional regex engines
Implementing the Parser interface
Our first Parser
Our first parser combinator
Using our OneOf parser combinator
Runtime parser generation
Closing thoughts

What are parser combinators?

A parser parses some text to produce a result:

A parser combinator is a higher-order function which takes parsers as input and produces a new parser as output:

Why are parser combinators useful?

Let’s say we want to parse the syntax which describes a regular expression: a[bc].*abc

We can define some parsers to help us parse this syntax (e.g. into tokens or AST nodes):

Suppose that for a[bc].*abc:

RegexLiteralParser can parse a, b, and c, but not abc (the string.)
RegexRangeOpenParser can parse [.
RegexRangeCloseParser can parse ]
RegexAnyParser can parse the . “any character” syntax.
RegexRepetitionParser can parse the * repetition operator.

Now that we have these parsers, we can define parser combinators to help us parse the full regular expression. First, we need something to parse a string abc which we can define as:

What is OneOrMore, though? That’s our first parser combinator!

It takes a single parser as input (in this case, RegexLiteralParser) and uses it to parse the input one or more times. If it succeeded once, the parser combinator succeeded. Otherwise, it failed to parse anything.

Now if we want to parse the [bc] part of our regex, let’s say it can only contain a literal like bc (of course, real regex allows far more than this) we can e.g. reuse our new RegexStringLiteralParser:

In this case, Sequence is a parser combinator which takes multiple parsers and tries to parse them one-after-the-other in order, requiring all to succeed or failing otherwise.

Building upon this basic idea, we can use parser combinators to build a full regex syntax parser:

Going deeper: runtime parser generation

From before, our parser combinator RegexSyntaxParser is built out of multiple parsers (Regex...Parser) and ultimately produces an AST describing the syntax for a given regex.

We can use the same combinatorial principle here to introduce a new parser generator called RegexParser which uses RegexSyntaxParser to create a brand new parser at runtime that is capable of parsing the actual semantics the regex describes - forming a full regex engine:

A note about traditional regex engines

Revised Mar 10, 2021 to clarify a misunderstanding I had about about the difference between DFA and NFA regex engines. Thanks @burntsushi for helping me to learn!

Production grade regex engines are either finite automata based or backtracking based, and are described in great detail in Russ Cox’s article here and his second article here covering the virtual-machine approach commonly used in regex engines.

It’s worth noting that combinatorial parsing and generating parsers at runtime is very much an uncommon method of implementing a regular expression engine. This is somewhat close to what Comby does in practice, although we use a runtime parser generator instead of parser parser combinators.

One could argue this makes what we’re parsing not strictly regular expressions, although as Larry Wall (author of the Perl programming language) writes, neither are the modern “regexp” pattern matchers you are likely used to:

“Regular expressions” […] are only marginally related to real regular expressions. Nevertheless, the term has grown with the capabilities of our pattern matching engines, so I’m not going to try to fight linguistic necessity here. I will, however, generally call them “regexes” (or “regexen”, when I’m in an Anglo-Saxon mood).

Implementing the Parser interface

Parser combinators tend to be written in higher-level languages with much fancier type-systems such as Haskell and OCaml, which lend themselves well to higher-order functions like parser combinators.

We’ll be implementing this in Zig, which is a new low-level language aiming to be a better C.

Compile-time vs. run-time

Zig has very cool compile-time code execution semantics which help provide its generics. We’ll be exploring these a bit, but since we want to ultimately build parser generators at runtime (in order to execute a regexp) what we’ll be looking at is mostly runtime parser interfaces rather than compile-time parser interfaces (which are very much possible!)

Since we’ll be dealing with heap allocations, our parser will not be able to run at comptime for now. Once Zig gets comptime heap allocations this should be possible and opens up interesting new opportunities.

The parser interface

We need an interface in Zig which describes a parser as we previously mentioned:

Here it is - there’s a lot to unpack here so we’ll walk through it step-by-step:

pub fn Parser(comptime Value: type, comptime Reader: type) type {
 return struct {
 const Self = @This();
 _parse: fn(self: *Self, allocator: *Allocator, src: *Reader) callconv(.Inline) Error!?Value,

 pub fn parse(self: *Self, allocator: *Allocator, src: *Reader) callconv(.Inline) Error!?Value {
 return self._parse(self, allocator, src);
 }
 };
}

Zig generics are provided via type parameters

pub fn Parser(comptime Value: type, comptime Reader: type) type {
 return struct {
 ...
 };
}

This is a Zig function which takes two arbitrary type arguments at comptime, named Value and Reader. Uppercase is used to denote the name of a type in Zig. Thes are:

Value will be the type of the actual value that the parser will produce (e.g. a string of matched text, or an AST note.)
Reader will be the type of the actual source of the raw text to parse (we’ll cover this more later.)

The function itself returns a new type.

What we’re seeing here is the key way in which Zig approaches generic data structures: you merely pass around types as parameters - as if they were values - and you write functions which take types as parameters and return types as values. Some examples of valid calls to this function are:

Parser(u8, []u8) where u8 is an unsigned 8-bit integer and []u8 is a slice of unsigned 8-bit integers.
Parser([]const u8, @TypeOf(reader)) where []const u8 is describing a slice of UTF-8 text (a string) and reader is some reader type, such as std.io.fixedBufferStream("foobar").

Zig runtime interfaces

Now, since we’re trying to define an interface whose actual implementation can be swapped out at runtime - what we need is pretty simple:

A struct type which has the methods we want every implementation to provide.
Those methods to call function pointers which are defined as fields of our struct.

Basically, if someone wants to implement our interface they just need to create a new instance of Parser and populate the fields (callbacks) so their implementation is called when the interface is used.

This is the same pattern used by the Zig std.mem.Allocator interface.

In our case here, the returned struct has a method that consumers of the interface would invoke called parse - and the function pointer field that implementors will set to get a callback is the _parse field:

Type parameters

Let’s look at some of the data types going around here:

A few other notes:

Error!?Value is just describing the function can return an Error OR no value OR a Value type. See Zig’s error union types and optional types.
callconv(.Inline) is just telling the compiler to inline the function call - since our function isn’t doing a ton.

Errors the Parser interface can produce

Our error type might start out looking something like this:

pub const Error = error{
 EndOfStream,
} || std.mem.Allocator.Error;

error{...} describes a set of potential errors and || std.mem.Allocator.Error merely says to merge the allocator type’s error set with ours - so our potential set of errors includes ours and theirs.

As we start performing different operations within parsers, it will become more complex to describe more potential sources of errors:

pub const Error = error{
 EndOfStream,
 Utf8InvalidStartByte,
} || std.fs.File.ReadError
 || std.fs.File.SeekError
 || std.mem.Allocator.Error;

Zig can often infer error sets but only in some contexts today.

Our first Parser

All we need to do in order to implement a Parser is provide the _parse method, and define its return Value type and Reader input type:

const parser: Parser([]u8, @TypeOf(reader)) = .{
 ._parse = myParse,
};

In the above, the type T in const parser: T is denoting the type of the constant named parser - in this case it’ll be the type returned by Parser([]u8, @TypeOf(reader)). And this:

something = .{
 ._parse = myParse,
}

Is the Zig syntax for populating a struct. We’re setting the _parse field to myParse. Zig can infer the type of the struct if you write a .{} instead of T{} - which avoids the need for us to repeat the call to the Parser() function which is verbose.

What actually is a “Reader”?

Up to this point, we’ve just talked about Reader as being any type.

Similar to our Parser interface, the Zig standard library provides a std.io.Reader interface and there are many implementors of it including:

std.fs.File
std.io.fixedBufferStream("foobar")
std.net.Stream (network sockets)

However, in contrast to our Parser type which invokes function pointers at runtime, the std.io.Reader interface is a compile time type - meaning calls to the underlying implementation do not involve a pointer dereference.

Today, Zig is in early stages (version 0.7) and does not have anything like an interface or trait type (although it seems likely this will be improved in the future.)

This means that, for now, we cannot simply define our function as accepting only an std.io.Reader interface - instead we must declare that we accept any type which we’ll call Reader, write our code as if it is an std.io.Reader - and the compiler will just barf if anybody passes something in that isn’t an std.io.Reader. This can sometimes lead to confusing compiler error messages (“there’s an error in the standard library code? Ah, no, I just needed to pass a .reader()!”).

A Parser that parses a literal string

If we want a Parser interface implementation that parses a specific string literal, one way to do that is to also make that a generic function which accepts any reader type (so we’re not restricted to e.g. just file inputs):

pub fn Literal(comptime Reader: type) type {
 return struct {
 // TODO
 };
}

This is pretty good - but we need some way to have the type we return implement the Parser interface we defined. The way to do this is by defining a field in our struct:

pub fn Literal(comptime Reader: type) type {
 return struct {
 parser: Parser([]u8, Reader) = .{
 ._parse = parse,
 },
 };
}

Now a consumer can write the following to get a literal string parser:

const parser = Literal(@TypeOf(reader)).parser;

Passing parameters to a parser implementation

If we want our Literal parser to accept a parameter – the literal string to look for – we need to give it a parameter.

In the case of merely passing it a string, we could adjust the signature so that this is possible:

const parser = Literal("some string", @TypeOf(reader)).parser;

However, we’ll define ours using an init method which is more common in Zig data structures:

pub fn Literal(comptime Reader: type) type {
 return struct {
 parser: Parser([]u8, Reader) = .{
 ._parse = parse,
 },
 want: []const u8,

 const Self = @This();

 // The `want` string must stay alive for as long as the parser will be used.
 pub fn init(want: []const u8) Self {
 return Self{
 .want = want
 };
 }
 };
}

In this case, want is the string literal we want to match - and []const u8 is Zig’s string type. It describes a slice of immutable (non-modifiable) encoded UTF-8 bytes.

Unlike C, []const u8 being a slice means it is a pointer to the string in memory and its length - so we don’t have to pass around the length parameter separately or use a null-terminated string. In Zig, there are two ways to represent a string:

[]const u8 (unmodifiable string, most common)
[]u8 (modifiable string)

Understanding Zig’s wild/confusing `@fieldParentPtr`

We’re finally ready to actually have our Literal parser parse something! We just need to implement our parse method:

pub fn Literal(comptime Reader: type) type {
 return struct {
 parser: Parser([]u8, Reader) = .{
 ._parse = parse,
 },
 want: []const u8,
 ...
 const Self = @This();

 fn parse(parser: *Parser([]u8, Reader), allocator: *Allocator, src: *Reader) callconv(.Inline) Error!?[]u8 {
 const self = @fieldParentPtr(Self, "parser", parser);
 ...
 }
 };
}

But wait a minute! In order for the ._parse = parse, assignment to work the first argument to parse needs to be the self parameter for a Parser([]u8, Reader) - so how does our parse implementation method get to access the want field of our struct?

This is where some Zig magic comes in: on obscure builtin function we can use inside of our parse method:

const self = @fieldParentPtr(Self, "parser", parser);

To understand this, first let’s get a look at what these parameters are referring to:

We can see from the Zig documentation that this function operates as follows:

Given a pointer to a field, returns the base pointer of a struct.

So in our case:

Self is the “parent struct” we’re trying to acquire a reference to (our type)
"parser" is the name of our struct’s field.
parser is the pointer to our parser struct field.

Hopefully you can start to see the link here: parser is a pointer to our struct field, so Zig has a little helper @fieldParentPtr which can rely on that fact to give us our struct given a pointer to our struct field.

Implementing the rest of `parse`

Our full parse method will look like this:

// If a value is returned, it is up to the caller to free it.
fn parse(parser: *Parser([]u8, Reader), allocator: *Allocator, src: *Reader) callconv(.Inline) Error!?[]u8 {
 const self = @fieldParentPtr(Self, "parser", parser);
 const buf = try allocator.alloc(u8, self.want.len);
 errdefer allocator.free(buf);
 const read = try src.reader().readAll(buf);
 if (read < self.want.len or !std.mem.eql(u8, buf, self.want)) {
 try src.seekableStream().seekBy(-@intCast(i64, read));
 allocator.free(buf); // parsing failed
 return null;
 }
 return buf;
}

There are a few notable things here:

We’re trying to return a string from our parse function, i.e. the value it emits is a string (instead of an AST node).
The want string we got inside of our init method is agreed to only be valid while parse will still be called. We’ve decided to create a contract that all of our Parser implementations will either not hold onto memory given by others - or if they do, only do so until parse returns. Hence, we need to allocate a new string in our method.
Normally we could rely solely on defer (“run at end of function”) or errdefer (“run if an error is returned”), but since we’ve chosen to reserve the none optional null as “we didn’t parse anything” we need to manually free if we return null;. A nulldefer and somedefer could be nice, maybe?

Putting it all together, you’ll get something like this: GitHub gist.

Our first parser combinator

To demonstrate how a parser combinator would be implemented, we’ll try implementing the OneOf operator. It will take any number of parsers as input and run them consecutively until one succeeds or none do.

Let’s first start by writing out the basic structure of our function:

pub fn OneOf(comptime Value: type, comptime Reader: type) type {
 return struct {
 parser: Parser(Value, Reader) = .{
 ._parse = parse,
 },

 ...
 };
}

You’ll notice here that in contrast to our Literal parser function from earlier, this function takes a second comptime Value: type parameter. This is because we want it to work with any existing Parser implementation, regardless of what type of value it produces.

We can start to fill in the type by adding our init method:

pub fn OneOf(comptime Value: type, comptime Reader: type) type {
 return struct {
 parser: Parser(Value, Reader) = .{
 ._parse = parse,
 },
 parsers: []*Parser(Value, Reader),

 const Self = @This();

 // `parsers` slice must stay alive for as long as the parser will be
 // used.
 pub fn init(parsers: []*Parser(Value, Reader)) Self {
 return Self{
 .parsers = parsers,
 };
 }
 };
}

As you can see here, we’re going to simply take in a list of pointers to parsers. They’ll all need to have the same return Value as was specified in the call to OneOf.

One reason for this is that Zig does not support return type inference. You can have a function which takes anytype as a parameter, but it cannot return an anytype. This just means we need to have a generic function (in this case, OneOf) which accepts a type parameter and then use that Value type later. In a language like Haskell or OCaml, this would not be true.

Finally, we can implement our parse method:

pub fn OneOf(comptime Value: type, comptime Reader: type) type {
 return struct {
 ...

 // Caller is responsible for freeing the value, if any.
 fn parse(parser: *Parser(Value, Reader), allocator: *Allocator, src: *Reader) callconv(.Inline) Error!?Value {
 const self = @fieldParentPtr(Self, "parser", parser);
 for (self.parsers) | one_of_parser | {
 const result = try one_of_parser.parse(allocator, src);
 if (result != null) {
 return result;
 }
 }
 return null;
 }
 };
}

There are a few things to unpack here:

try one_of_parser.parse(allocator, src); indicates that if parsing using one_of_parser returns an error that our function should return immediately and not continue attempting to parse with other parsers.
if (result != null) { is how you check if an Optional type in Zig is “None”. I find this pretty interesting: it’s not null, it’s actually an optional “none” type - but it is called null. I’m not sure why, but can imagine this making the language friendlier to people unfamiliar with optional types.

Using our OneOf parser combinator

Now for the cool part: we get to put both our Literal parser and OneOf parser combinator to build a new parser!

// Define our parser.
var one_of = OneOf([]u8, @TypeOf(reader)).init(&.{
 &Literal(@TypeOf(reader)).init("dog").parser,
 &Literal(@TypeOf(reader)).init("sheep").parser,
 &Literal(@TypeOf(reader)).init("cat").parser,
});

The above will parse one of "dog", "sheep", or "cat" from the input reader.

We’re passing @TypeOf(reader) frequently above which makes the code a bit more cryptic than needed, and it would be possible to introduce a OneOfLiteral helper which makes the above instead read:

// Define our parser.
var one_of = OneOfLiteral([]u8, @TypeOf(reader)).init(&.{
 "dog",
 "cat",
 "sheep",
});

One thing to unpack here is this syntax for passing an array to init: &.{...}:

The function takes a parameter parsers: []*Parser(Value, Reader)
.{...} would give us a fixed size array [3]*Parser(Value, Reader)
&.{} gives us a pointer to an array, i.e. a slice []*Parser(Value, Reader).

Since our list is known at compile time, we don’t have to allocate or free memory for the slice. If our list was dynamic, we would need to do so.

Finally, we can actually use our parser above:

var p = &one_of.parser;
var result = try p.parse(allocator, &reader);
std.testing.expectEqualStrings("cat", result.?);
if (result) |r| {
 allocator.free(r);
}

Runtime parser generation

You might be wondering how we would go from the Literal parser and OneOf parser combinator to actually generating a parser at runtime that can parse the semantics defined in a regexp string.

Since our Parser interface is a runtime interface (you can swap out the implementation at runtime) and since our parser combinator OneOf operates using that interface (only the return value must be known at compile time, it could be a generic AST node) it means that we can easily dynamically create slices of []*Parser(...) at runtime based on the result of a parser combinator we have built - like our “dog, cat, sheep” parser from earlier.

The challenge left for you as a reader is to:

Write parsers like our Literal parser that can parse the components of our regexp a[bc].*abc:
- RegexLiteralParser can parse a, b, and c, but not abc (the string.)
- RegexRangeOpenParser can parse [.
- RegexRangeCloseParser can parse ]
- RegexAnyParser can parse the . “any character” syntax.
- RegexRepetitionParser can parse the * repetition operator.
Write a parser combinators like our OneOf parser, except have it parse a Sequence of parsers.
Use our Sequence parser combinator and RegexLiteralParser to build a RegexStringLiteralParser - similar to how we built out “dog, cat, sheep” parser.
Write a new kind of function called a runtime parser generator named RegexParser which will be super familiar:
- Take in a parser combinator called RegexSyntaxParser which can turn your regexp syntax into some intermediary like an AST.
- Have your function use parser combinators like OneOf, Sequence, etc. to build a brand new parser at runtime based on that intermediary AST.
- Return that new parser which parses the actual semantics described by the input regexp!

Closing thoughts

I am sorry for not giving you a full (or even partial) regex engine :) I am still exploring this and it is a large undertaking, this blog post would be far too long if it was included.

You can find a copy of the final code with parsers and parser combinators here. Just zig init-exe and plop them into your src/ directory.

You may also want to check out Mecha, a parser combinator library for Zig.

If anything was unclear or confusing, I’m happy to help: shoot me an email stephen@hexops.com or leave a comment on Hacker News / Reddit and I’ll follow up.

Postgres regex search over 10,000 GitHub repositories (using only a Macbook)

Wed, 17 Feb 2021 00:00:00 +0000

In this article, we share empirical measurements from our experiments in using Postgres to index and search over 10,000 top GitHub repositories using pg_trgm on only a Macbook.

This is a follow up to “Postgres Trigram search learnings”, in which we shared several learnings and beliefs about trying to use Postgres Trigram indexes as an alterative to Google’s Zoekt (“Fast trigram based code search”).

We share our results, as well as the exact steps we performed, scripts, and lists of the top 20,000 repositories by stars/language on GitHub so you can reproduce the results yourself should you desire.

TL;DR

This article is extensive and more akin to a research paper than a blog post. If you’re interested in our conclusions, see conclusions instead.

Goals

We wanted to get empirical measurements for how suitable Postgres is in providing regexp search over documents, e.g. as an alterative to Google’s Zoekt (“Fast trigram based code search”). In specific:

How many repositories can we index on just a 2019 Macbook Pro?
How fast are different regexp searches over the corpus?
What Postgres 13 configuration gives best results?
What other operational effects need consideration if seriously attempting to use Postgres as the backend for a regexp search engine?
What is the best database schema to use?

Hardware

We ran all tests on a 2019 Macbook Pro with:

2.3 GHz 8-Core Intel Core i9
16 GB 2667 MHz DDR4

During test execution, few other Mac applications were in use such that effectively all CPU/memory was available to Postgres.

Corpus

We scraped lists of the top 1,000 repositories from the GitHub search API ranked by stars for each of the following languages (~20.5k repositories in total):

C++, C#, CSS, Go, HTML, Java, JavaScript, MatLab, ObjC, Perl, PHP, Python, Ruby, Rust, Shell, Solidity, Swift, TypeScript, VB .NET, and Zig.

Cloning all ~20.5k repositories in parallel took ~14 hours with a fast ~100 Mbps connection to GitHub’s servers.

Dataset reduction

We found the amount of disk space required by git clone --depth 1 on these repositories to be a sizable ~412G for just 12,148 repositories - and so we put in place several processes for further reduce the dataset size by about 66%:

Removing .git directories resulted in a 30% reduction (412G -> 290G, for 12,148 repositories)
Removing files > 1 MiB resulted in another 51% reduction (290G -> 142G, for 12,148 repositories - note GitHub does not index files > 384 KiB in their search engine)

Database insertion

We concurrently inserted the entire corpus into Postgres, with the following DB schema:

CREATE EXTENSION IF NOT EXISTS pg_trgm;
CREATE TABLE IF NOT EXISTS files (
 id bigserial PRIMARY KEY,
 contents text NOT NULL,
 filepath text NOT NULL
);

In total, this took around ~8 hours to complete and Postgres’s entire on-disk utilization was 101G.

Creating the Trigram index

We tried three separate times to index the dataset using the following GIN Trigram index:

CREATE INDEX IF NOT EXISTS files_contents_trgm_idx ON files USING GIN (contents gin_trgm_ops);

In the first attempt, we hit an OOM after 11 hours and 34 minutes. This was due to a rapid spike in memory usage at the very end of indexing. We used a fairly aggressive Postgres configuration with a very large max WAL size, so it was not entirely unexpected.
In the second attempt, we ran out of SSD disk space after ~27 hours. Notable is that the disk space largely grew towards the end of indexing, similar to when we faced an OOM - it was not a gradual increase over time. For this attempt, we used the excellent pgtune tool to reduce our first Postgres configuration as follows:

shared_buffers = 4GB → 2560MB
effective_cache_size = 12GB → 7680MB
maintenance_work_mem = 16GB → 1280MB
default_statistics_target = 100 → 500
work_mem = 5242kB → 16MB
min_wal_size = 50GB → 4GB
max_wal_size = 4GB → 16GB
max_parallel_workers_per_gather = 8 → 4
max_parallel_maintenance_workers = 8 → 4

In our third and final attempt, we cut the dataset in half and indexing succeeded after 22 hours. In specific, we deleted half of the files in the database (from 19,441,820 files / 178GiB of data to 9,720,910 files / 82 GiB of data.) The Postgres configuration used was the same as in attempt 2.

Indexing performance: Memory usage

In our first attempt, we see the reported docker stats memory usage of the container grow up to 12 GiB (chart shows MiB of memory used over time):

In our second and third attempts, we see far less memory usage (~1.6 GiB consistently):

Indexing performance: CPU usage

Postgres’ Trigram indexing appears to be mostly single-threaded (at least when indexing a single table, we test multiple tables later.)

In our first attempt, CPU usage for the container did not rise above 156% (one and a half virtual CPU cores):

Our second attempt was around 150-200% CPU usage on average:

Our third attempt similarly saw an average of 150-200%, but with a brief spike towards the end to ~350% CPU:

Indexing performance: Disk IO

Disk reads/writes during indexing averaged about ~250 MB/s for reads (blue) and writes (red). Native in-software tests show the same Macbook able to achieve read/write speeds of ~860 MB/s with <5% affect on CPU utilization.

Addition made Feb 20, 2021: We ran tests using native Postgres as well (instead of in Docker with a bind mount) and found better indexing and query performance, more on this below.

Indexing performance: Disk space

The database contains 9,720,910 files totalling 82.07 GiB:

postgres=# select count(filepath) from files;
count
---------
9720910
(1 row)
postgres=# select SUM(octet_length(contents)) from files;
sum
-------------
88123563320
(1 row)

Before indexing, we find that all of Postgres is consuming 54G:

$ du -sh .postgres/
54G .postgres/

After CREATE INDEX, Postgres uses:

$ du -sh .postgres/
73G .postgres/

Thus, the index size for 82 GiB of text is 19 GiB (or 23% of the data size.)

Database startup times

From an operational standpoint, it is worth noting that if Postgres is starting clean (i.e. previous shutdown was graceful) then startup time is almost instantaneous: it begins accepting connections immediately and loads the index as needed.

However, if Postgres experienced a non-graceful termination during e.g. startup, it can take a hefty ~10 minutes with this dataset to start as it goes through an automated recovery process.

Queries executed

In total, we executed 19,936 search queries against the index. We chose queries which we expect give reasonably varying amounts of coverage over the trigram index (that is, queries whose trigrams are more or less likely to occur in many files):

Regexp query	Matching # files in entire dataset
`var`	unknown (2m+ suspected)
`error`	1,479,452
`123456789`	59,841
`fmt\.Error`	127,895
`fmt\.Println`	22,876
`bytes.Buffer`	34,554
`fmt\.Print.*`	37,319
`ac8ac5d63b66b83b90ce41a2d4061635`	0
`d97f1d3ff91543[e-f]49.8b07517548877`	0

Detailed breakdown

Query	Result Limit	Times executed
`var`	10	1000
`var`	100	1000
`var`	1000	100
`var`	unlimited	4
`error'`	10	2000
`error'`	100	2000
`error'`	1000	200
`error'`	unlimited	18
`123456789`	10	1000
`123456789`	100	1000
`123456789`	1000	100
`123456789`	unlimited	2
`fmt\.Error`	10	1000
`fmt\.Error`	100	1000
`fmt\.Error`	1000	100
`fmt\.Error`	unlimited	2
`fmt\.Println`	10	1000
`fmt\.Println`	100	1000
`fmt\.Println`	1000	100
`fmt\.Println`	unlimited	2
`bytes.Buffer`	10	4
`bytes.Buffer`	100	4
`bytes.Buffer`	1000	4
`bytes.Buffer`	unlimited	2
`fmt\.Print.*`	10	1000
`fmt\.Print.*`	100	1000
`fmt\.Print.*`	1000	100
`fmt\.Print.*`	unlimited	2
`ac8ac5d63b66b83b90ce41a2d4061635`	10	1000
`ac8ac5d63b66b83b90ce41a2d4061635`	100	1000
`ac8ac5d63b66b83b90ce41a2d4061635`	1000	100
`ac8ac5d63b66b83b90ce41a2d4061635`	unlimited	2
`d97f1d3ff91543[e-f]49.8b07517548877`	10	1000
`d97f1d3ff91543[e-f]49.8b07517548877`	100	1000
`d97f1d3ff91543[e-f]49.8b07517548877`	1000	100
`d97f1d3ff91543[e-f]49.8b07517548877`	unlimited	2

Query performance

In total, we executed 19,936 search queries against the database (linearly, not in parallel) which completed in the following times:

Time bucket	Percentage of queries	Number of queries
Under 50ms	30%	5,933
Under 250ms	41%	8,088
Under 500ms	52%	10,275
Under 750ms	63%	12,473
Under 1s	68%	13,481
Under 1.5s	74%	14,697
Under 3s	79%	15,706
Under 25s	79%	15,708
Under 30s	99%	19,788

Query performance vs. planning time

The following scatter plot shows how 79% of queries executed in under 3s (Y axis, in ms), while Postgres’s query planner had planned them for execution in under 100-250ms generally (X axis, in ms):

If we expand the view to include all queries, we start to get a picture of just how outlier these 21% of queries are (note that the small block of dots in the bottom left represents the same diagram shown above):

Query time vs. CPU & Memory usage

The following image shows:

(top) Query time in milliseconds
(middle) CPU usage percentage (e.g. 801% refers to 8 out of 16 virtual CPU cores being consumed)
(bottom) Memory usage in MiB.

Notable insights from this are:

The large increase in resource usage towards the end is when we began executing queries with no LIMIT.
CPU usage does not exceed 138%, until the spike at the end.
Memory usage does not exceed 42 MiB, until the spike at the end.

We suspect pg_trgm is single-threaded within the scope of a single table, but with table data partitioning (or splitting data into multiple tables with subsets of the data), we suspect better parallelism could be achieved.

Investigating slow queries

If we plot the number of index rechecks (X axis) vs. execution time (Y axis), we can clearly see one of the most significant aspects of slow queries is that they have many more index rechecks:

And if we look at the EXPLAIN ANALYZE output for one of these queries we can also confirm Parallel Bitmap Heap Scan is slow due to Rows Removed by Index Recheck.

Table splitting

Splitting up the search index into multiple smaller tables seems like an obvious approach to getting pg_trgm to use multiple CPU cores. We tried this by taking the same exact data set and splitting it into 200 tables, and found numerous benefits:

Benefit 1: Incremental indexing

If indexing fails after 11-27 hours, as happened to us twice in the non-splitting approach, all progress is not lost.

Benefit 2: Parallel indexing

Unlike our first non-splitting approach, which showed we were only able to utilize 1.5-2 virtual CPU cores, with multiple tables we are able to utilize 8-9 virtual CPU cores:

Benefit 3: Indexing is 84% faster

Unlike our first attempt which took 22 hours in total, parallel indexing completed in only 3h27m.

Benefit 4: Indexing uses 69% less memory

With non-splitting we saw peak memory usage up to 12 GiB. With the same exact Postgres configuration, we were able to index with only 3.7 GiB peak memory usage:

Benefit 4: Parallel querying

Previously, we saw CPU utilization of only 138% (1.3 virtual CPU cores), with table splitting we see CPU utilization during queries of 1600% (16 virtual CPU cores) showing we are doing work fully in parallel:

Similarly, we saw memory usage average around ~380 MiB, compared to only ~42 MiB before:

Benefit 5: Query performance

We reran the same exact set of search queries, but a smaller number of times overall (350 queries, instead of 19.9k - which we found to still be a representative enough sample.)

As we can see below, table splitting in general led to a 200-300% improvement in query time for heavier queries that previously took 20-30s, now taking only 7-15s thanks to parallel querying (top chart is before, bottom is after, both in milliseconds):

We also grouped queries based on the LIMIT specified in the query and placed them into time buckets (“how many queries completed in under 50ms?”) - comparing the two shows that less complex queries and/or queries for fewer results were negatively affected slightly, while larger queries were helped substantially:

Change (positive is good)	Results limit	Bucket	Queries in bucket before	Queries in bucket after
-33%	10	<50ms	33%	0%
+13%	10	<250ms	44%	57%
+33%	10	<1s	77%	100%
-29%	100	<100ms	29%	0%
+20%	100	<500ms	50%	70%
+19%	100	<10s	80%	99%
-12%	1000	<250ms	12%	0%
-13%	1000	<2.5s	77%	64%
+23%	1000	<20s	77%	100%
+4%	none	<20s	0%	4%
+18%	none	<60s	0%	18%

Detailed comparisons are available below for those interested:

Queries with `LIMIT 10`

Time bucket	Percentage of queries (before)	Percentage of queries (after splitting)
50ms	33.00% (2999 of 9004)	0% (0 of 100)
100ms	33.00% (2999 of 9004)	1.00% (1 of 100)
250ms	44.00% (3999 of 9004)	57.00% (57 of 100)
500ms	55.00% (4999 of 9004)	79.00% (79 of 100)
1000ms	77.00% (6998 of 9004)	80.00% (80 of 100)
2500ms	77.00% (7003 of 9004)	80.00% (80 of 100)
5000ms	77.00% (7004 of 9004)	80.00% (80 of 100)
10000ms	77.00% (7004 of 9004)	100.00% (100 of 100)
20000ms	77.00% (7004 of 9004)	100.00% (100 of 100)
30000ms	99.00% (8985 of 9004)	100.00% (100 of 100)
40000ms	99.00% (9003 of 9004)	100.00% (100 of 100)
50000ms	100.00% (9004 of 9004)	100.00% (100 of 100)
60000ms	100.00% (9004 of 9004)	100.00% (100 of 100)

Queries with `LIMIT 100`

Time bucket	Percentage of queries (before)	Percentage of queries (after splitting)
50ms	29.00% (2934 of 10000)	0% (0 of 100)
100ms	29.00% (2978 of 10000)	0% (0 of 100)
250ms	39.00% (3975 of 10000)	31.00% (31 of 100)
500ms	50.00% (5000 of 10000)	70.00% (70 of 100)
1000ms	59.00% (5984 of 10000)	79.00% (79 of 100)
2500ms	79.00% (7996 of 10000)	80.00% (80 of 100)
5000ms	80.00% (8000 of 10000)	80.00% (80 of 100)
10000ms	80.00% (8000 of 10000)	99.00% (99 of 100)
20000ms	80.00% (8000 of 10000)	100.00% (100 of 100)
30000ms	99.00% (9999 of 10000)	100.00% (100 of 100)
40000ms	100.00% (10000 of 10000)	100.00% (100 of 100)
50000ms	100.00% (10000 of 10000)	100.00% (100 of 100)
60000ms	100.00% (10000 of 10000)	100.00% (100 of 100)

Queries with `LIMIT 1000`

Time bucket	Percentage of queries (before)	Percentage of queries (after splitting)
50ms	0% (0 of 904)	0% (0 of 100)
100ms	0% (1 of 904)	0% (0 of 100)
250ms	12.00% (114 of 904)	0% (0 of 100)
500ms	30.00% (276 of 904)	21.00% (21 of 100)
1000ms	55.00% (499 of 904)	41.00% (41 of 100)
2500ms	77.00% (700 of 904)	64.00% (64 of 100)
5000ms	77.00% (704 of 904)	77.00% (77 of 100)
10000ms	77.00% (704 of 904)	98.00% (98 of 100)
20000ms	77.00% (704 of 904)	100.00% (100 of 100)
30000ms	88.00% (804 of 904)	100.00% (100 of 100)
40000ms	99.00% (901 of 904)	100.00% (100 of 100)
50000ms	99.00% (903 of 904)	100.00% (100 of 100)
60000ms	100.00% (904 of 904)	100.00% (100 of 100)

Queries with no limit`

Time bucket	Percentage of queries (before)	Percentage of queries (after splitting)
50ms	0% (0 of 28)	0% (0 of 50)
100ms	0% (0 of 28)	0% (0 of 50)
250ms	0% (0 of 28)	0% (0 of 50)
500ms	0% (0 of 28)	0% (0 of 50)
1000ms	0% (0 of 28)	0% (0 of 50)
2500ms	0% (0 of 28)	0% (0 of 50)
5000ms	0% (0 of 28)	0% (0 of 50)
10000ms	0% (0 of 28)	0% (0 of 50)
20000ms	0% (0 of 28)	4.00% (2 of 50)
30000ms	0% (0 of 28)	16.00% (8 of 50)
40000ms	0% (0 of 28)	16.00% (8 of 50)
50000ms	0% (0 of 28)	18.00% (9 of 50)
60000ms	0% (0 of 28)	18.00% (9 of 50)

Postgres-in-Docker vs. native Postgres

Addition made Feb 20, 2021

In our original article we did not clarify the performance impacts of running Postgres inside of Docker with a volume bind mount. This was raised as a potential source of IO performance difference to us by Thorsten Ball.

We ran all tests above with Postgres in Docker, using a volume bind mount (the osxfs driver, not the experimental FUSE gRPC driver.)

We additionally ran the same table-splitting benchmarks on a native Postgres server (reproduction steps here) and found the following key changes:

CPU usage & memory usage: approximately the same

CPU and memory usage was approximately the same as in our Docker Postgres tests.

We anticipated this would be the case as the Macbook does have VT-x virtualization enabled (default on all i7/i9 Macbooks, and confirmed through sysctl kern.hv_support)

Indexing speed was ~88% faster

Running the statements to split up the large table into multiple smaller ones, i.e.:

CREATE TABLE files_000 AS SELECT * FROM files WHERE id > 0 AND id < 50000;
CREATE TABLE files_001 AS SELECT * FROM files WHERE id > 50000 AND id < 100000;
...

Was much faster in native Postgres, taking about 2-8s for each table instead of 20-40s previously, and taking only 15m in total instead of 2h before.

Parallel creation of the Trigram indexes using e.g.:

CREATE INDEX IF NOT EXISTS files_000_contents_trgm_idx ON files USING GIN (contents gin_trgm_ops);

Was also much faster, taking only 23m compared to ~3h with Docker.

Query performance is 12-99% faster, depending on query

We re-ran the same 350 queries as in our earlier table-splitting benchmark, and found the following substantial improvements:

Queries that were previously very slow noticed a ~12% improvement. This is likely due to IO operations needed when interfacing with the 200 separate tables.
Queries that were previously in the middle-ground noticed meager ~5% improvements.
Queries that were previously fairly fast (likely searching only over a one or two tables before returning) noticed substantial 16-99% improvements.

Exhaustive comparison details (negative change is good)

Change	Time bucket	Queries under bucket before	Queries under bucket after
0%	500s	350 of 350	350 of 350
-12%	100s	309 of 350	350 of 350
-12%	50s	309 of 350	350 of 350
-12%	40s	308 of 350	350 of 350
-12%	30s	308 of 350	349 of 350
-7%	25s	307 of 350	330 of 350
-7%	25s	307 of 350	330 of 350
-8%	20s	302 of 350	330 of 350
-8%	20s	302 of 350	330 of 350
-5%	10s	297 of 350	311 of 350
-26%	5s	237 of 350	319 of 350
-7%	2500ms	224 of 350	240 of 350
-9%	2000ms	219 of 350	240 of 350
-9%	1500ms	219 of 350	240 of 350
-16%	1000ms	200 of 350	237 of 350
-14%	750ms	190 of 350	221 of 350
-23%	500ms	170 of 350	220 of 350
-59%	250ms	88 of 350	217 of 350
-99%	100ms	1 of 350	168 of 350
-99%	50ms	1 of 350	168 of 350

Conclusions

We think the following learnings are most important:

.git directories, even with --depth=1 clones, account for 30% of a repositories size on disk (at least in top 10,000 GitHub repositories.)
Files > 1 MiB (often binaries) account for another 51% of the data size on disk of repositories.
On only a Macbook Pro, it is possible to get Postgres Trigram regex search over 10,000 repositories to run most reasonable queries in under 5s - and certainly much faster with more hardware.
pg_trgm performs single-threaded indexing and querying, unless you split your data up into multiple tables.
By default, a Postgres text colum will be compressed by Postgres on disk out of the box - resulting in a 23% reduction in size (with the files we inserted.)
pg_trgm GIN indexes take around 26% the size of your data on disk. So if indexing 1 GiB of raw text, expect Postgres to store that text in around ~827 MiB plus 279 MiB for the GIN trigram index.
Splitting your data into multiple tables if using pg_trgm is an obvious win, as it allows for paralle indexing which can be the difference between 4h vs 22h. It also reduces the risk of an indexing failure after 22h due to e.g. lack of memory and uses much less peak memory overall.
Docker bind mounts (not volumes) are quite slow outside of Linux host environments (there are many other articles on this subject.)

If you are looking for fast regexp or code search today, consider:

Sourcegraph (disclaimer: the author works here, but this article is not endorsed or affiliated in any way)
Zoekt
Ripgrep

Follow this devlog for updates as we continue investigating faster ways to do regexp & ngram search at large scales.

Postgres Trigram search learnings

Tue, 26 Jan 2021 00:00:00 +0000

In this article I talk about learnings I have from trying to use pg_trgm as the backend for a search engine, Tridex, which aims to be a competitor to Google’s Zoekt (“Fast trigram based code search”)

Background

I work @ Sourcegraph, which provides code search, code intelligence, and other developer tooling. (If you’re one of my Sourcegraph co-workers, hey! Hexops is the name of my GitHub organization for after-hours experiments and what I hope will one day in the distant future become a successful game development company.)

Over the past ~8 months I have been exploring the intersection between game development and my work at Sourcegraph, and finding interesting overlap between the two. I have been working on a competitor to Google’s zoekt (“Fast trigram based code search”), which is one of the search backends used by Sourcegraph with a few goals in mind:

Produce a search backend I can use for Hexops, to provide search functionality for a user-voice type platform (think GitHub issues, but with duplicate issue removal and upvote/downvote capability), and other social-network type features in games I hope to one day create. i.e. not just code search
Learn more about search in hopes of being able to bring some of those learnings to Sourcegraph (I really want our commit/diff search to be indexed, and wish we could index more things in general.) It would also be cool to solve the ominous and difficult Zoekt memory usage problem we have had at Sourcegraph for as long as I can remember.
Provide a foundation for more bleeding-edge “here’s a whacky idea” type experiments that are cool surrounding developer tools, but that are not necessarily guaranteed wins / anything I could reasonable pitch elsewhere.

Indexing every character is different than regular FTS

First, it’s important to note that my usage of pg_trgm is not the same as general FTS (Full Text Search) usage of pg_trgm in general. My usage (and the use case of code search) cares about every character being indexed, and being able to do regex searches - this is different than traditional FTS where only words matter.

pg_trgm indexes apply to all data in that column

This sounds obvious, but in practice has interesting/weird implications when trying to build a code search engine like Zoekt.

For example, Zoekt builds indexes of repositories code in chunks (from what I understand) and then concurrently, in an unordered fashion, searches those repository code chunks (inverted trigram indexes). This plays a key role in the strategy of pagination that Zoekt implements: you can search over those chunks and give up searching further chunks after you’ve found enough results.

With pg_trgm, a naive approach would be to have a file_contents column with a pg_trgm GIN index over it, and put every file from every repository into that column. But that index would apply to all file contents across every repository, so when you want to LIMIT search over that column you are searching over one giant trigram GIN index instead of many smaller ones. It’s faster if your aim is to search the entire index, but if your aim is “get enough results and then get out” it’s much slower, because you have to deal with the entire index instead of multiple smaller indexes. But, I have not tested this empirically - so take this statement with a grain or two of salt. It’s possible I am wrong here.

There is an obvious way to counteract this effect, though: use one pg_trgm GIN index (i.e. a distinct table or column) per repository. You now have a distinct GIN trigram index per repository chunk. Of course, when there are thousands of repositories this introduces major complexity in schema management, query execution, etc. as you might imagine having thousands of tables/columns not exactly being great either.

It is possible that Postgres’ table data partitioning could interoperate with pg_trgm nicely to solve this problem, but I didn’t explore this in-depth and found no information on the subject. Importantly, you would need to partition tables based on repository (or better, file-size-based chunks.) It is worth exploring this approach more.

Naive usage of pg_trgm is competitive!

The good news is that even with the naive approach previously described, pg_trgm turns out to be approximately competitive with Zoekt, I assume due to it using a GIN index for trigram matching instead of an inverted index like Zoekt:

I don’t have empirical measurements of this that I can share, you’ll just have to take my word for it, but approximately on a 2020 Macbook pro with several thousand source code repositories:

Query time performance is roughly the same for needle-in-the-haystack and haystack-full-of-needles queries over all repositories file contents.
Pagination can be quite slow towards the tail end of the table, each subset being fetched requires a full search of the index to find the results at the end of the table. A streaming approach rather than traditional SQL pagination would be ideal.
On-disk data size is quite small compared to Zoekt’s, Postgres trigram GIN indexes appear to be quite small and its on-by-default data compression works really well with text.
Postgres uses MUCH less memory than Zoekt. Like several orders of magnitude less.

Interestingly, however, Postgres uses MUCH less memory. The choice of using an inverted trigram index in Zoekt, as I understand it, is also one of the reasons that its memory usage is so large (among other things, like Go being fairly relaxed about returning memory to the OS.) I also suggest reading these Hacker News comments from 2018 and the linked article from Russ Cox about Google Code Search, from which Zoekt was ultimately born.

Horizontally scaling pg_trgm is hard

Once your data no longer fits into a single machine / Postgres instance, things get tricky. How do you scale pg_trgm across multiple machines?

Postgres supports a nauseating amount of complex High Availability deployment options for scaling horizontally, and ideally for a search engine you would want something like data partitioning where data is split across multiple hosts but also with the possibility of replication across multiple hosts (for the event a host goes down.)

One of the options it supports is horizontal data partitioning through splitting tables into multiple smaller tables, and then using a foreign data wrapper (postgres_fdw) to execute queries that access all of those tables across the network. This is described in a bit more depth in this blog post. This could be a good approach, but I decided not to explore this option further.

Ultimately I decided to go with a multiple-table approach, with each table representing a type of data (e.g. repository code) and performing horizontal sharding and scaling at the application layer outside of the DB entirely. I will explain why I took this approach in the next section.

Deploying and tuning Postgres configuration is hard

In stark contrast to Zoekt, which uses a ridiculous amount of memory, with Postgres I was left with a different problem: I could not get it to use all available memory/CPU to perform search queries faster.

Raising shared_buffers = 128MB (default 32MB) helped a fair amount, but still a similar issue. Ultimately I believe the majority of query time is spent not on CPU latency, but rather a combination of RAM lookups / L3 cache misses and IO latency.

Nonetheless, this introduced a new problem for me: I wanted this search engine to be as simple to deploy as possible, and the idea of having Postgres tuning being a requirement did not appeal to me. I have also seen in the field how many use Amazon RDS, which does not allow for tuning (and is often a several-years-outdated Postgres version anyway.)

With all of this in mind, I ended up going with deploying Docker containers with my own Postgres binary and configuration built-in and managing Postgres on behalf of the user. This ended up being interesting for other reasons I won’t get into (think automatic zero-downtime upgrades from Postgres 12 -> 13.)

Ultra large scales

Although using pg_trgm is competitive (much better than?) Zoekt - it’s still not enough to be able to efficiently scale up to a massive scale such as Google. The index is still relatively large (in the hundreds of MB for thousands of repositories) well outside the bounds of CPU caches and that makes it kind of slow at truly large scales where the index grows near-linearly.

Ultimately.. Once you add in deployment pains, configuration tuning, trigram index splitting, horizontal scaling, etc. it’s a lot less like using Postgres to build a search engine - and a lot more like using Postgres as a trigram index provider. It’s interesting, and works, but there may be better options.

Future exploration

A more fruitful direction may be to explore effectively the same architecture (i.e. roll-your-own-search-engine), but replacing pg_trgm and Postgres entirely with a custom ngram index built on top of the bloom-filter successor which is more L3-cache-friendly, xor filters.

I believe with this approach you could achieve scales/performance similar to Google, Bing, etc. while providing full regex search and more. This idea is not completely unfounded, it has been suggested for indexing in ripgrep, for example (although it appears they’ll be going with an inverted trigram index similar to Zoekt instead.)

Closing thoughts

In Tridex (the search engine I am working on), we’re planning on exploring this avenue by replacing Postgres and pg_trgm with a custom trigram index based on xor-filters, and will likely write it in Zig. I only realized the opportunity here in a late-night conversation with a coworker who has an affinity for bloom filters, so perhaps I am misguided and this will turn up no fruit.

Follow this devlog for updates.