pkgmirror

Zig toolchain and package mirroring HTTP service.

The source code for https://pkg.hexops.org

Zig toolchain mirroring

Zig toolchain mirrors are used by various tools in the Zig ecosystem such as anyzig and setup-zig when fetching a Zig version, and a list of community mirrors are maintained by the Zig community and regularly monitored for performance.

pkgmirror effectively rewrites /zig/$FILE -> https://ziglang.org/builds/$FILE to provide a Zig toolchain mirror.

pkgmirror serves a Zig index.json file at /zig/index.json, mirroring the official ziglang.org/download/index.json — but with downloads pointing to this mirror, and with nominated zig versions added.

Some notable aspects:

• ziglang.org periodically purges nightly Zig builds, so this ensures you always have the specific nightly Zig version your projects depend on.
• When a specific Zig version is fetched, pkgmirror proactively fetches all available ziglang.org downloads for that Zig version (each OS/platform, signature files, the source and binary versions, etc.)
• pkgmirror is aware of Zig stable versions and Nominated zig versions - placing them in different directories on disk to ensure you can purge old nightly builds that take up too much space over time.
• pkgmirror proactively fetches all Zig stable versions and all nominated Zig versions, to ensure your mirror has a copy of them on disk even if nobody has requested them through your mirror yet.

Zig package mirroring (optional)

Zig packages (the files you depend on in build.zig.zon files) live at various locations, but having the ability to mirror these is important if you want reproducable builds long into the future:

• GitHub repositories can be deleted
• Microsoft can stop being generous in their services and/or break URLs
• Alternative forges can go down/offline

Configure the ZIG_PKG environment variable with a comma-separated list of <org>::<template URL> entries. When a request matches /pkg/$ORG/$REPO/$FILE, pkgmirror looks up the template for $ORG and substitutes the $REPO and $FILE variables to produce the upstream URL.

For example:

ZIG_PKG=hexops::https://github.com/hexops/$REPO/archive/$FILE,myorg::https://codeberg.org/myorg/$REPO/archive/$FILE

Produces two rewrite rules:

• /pkg/hexops/$REPO/$FILE -> https://github.com/hexops/$REPO/archive/$FILE
• /pkg/myorg/$REPO/$FILE -> https://codeberg.org/myorg/$REPO/archive/$FILE

This ensures you always have a copy of the Zig package on your own, self-hosted domain.

Zig artifact mirroring (optional)

pkgmirror also supports binary artifact mirroring, such as precompiled binaries or assets produced by a CI pipeline.

Configure the ZIG_ARTIFACT environment variable with a comma-separated list of <org>::<template URL> entries. When a request matches /artifact/$ORG/$REPO/$VERSION/$FILE, pkgmirror looks up the template for $ORG and substitutes the $REPO, $VERSION, and $FILE variables to produce the upstream URL.

For example:

ZIG_ARTIFACT=hexops::https://github.com/hexops/$REPO/releases/download/$VERSION/$FILE,myorg::https://codeberg.org/myorg/$REPO/releases/download/$VERSION/$FILE

Produces two rewrite rules:

• /artifact/hexops/$REPO/$VERSION/$FILE → https://github.com/hexops/$REPO/releases/download/$VERSION/$FILE
• /artifact/myorg/$REPO/$VERSION/$FILE → https://codeberg.org/myorg/$REPO/releases/download/$VERSION/$FILE

Storage layout

The on-disk layout of the data directory is as follows:

pkg/$ORG/$REPO/$FILE
artifact/$ORG/$REPO/$VERSION/$FILE
zig/dev/<ZIG VERSION>/$FILE
zig/nominated/<ZIG VERSION>/$FILE
zig/stable/<ZIG VERSION>/$FILE

Note that pkgmirror seperates Zig versions into distinct dev, nominated, and stable versions - this is to allow you to purge old dev versions periodically to free space if desired, without losing noteable nominated or stable versions.

Configuration

Suggested to configure:

Variable	Description	Default
HTTP_PORT	Port for the HTTP server.	8080
HTTPS_PORT	Port for the TLS proxy (only used when ACME_DOMAIN is set).	8443
ACME_DOMAIN	Domain to obtain a TLS certificate for. If unset, TLS is disabled.	(none)
BASE_URL	Public base URL used for the site.	http://localhost:8080
ZIG_PKG	Comma-separated <org>::<template URL> list for Zig package mirroring.	(none)
ZIG_ARTIFACT	Comma-separated <org>::<template URL> list for Zig artifact mirroring.	(none)

Environment variables that already have good defaults:

Variable	Description	Default
DATA_DIR	Directory for cached downloads.	data
ZIG_NOMINATED_VERSION_BASE_URL	Base URL for fetching nominated zig versions. ziglang.org typically doesn't have these because nightly builds are purged after some time, so it is recommended to leave this as the default.	https://pkg.hexops.org/zig/
ZIG_NOMINATED_INDEX_URL	URL of the nominated zig index.json to merge with the official ziglang.org index.	https://pkg.hexops.org/zig/index.json
ACME_CERT_DIR	Directory where cert/key files are installed.	/var/lib/pkgmirror/certs
ACME_SH_PATH	Path to the acme.sh script.	$HOME/.acme.sh/acme.sh
ACME_WEBROOT	Directory for ACME HTTP-01 challenge files (used during renewal).	/var/lib/pkgmirror/acme-challenge
ACME_RELOAD_CMD	Command acme.sh runs after installing renewed certificates.	systemctl restart pkgmirror

TLS / HTTPS with LetsEncrypt

pkgmirror has built-in TLS termination and automatically obtains and renews certificates from LetsEncrypt via acme.sh. Install acme.sh once, set ACME_DOMAIN, and everything is automatic.

Create a dedicated pkgmirror user, then install acme.sh as that user (so its config lives in the service's home directory). Replace you@example.com with an email address LetsEncrypt can contact you at:

sudo useradd --system --create-home --home-dir /var/lib/pkgmirror --shell /usr/sbin/nologin pkgmirror
sudo -u pkgmirror -H bash -c 'curl https://get.acme.sh | sh -s email=you@example.com'
sudo -u pkgmirror -H bash -c '~/.acme.sh/acme.sh --register-account -m you@example.com --server letsencrypt'
sudo -u pkgmirror -H bash -c '~/.acme.sh/acme.sh --set-default-ca --server letsencrypt'

On first start, if no certificates exist, pkgmirror invokes acme.sh --issue --standalone to obtain them. On subsequent starts, it runs acme.sh --renew to renew certificates approaching expiry. When certificates are available, pkgmirror starts a TLS proxy on HTTPS_PORT that terminates TLS and forwards traffic to the HTTP server on HTTP_PORT. pkgmirror also serves /.well-known/acme-challenge/ responses so that acme.sh can perform HTTP-01 challenges while the server is running.

Installation

Building

Install anyzig or the specific .minimum_zig_version mentioned in the build.zig.zon file; then:

zig build -Dtarget=x86_64-linux -Doptimize=ReleaseFast -Dcpu=x86_64_v3

Copy the resulting zig-out/bin/pkgmirror binary to your server /usr/local/bin/pkgmirror

Running as a systemd service

A pkgmirror.service systemd file is included in this repository. Configure the Environment section of the file according to your needs, then install the service:

sudo cp pkgmirror.service /etc/systemd/system/
sudo systemctl daemon-reload

Use systemctl [start|stop|restart|status] pkgmirror to manage the service, and journalctl -u pkgmirror -f to view logs.

Note: Be sure to do backups of /var/lib/pkgmirror — all data, SSL certs, etc. are stored there.

(optional) Migrating from Wrench

To migrate from wrench, locate your Wrench /cache directory. Back it up first, then perform the following (set the WRENCH_DATA var to your cache directory):

export WRENCH_DATA=/cache
export DATA=/var/lib/pkgmirror/data

# Remove empty directories (wrench's cache may contain empty junk directories from bot probes)
find -L $WRENCH_DATA -type d -empty -delete

# Move zig downloads, renaming mach -> nominated
mkdir -p $DATA
mv $WRENCH_DATA/zig $DATA/
mv $DATA/zig/mach $DATA/zig/nominated

# Move package caches (wrench stored repos flat; pkgmirror expects an org prefix)
# Replace "hexops" with the org name you configured in ZIG_PKG / ZIG_ARTIFACT.
mkdir -p $DATA/pkg/hexops
mv $WRENCH_DATA/pkg/* $DATA/pkg/hexops/

# Move artifact caches
mkdir -p $DATA/artifact/hexops
mv $WRENCH_DATA/pkg-artifact/* $DATA/artifact/hexops/