pkgmirror

Zig toolchain and package mirroring HTTP service.

The source code for https://pkg.hexops.org

Zig toolchain mirroring

Zig toolchain mirrors are used by various tools in the Zig ecosystem such as anyzig and setup-zig when fetching a Zig version, and a list of community mirrors are maintained by the Zig community and regularly monitored for performance.

pkgmirror effectively rewrites /zig/$FILE -> https://ziglang.org/builds/$FILE to provide a Zig toolchain mirror.

pkgmirror serves a machine-readable Zig index.json file at /zig/index.json, which mirrors the official ziglang.org/download/index.json but with downloads pointing to this mirror, and with nominated zig versions added.

Some notable aspects:

• ziglang.org periodically purges nightly Zig builds, so this ensures you always have the specific nightly Zig version your projects depend on.
• When a specific Zig version is fetched, pkgmirror proactively fetches all available ziglang.org downloads for that Zig version (each OS/platform, signature files, the source and binary versions, etc.)
• pkgmirror is aware of Zig stable versions and Nominated zig versions - placing them in different directories on disk to ensure you can purge old nightly builds that take up too much space over time.
• pkgmirror proactively fetches all Zig stable versions and all nominated Zig versions, to ensure your mirror has a copy of them on disk even if nobody has requested them through your mirror yet.

Zig package mirroring (optional)

Zig packages (the files you depend on in build.zig.zon files) live at various locations, but having the ability to mirror these is important if you want reproducable builds long into the future:

• GitHub repositories can be deleted
• Microsoft can stop being generous in their services and/or break URLs
• Alternative forges can go down/offline

Configure the ZIG_PKG environment variable with a comma-separated list of <org>::<template URL> entries. When a request matches /pkg/$ORG/$REPO/$FILE, pkgmirror looks up the template for $ORG and substitutes the $REPO and $FILE variables to produce the upstream URL.

For example:

ZIG_PKG=hexops::https://github.com/hexops/$REPO/archive/$FILE,myorg::https://codeberg.org/myorg/$REPO/archive/$FILE

Produces two rewrite rules:

• /pkg/hexops/$REPO/$FILE -> https://github.com/hexops/$REPO/archive/$FILE
• /pkg/myorg/$REPO/$FILE -> https://codeberg.org/myorg/$REPO/archive/$FILE

This ensures you always have a copy of the Zig package on your own, self-hosted domain.

Zig artifact mirroring (optional)

pkgmirror also supports binary artifact mirroring, such as precompiled binaries or assets produced by a CI pipeline.

Configure the ZIG_ARTIFACT environment variable with a comma-separated list of <org>::<template URL> entries. When a request matches /artifact/$ORG/$REPO/$VERSION/$FILE, pkgmirror looks up the template for $ORG and substitutes the $REPO, $VERSION, and $FILE variables to produce the upstream URL.

For example:

ZIG_ARTIFACT=hexops::https://github.com/hexops/$REPO/releases/download/$VERSION/$FILE,myorg::https://codeberg.org/myorg/$REPO/releases/download/$VERSION/$FILE

Produces two rewrite rules:

• /artifact/hexops/$REPO/$VERSION/$FILE → https://github.com/hexops/$REPO/releases/download/$VERSION/$FILE
• /artifact/myorg/$REPO/$VERSION/$FILE → https://codeberg.org/myorg/$REPO/releases/download/$VERSION/$FILE

Storage layout

The on-disk layout of the data directory is as follows:

pkg/$ORG/$REPO/$FILE
artifact/$ORG/$REPO/$VERSION/$FILE
zig/dev/<ZIG VERSION>/$FILE
zig/nominated/<ZIG VERSION>/$FILE
zig/stable/<ZIG VERSION>/$FILE

Note that pkgmirror seperates Zig versions into distinct dev, nominated, and stable versions - this is to allow you to purge old dev versions periodically to free space if desired, without losing noteable nominated or stable versions.

Configuration

Suggested to configure:

Variable	Description	Default
HTTP_PORT	Port for the HTTP server.	8080
HTTPS_PORT	Port for the TLS reverse proxy to run on. If unset, TLS is disabled.	8443
ACME_DOMAIN	Domain to obtain a TLS certificate for. If unset, TLS is disabled.	(none)
ACME_CONTACT_EMAIL	Optional contact email for the ACME account.	(none)
BASE_URL	Public base URL used for the site.	http://localhost:8080
ZIG_PKG	Comma-separated <org>::<template URL> list for Zig package mirroring.	(none)
ZIG_ARTIFACT	Comma-separated <org>::<template URL> list for Zig artifact mirroring.	(none)

Environment variables that already have good defaults:

Variable	Description	Default
HTTPS_BIND_HOST	Address the TLS reverse proxy binds to.	0.0.0.0
HTTPS_UPSTREAM_HOST	Host the TLS reverse proxy forwards plaintext requests to (the local HTTP server).	127.0.0.1
DATA_DIR	Directory for cached downloads.	data
ZIG_NOMINATED_VERSION_BASE_URL	Base URL for fetching nominated zig versions, which ziglang.org purges after a short while.	https://pkg.hexops.org/zig/
ZIG_NOMINATED_INDEX_URL	URL of the nominated zig index.json to merge with the official ziglang.org index.	https://pkg.hexops.org/zig/index.json
ACME_CERT_DIR	Directory where ACME account keys, certificate keys, and certificate files are stored.	/var/lib/pkgmirror/certs
ACME_TERMS_OF_SERVICE_AGREED	Whether to agree to the ACME CA terms of service when creating the account.	true
ACME_DIRECTORY_URL	ACME directory URL. Defaults to Let's Encrypt production.	https://acme-v02.api.letsencrypt.org/directory

TLS / ACME knobs

These have sensible defaults and most deployments will not need to configure them. They only apply when HTTPS_PORT is set.

Variable	Description	Default
TLS_MAX_MEMORY_MIB	Cap, in MiB, on memory used by all connections before further connections are refused.	16384 (16 GiB)
TLS_KERNEL_BACKLOG	Number of connections the kernel may backlog before the proxy accepts them.	65535
TLS_CONNECTION_QUEUE_CAPACITY	Number of accepted connections queued in-process before a worker handles them. Excess connections are dropped.	65535
TLS_CONNECTION_WORKER_COUNT	Number of workers handling connections in parallel.	512
TLS_MAX_IDLE_UPSTREAM_CONNECTIONS	Number of idle upstream connections kept for reuse, to avoid reconnecting to the local HTTP server.	512
ACME_REQUEST_TIMEOUT_SECONDS	Timeout for individual HTTP requests to the ACME CA. Must be greater than zero.	30
ACME_RENEW_BEFORE_DAYS	Renew the certificate when it expires within this many days.	30

TLS / HTTPS with LetsEncrypt

If HTTPS_PORT is set, pkgmirror will use built-in TLS termination and automatically obtain, renew, and hot-reload certificates from LetsEncrypt using the in-process autocert reverse proxy.

For this to work, you should configure an ACME_CONTACT_EMAIL that they can contact you at for renewal/service concerns, Ensure ACME_DOMAIN is set, public port 80 reaches HTTP_PORT, and public port 443 reaches HTTPS_PORT.

When ACME_DOMAIN is set, pkgmirror keeps the HTTP server listening publicly on HTTP_PORT so LetsEncrypt can reach /.well-known/acme-challenge/ for HTTP-01 validation. All other HTTP requests are redirected to BASE_URL. The TLS reverse proxy listens on HTTPS_PORT, terminates TLS, forwards requests to the local HTTP server, and reloads certificates after renewal without restarting the process.

Installation

Building

Install anyzig or the specific .minimum_zig_version mentioned in the build.zig.zon file; then:

zig build -Dtarget=x86_64-linux -Doptimize=ReleaseFast -Dcpu=x86_64_v3

Copy the resulting zig-out/bin/pkgmirror binary to your server /usr/local/bin/pkgmirror

Running as a systemd service

A pkgmirror.service systemd file is included in this repository. Configure the Environment section of the file according to your needs, then ceate the dedicated pkgmirror user:

sudo useradd --system --create-home --home-dir /var/lib/pkgmirror --shell /usr/sbin/nologin pkgmirror

Then install the service:

sudo cp pkgmirror.service /etc/systemd/system/
sudo systemctl daemon-reload

Use systemctl [start|stop|restart|status] pkgmirror to manage the service, and journalctl -u pkgmirror -f to view logs.

Note: Be sure to backup /var/lib/pkgmirror (which is where all data, SSL certs, etc. are stored.)

(optional) Migrating from old pkgmirror + acme.sh deployment

Older pkgmirror versions used acme.sh as an external ACME client. The current version uses the in-process autocert client instead, so acme.sh is no longer used/supported in pkgmirror for new certificates or renewals.

When upgrading an existing deployment:

• Remove ACME_SH_PATH, ACME_WEBROOT, and ACME_RELOAD_CMD from your environment or systemd service - they're no longer used.
• Keep ACME_DOMAIN set to the public hostname for the mirror.
• Set ACME_CONTACT_EMAIL if you want the ACME account to have a contact email.
• Ensure public port 80 reaches HTTP_PORT, and public port 443 reaches HTTPS_PORT.
• Deploy the new binary and restart pkgmirror

Existing certificates installed by acme.sh are not imported automatically. On first startup after the upgrade, pkgmirror will request a certificate.

Note: ACME_CERT_DIR is not just an install destination for cert.pem and key.pem. It is autocert's persistent state directory, and stores the ACME account key, certificate private key, certificate chain, and renewal state. Back it up with the rest of /var/lib/pkgmirror. The default ACME_CERT_DIR is /var/lib/pkgmirror/certs.

You may also remove acme.sh from your server, if desired:

sudo -u pkgmirror -H /var/lib/pkgmirror/.acme.sh/acme.sh --uninstall

(optional) Migrating from Wrench

To migrate from wrench, locate your Wrench /cache directory. Back it up first, then perform the following (set the WRENCH_DATA var to your cache directory):

export WRENCH_DATA=/cache
export DATA=/var/lib/pkgmirror/data

# Remove empty directories (wrench's cache may contain empty junk directories from bot probes)
find -L $WRENCH_DATA -type d -empty -delete

# Move zig downloads, renaming mach -> nominated
mkdir -p $DATA
mv $WRENCH_DATA/zig $DATA/
mv $DATA/zig/mach $DATA/zig/nominated

# Move package caches (wrench stored repos flat; pkgmirror expects an org prefix)
# Replace "hexops" with the org name you configured in ZIG_PKG / ZIG_ARTIFACT.
mkdir -p $DATA/pkg/hexops
mv $WRENCH_DATA/pkg/* $DATA/pkg/hexops/

# Move artifact caches
mkdir -p $DATA/artifact/hexops
mv $WRENCH_DATA/pkg-artifact/* $DATA/artifact/hexops/