zigdist is an experimental demonstration that integrates the Zig
programming language into an R package. It
provides high-performance distance metric calculations as a
proof-of-concept. While Zig is not currently supported on CRAN, its
exceptional performance, simple FFI, data-oriented design, and robust
systems programming capabilities suggest it could be of significant
interest to the R package development community.
- Euclidean (
zd_euclidean()): Pairwise row distance and cross-distance for numeric matrices, supporting missing values (partial distance scaling) - Gower (
zd_gower()): Mixed-type distance computations (scaling numeric features by range, matching categorical features), supporting missing values (pairwise deletion)
These distance methods use multi-threaded parallel execution by
specifying num_threads > 1.
You can install the development version of zigdist from GitHub:
# install.packages("remotes")
remotes::install_github("brownag/zigdist")Note: A Zig compiler (v0.13.0 or later) must be in your PATH to
compile the package from source.
The recommended way to install and manage Zig versions is using zvm
(Zig Version Manager).
1. Install zvm
curl -fsSL https://www.zvm.app/install.sh | bashThis script will install zvm to ~/.zvm and add the necessary
environment variables to your shell profile (e.g., .bashrc, .zshrc).
You might need to restart your terminal or source your shell profile for
the changes to take effect.
2. Install a Zig version
Once zvm is installed, you can install the current stable Zig version:
zvm install stable
zvm use stableYou can also set a default version with zvm alias default stable.
library(zigdist)
# Euclidean
x <- matrix(rnorm(15), nrow = 5, ncol = 3)
zd_euclidean(x)## [,1] [,2] [,3] [,4] [,5]
## [1,] 0.0000000 1.780781 0.9846057 2.616892 2.198979
## [2,] 1.7807811 0.000000 1.6287150 1.697149 1.549304
## [3,] 0.9846057 1.628715 0.0000000 2.163073 1.443739
## [4,] 2.6168917 1.697149 2.1630732 0.000000 2.304916
## [5,] 2.1989793 1.549304 1.4437385 2.304916 0.000000
# Gower (supports mixed types: numeric, nominal factors, and ordered factors)
df <- data.frame(
a = c(1, 2, 5),
b = factor(c("A", "B", "A")),
c = ordered(c("Low", "Medium", "High"), levels = c("Low", "Medium", "High"))
)
zd_gower(df)## [,1] [,2] [,3]
## [1,] 0.0000000 0.5833333 0.6666667
## [2,] 0.5833333 0.0000000 0.7500000
## [3,] 0.6666667 0.7500000 0.0000000
To demonstrate functional equivalence, we verify that the outputs of
zigdist match the standard R implementations (stats::dist and
cluster::daisy) using all.equal():
# Euclidean vs. stats::dist
x_check <- matrix(rnorm(300), nrow = 20, ncol = 15)
all.equal(zd_euclidean(x_check), as.matrix(dist(x_check)), check.attributes = FALSE)## [1] TRUE
# Gower vs. cluster::daisy vs. gower package
library(cluster)
library(gower)
df_check <- data.frame(
n = rnorm(20),
f = factor(sample(letters[1:3], 20, replace = TRUE), levels = letters[1:3]),
c = factor(sample(c("Yes", "No"), 20, replace = TRUE))
)
# vs daisy
all.equal(zd_gower(df_check), as.matrix(daisy(df_check, metric = "gower")), check.attributes = FALSE)## [1] TRUE
# vs gower package (reconstructed self-distance matrix)
gower_check <- t(sapply(seq_len(nrow(df_check)), function(i) gower_dist(df_check[i, , drop = FALSE], df_check)))
all.equal(zd_gower(df_check), gower_check, check.attributes = FALSE)## [1] TRUE
Below are benchmarks comparing zigdist against standard R
implementations, using the microbenchmark package.
library(microbenchmark)
library(cluster)
library(gower)We compare zigdist::zd_euclidean (single-threaded and multi-threaded)
against R’s built-in stats::dist (converted to a full matrix) on a 500
by 100 numeric matrix.
set.seed(42)
x_bench <- matrix(rnorm(50000), nrow = 500, ncol = 100)
bench_eucl <- microbenchmark(
`zigdist 1 thread` = zd_euclidean(x_bench, num_threads = 1L),
`zigdist 4 thread` = zd_euclidean(x_bench, num_threads = 4L),
stats_dist = as.matrix(dist(x_bench)),
times = 50
)
print(bench_eucl)## Unit: milliseconds
## expr min lq mean median uq max neval
## zigdist 1 thread 3.063626 3.128998 3.278182 3.198264 3.382754 4.050758 50
## zigdist 4 thread 1.018223 1.339149 1.698181 1.534324 1.889571 4.077439 50
## stats_dist 7.247508 7.396621 7.699355 7.594195 7.842951 9.285812 50
Even on a single thread (num_threads = 1L), zigdist significantly
outperforms R’s native stats::dist (written in optimized C) by more
than 2.4x. Spawning multiple threads (e.g., num_threads = 4L)
achieves a 4.9x speedup.
We compare zigdist::zd_gower (single-threaded and multi-threaded)
against cluster::daisy(..., metric = "gower") and the gower package
(https://github.com/markvanderloo/gower).
Since gower::gower_dist computes pairwise distances row-by-row (using
recycling) rather than a full distance matrix, we present two separate
benchmarks:
For this benchmark, we compute the full distance matrix for a dataset of
500 rows and 6 columns (3 numeric, 3 categorical). To compute a full
distance matrix using the gower package, we use sapply to calculate
the distance of each row against the entire dataset.
set.seed(42)
N_bench <- 500
df_bench <- data.frame(
n1 = rnorm(N_bench),
n2 = runif(N_bench),
n3 = rnorm(N_bench, mean = 10, sd = 2),
c1 = factor(sample(letters[1:4], N_bench, replace = TRUE), levels = letters[1:4]),
c2 = factor(sample(c("Yes", "No"), N_bench, replace = TRUE)),
c3 = factor(sample(colors()[1:10], N_bench, replace = TRUE))
)
bench_gow_mat <- microbenchmark(
`zigdist 1 thread` = zd_gower(df_bench, num_threads = 1L),
`zigdist 4 thread` = zd_gower(df_bench, num_threads = 4L),
cluster_daisy = as.matrix(daisy(df_bench, metric = "gower")),
`gower package` = t(sapply(seq_len(N_bench), function(i) gower_dist(df_bench[i, , drop = FALSE], df_bench))),
times = 20
)
print(bench_gow_mat)## Unit: microseconds
## expr min lq mean median uq
## zigdist 1 thread 533.882 591.0175 927.4873 709.2005 910.9095
## zigdist 4 thread 576.532 835.1885 1200.5112 1035.1730 1416.0705
## cluster_daisy 4908.707 5540.9450 6165.9361 5824.0570 6297.9920
## gower package 333371.570 606189.7195 1428244.9606 1045701.0025 1852848.8030
## max neval
## 3009.446 20
## 2959.935 20
## 8929.524 20
## 6557568.413 20
In this full-matrix scenario, zigdist is 8.2x faster than
cluster::daisy, and 1474.5x faster than the gower package (which
is bottlenecked by the R-level loop).
For a fairer comparison against the gower package, we compute the
distance between a single query record and a database of 1,000,000
records. This matches the native pairwise/recycling capability of
gower::gower_dist without any R-level loops.
To simulate real-world usage where calculations are not executed
back-to-back in a tight loop, we add an untimed setup = Sys.sleep(1.0)
to the benchmark. This allows OpenMP’s helper threads (used by the
gower package) to exceed their active spin-wait period and fall
asleep, exposing the true OS thread wakeup latency that occurs between
distinct function calls.
set.seed(42)
N_db <- 1000000
df_db <- data.frame(
n1 = rnorm(N_db),
n2 = runif(N_db),
n3 = rnorm(N_db, mean = 10, sd = 2),
c1 = factor(sample(letters[1:4], N_db, replace = TRUE), levels = letters[1:4]),
c2 = factor(sample(c("Yes", "No"), N_db, replace = TRUE)),
c3 = factor(sample(colors()[1:10], N_db, replace = TRUE))
)
df_query <- df_db[1, , drop = FALSE]
bench_gow_query <- microbenchmark(
`zigdist 1 thread` = zd_gower(df_query, df_db, num_threads = 1L),
`zigdist 4 thread` = zd_gower(df_query, df_db, num_threads = 4L),
`gower package` = gower_dist(df_query, df_db),
times = 30,
setup = Sys.sleep(1.0)
)
print(bench_gow_query)## Unit: milliseconds
## expr min lq mean median uq max neval
## zigdist 1 thread 17.78539 19.76445 22.43841 20.92187 22.19543 61.98366 30
## zigdist 4 thread 16.48880 19.17190 20.38906 20.14828 21.38995 27.30032 30
## gower package 15.66874 23.73765 61.39545 52.08770 99.75978 128.49248 30
When accounting for real-world thread wakeup latency, zigdist is
2.5x faster than the gower package on a single thread. Without the
sleep, gower’s OpenMP threads remain warm (spinning in user space)
between runs, artificially hiding the 20–40ms thread-wakeup latency.
