vla.simd: Efficient CPU Inference for Language-Conditioned Manipulation

A language-conditioned manipulation policy that supplies actions faster than a 30 Hz controller consumes them — on a Raspberry Pi 5, CPU only.

The author has been anonymized from the linked repositories.

≈1.4×
median speedup over compiled PyTorch
41 Hz
IMPACT action supply on a Raspberry Pi 5
76.4%
LIBERO success, no robot pretraining
≤2.6%
of per-query compute spent on language

Abstract

We present vla.simd, a CPU inference engine for language-conditioned manipulation policies, and IMPACT, an ACT-based policy that adds language conditioning through cached text representations and visual modulation. vla.simd combines shared SIMD micro-kernels, reusable computation and target-specific implementation choices, reaching a median speedup of approximately 1.4× over compiled PyTorch references while maintaining numerical fidelity. IMPACT encodes the instruction once per episode, spending an estimated at most 2.6% of per-query compute on language conditioning, and is the only language-conditioned policy in our evaluated set that supplies at least 30 actions per second on the Raspberry Pi 5.

Overview of vla.simd and IMPACT .

Try it: choosing the SMK register tile

vla.simd runs every linear layer, convolution and attention projection through one packed-panel SIMD micro-kernel, and picks its register tile per target. The paper’s SMK Tiling Algorithm, running live: pick a target, edit its parameters, click any row to see the register budget.

Method

Linear layers, convolutions via panel im2col and attention projections feeding a shared packed-panel SIMD micro-kernel.
vla.simd. Linear layers, convolutions via panel im2col, and attention projections share one packed-panel SIMD micro-kernel, whose register tile is selected per target (6×16 on AVX2, 4×16 on NEON). Work is organised by lifetime: weights are packed once per process, instruction-dependent tensors once per episode, and each query computes only observation-dependent features.
IMPACT: a frozen T5-small encodes the instruction once per episode, giving tokens to the transformer and FiLM coefficients to a frozen ResNet-18.
IMPACT. A frozen T5-small encodes the instruction once per episode, providing tokens for the transformer and FiLM coefficients for each stage of a frozen ResNet-18. Each query runs the modulated backbone and encoder–decoder to predict a 50-step action chunk, for an estimated at most 2.6% of per-query compute spent on language conditioning.
Six bar panels showing control rate on four CPUs, with the torch.compile baseline and the rate added or lost by vla.simd.
Action supply against compiled PyTorch. Blue segments show the rate gained by vla.simd, red segments a loss; the dashed line marks the 30 Hz budget. On the Ryzen, Octo-Small and SmolVLA cross that line with vla.simd.

Real-robot experiments

Six conditions across two embodiments, 20 episodes each. Success rates and 95% confidence intervals are those reported in the paper; latency is the mean query roundtrip on that bench.

SO-101 · AInstruction following in a shared scene

IMPACT, fp32. The scene is fixed and holds both objects; only the instruction changes.

Video placeholder
assets/videos/
so101-impact-tape-box.mp4
IMPACT90% (70–97)

“Put the tape into the box”

multi-task checkpoint · 996.6 ms
Video placeholder
assets/videos/
so101-impact-tape-cup.mp4
IMPACT85% (64–95)

“Put the tape into the cup”

multi-task checkpoint · 996.6 ms
Video placeholder
assets/videos/
so101-impact-cup-box.mp4
IMPACT60% (39–78)

“Put the cup into the box”

multi-task checkpoint · 996.6 ms
What to look for. The first two instructions share a grasp target and differ only in the destination; the third changes the object to be grasped. Destination selection is the stronger effect on this bench.

SO-101 · BLong-horizon drawer task

Open the drawer, place the tape inside, close it. Twenty demonstrations, a separate checkpoint.

Video placeholder
assets/videos/so101-impact-drawer.mp4
IMPACT65% (43–82)

“Put the tape into the drawer”

drawer checkpoint · 996.6 ms

UR10e · CA second embodiment, served from two CPUs

A UR10e with a Robotiq gripper running SmolVLA under vla.simd. The mean query roundtrip differs by 2.0× between hosts, but success does not: both reach 12 of 20, and the intervals span 39–78%, so these trials cannot resolve a host effect. Both clips play at 3× speed.

Video placeholder
assets/videos/
ur10e-smolvla-m4-3x.mp4
SmolVLA60% (39–78)

“Put the cup into the box”

served from an Apple M4 · 1177 ms · 3× speed
Video placeholder
assets/videos/
ur10e-smolvla-ryzen-3x.mp4
SmolVLA60% (39–78)

“Put the cup into the box”

served from a Ryzen 5 5500 · 2343 ms · 3× speed