Speed up decimal integer parsing with SWAR - #161019
Conversation
Use SIMD-within-a-register to process 8 ASCII digits at once in from_str_radix when radix == 10 and the result is guaranteed not to overflow. Falls back to the existing per-digit loop for the remaining 0-7 digits. The fast path uses two helper functions: - is_8digits: branch-free check that all 8 bytes are b'0'..=b'9' - parse_8digits: 3 multiplications to pack 8 digits into a u64 Benchmark on 16-20 digit decimal strings (5000 iterations, stage 1): bench_u64_from_str_radix_10_long 98818 -> 73194 ns (-25.9%) bench_i64_from_str_radix_10_long 149705 -> 120089 ns (-19.8%) Also add LONG_ASCII_NUMBERS and from_str_radix_long_bench macro to exercise the fast path with strings that trigger 2+ SWAR iterations.
rustbot
added
S-waiting-on-review
|
Thanks for the pull request, and welcome! The Rust Project is excited to review your changes, and you should hear from @JohnTitor (or someone else) some time within the next two weeks. Please see the contribution instructions for more information. Namely, in order to ensure the minimum review times lag, PR authors and assigned reviewers should ensure that the review label (
Why was this reviewer chosen?The reviewer was selected based on:
|
This comment has been minimized.
This comment has been minimized.
Align trailing `//` comments vertically in the new benchmark data constant. rustfmt in nightly runs during the tidy CI job flags the misaligned comments.
| if radix == 10 { | ||
| while let [a, b, c, d, e, f, g, h, rest @ ..] = digits { | ||
| let chunk = u64::from_le_bytes([*a, *b, *c, *d, *e, *f, *g, *h]); | ||
| if !is_8digits(chunk) { | ||
| return Err(PIE { kind: InvalidDigit }); | ||
| } | ||
| let parsed = parse_8digits(chunk) as $int_ty; | ||
| result = result * (100_000_000u32 as $int_ty); | ||
| if is_positive { | ||
| result = result + parsed; | ||
| } else { | ||
| result = result - parsed; | ||
| } | ||
| digits = rest; | ||
| } | ||
| } |
There was a problem hiding this comment.
There should probably be a 32- and (maybe) 16-bit version so this doesn't wind up slower on those platforms
There was a problem hiding this comment.
Added 32-bit support in 68fd666. Since 8 digits don't fit in a u32 without overflow, I went with a 4-digit SWAR path using the same idea but scaled down — 2 wrapping_mul instead of 3. Anything left after that falls through to the original loop.
8-digit SWAR uses u64 ops that are emulated on 32-bit and slower than the per-byte loop. Add a 4-digit u32 variant so 32-bit targets get the speedup too. On 64-bit the 4-digit path also picks up the tail after the 8-digit loop. The 8-digit functions are cfg-gated to avoid dead code on 32-bit.
|
As per https://forge.rust-lang.org/policies/llm-usage.html, could you let me know did you use/how you used an LLM to create this PR? |
Used an LLM to help me understand the issue and brainstorm the SWAR approach. I wrote and tested the code myself on my machine, including checking the 4-digit variant against a Python reference. Could you help add the llm-assisted label? |
5 participants
Speed up from_str_radix for base 10 with SWAR
Closes #87249
Process 8 decimal digits at a time using SIMD-within-a-register
instead of one digit per loop iteration. Only applies when radix
is 10 and can_not_overflow guarantees the result fits, so the
fast path needs no overflow checks.
The SWAR technique was suggested by @bormand and @Alexhuszagh in
the issue thread. Two helpers:
is_8digits(v: u64) -> bool — branch-free check that all 8
bytes are b'0'..=b'9'
parse_8digits(v: u64) -> u64 — 3 multiplications to pack 8
digits into a single value
Remaining <8 digits fall through to the existing per-digit loop.
Benchmark (stage 1, x86_64, 16-20 digit strings × 5000):
bench_u64_from_str_radix_10_long 98818 → 73194 ns (−25.9%)
bench_i64_from_str_radix_10_long 149705 → 120089 ns (−19.8%)
The mixed-input benchmark barely moves because only 2 of 19 inputs
are long enough to trigger the fast path. Added LONG_ASCII_NUMBERS
and from_str_radix_long_bench! to cover it properly.