cut: fix more compat issues - #13194

sylvestre · 2026-06-29T07:27:15Z

Add locale-aware character handling so -c counts characters, -b -n keeps multibyte characters whole, and -d accepts a single multibyte delimiter.

Should make test tests/cut/mb-non-utf8.sh pass

github-actions · 2026-06-29T07:42:58Z

GNU testsuite comparison:

Skip an intermittent issue tests/date/date-locale-hour (fails in this run but passes in the 'main' branch)
Skip an intermittent issue tests/date/resolution (fails in this run but passes in the 'main' branch)
Skipping an intermittent issue tests/tail/symlink (passes in this run but fails in the 'main' branch)
Skipping an intermittent issue tests/tail/tail-n0f (passes in this run but fails in the 'main' branch)
Congrats! The gnu test tests/cut/mb-non-utf8 is no longer failing!
Congrats! The gnu test tests/cut/bounded-memory is now passing!

codspeed-hq · 2026-06-29T08:46:48Z

Merging this PR will degrade performance by 26.55%

⚠️

Different runtime environments detected

Some benchmarks with significant performance changes were compared across different runtime environments,
which may affect the accuracy of the results.

Open the report in CodSpeed to investigate

❌ 1 regressed benchmark
✅ 344 untouched benchmarks
⏩ 46 skipped benchmarks¹

Warning

Please fix the performance issues or acknowledge them on CodSpeed.

Performance Changes

	Mode	Benchmark	`BASE`	`HEAD`	Efficiency
❌	Simulation	`cut_characters`	17.1 ms	23.3 ms	-26.55%

Tip

Investigate this regression by commenting @codspeedbot fix this regression on this PR, or directly use the CodSpeed MCP with your agent.

_{Comparing sylvestre:mb-non-utf8 (e34f4db) with main (4e95dfe)}

46 benchmarks were skipped, so the baseline results were used instead. If they were deleted from the codebase, click here and archive them to remove them from the performance reports. ↩

wasmtime's argument marshaling requires valid UTF-8, so a raw non-UTF-8 byte delimiter can't be passed through to the WASI binary the way it can natively on Linux.

The character path walked the line one character at a time through the encoding decoder and looked the locale encoding up per character, which dominated the per-line cost on mixed ASCII and multi-byte text. Take runs of ASCII bytes a machine word at a time. They are single-byte characters in every encoding handled here, so byte offset and character position move together across a run, and only the bytes above 0x7F need the decoder. Bundle the parts that are fixed for the whole run -- the ranges, the output delimiter and whether it was given, the position mode and the encoding -- into a CharCut built once by cut_chars, and make the line body a method on it, so the per-line call passes one pointer rather than seven arguments. In uucore, MbEncoding becomes the public Encoding and locale_encoding() returns it by value, so a caller decoding many characters resolves the locale once and keeps it in a register instead of reaching through a OnceLock per character. is_multibyte_locale() had no callers left. Instruction counts against the parent commit (cachegrind, LC_ALL=C.UTF-8): -c 5-30, 100k mixed short lines 30.44M -> 28.95M -c 20-70, 20k long multibyte lines 44.38M -> 41.02M That is roughly 1.1x in wall clock on both shapes; the machine was too loaded to quote a tighter figure. The single-byte path (LC_ALL=C) and field mode are unchanged. Selecting a range of characters still costs more than the same range of bytes, and always will: -c used to be an alias for -b, and characters have to be decoded to be counted. Tests cover advance directly -- character counting for -c, byte counting for -b -n, and the word boundary crossings -- plus a cut -c case over mixed ASCII and multi-byte lines.