Skip to content

feat(parse): add narrow token boundary controls - #1097

Merged
jdx merged 4 commits into
agent/remove-inferred-prefixesfrom
agent/token-boundary-controls
Aug 20, 2026
Merged

feat(parse): add narrow token boundary controls#1097
jdx merged 4 commits into
agent/remove-inferred-prefixesfrom
agent/token-boundary-controls

Conversation

@jdx

@jdx jdx commented Aug 20, 2026

Copy link
Copy Markdown
Owner

Adds portable, typed support for clap-compatible token boundaries:

  • allow_negative_numbers for positional and flag values without accepting arbitrary dash-prefixed words
  • value_terminator for variadic flags and positionals
  • Rust static metadata, usage-lib, generated Go, and direct Go parsing
  • derive attributes and canonical KDL emission
  • clap bridge conversion and fidelity reporting
  • conformance vectors, migration docs, compatibility matrix, and PLAN updates

Leaves dont_delimit_trailing_values as the remaining distinct PLAN gap.

Validation:

  • cargo test --all --all-features
  • go test ./...

This pull request was generated by Codex.


Note

Medium Risk
Core argv binding semantics change (negative numbers now require opt-in), with coordinated updates across Rust, Go, KDL, and the clap bridge; behavior is heavily tested but migrations may need explicit allow_negative_numbers on affected args.

Overview
Introduces portable allow_negative_numbers and value_terminator so clap-style token boundaries can be declared on flags and positionals and carried through KDL, the derive, static tables, usage-lib, usage-argv, generated Go, and the clap bridge.

allow_negative_numbers is opt-in: dash-prefixed tokens are treated as flags unless a value-taking field explicitly allows negative numeric tokens (e.g. --jobs -1 without swallowing --force). Corpus and parser tests were updated because negative numbers are no longer accepted implicitly everywhere.

value_terminator ends a variadic flag or positional at a given token without binding that token (e.g. --include a ; tail).

Fidelity reporting no longer flags these as bridge losses when they round-trip; dont_delimit_trailing_values remains the outstanding PLAN gap.

Reviewed by Cursor Bugbot for commit 2339b2a. Bugbot is set up for automated code reviews on this repo. Configure here.

@coderabbitai

coderabbitai Bot commented Aug 20, 2026

Copy link
Copy Markdown

Important

Review skipped

Auto reviews are disabled on base/target branches other than the default branch.

Please check the settings in the CodeRabbit UI or the .coderabbit.yaml file in this repository. To trigger a single review, invoke the @coderabbitai review command.

⚙️ Run configuration

Configuration used: Central YAML (base), Organization UI (inherited)

Review profile: CHILL

Plan: Pro Plus

Run ID: b7bec3e1-a96e-4a4d-85e9-28bf59aa1aaa

You can disable this status message by setting the reviews.review_status to false in the CodeRabbit configuration file.

Use the checkbox below for a quick retry:

  • 🔍 Trigger review

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands.

Comment thread argv/src/lib.rs
Comment thread clap_usage/src/generate.rs
@jdx
jdx force-pushed the agent/token-boundary-controls branch from 8630b5d to f3f5c39 Compare August 20, 2026 00:51
Comment thread argv/src/lib.rs
@jdx
jdx force-pushed the agent/token-boundary-controls branch from e505788 to 39704e2 Compare August 20, 2026 00:58

@cursor cursor Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Cursor Bugbot has reviewed your changes and found 1 potential issue.

Fix All in Cursor

❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, enable autofix in the Cursor dashboard.

Reviewed by Cursor Bugbot for commit 39704e2. Configure here.

Comment thread lib/src/parse.rs
@github-actions

Copy link
Copy Markdown
Contributor

Instruction counts

benchmark trend instructions Δ wall (min) Δ
markdown ██▁ 198,671,176 → 195,842,945 -1.42% 18.71 → 18.18ms -2.83%
startup ▁▁█ 1,217,543 → 1,218,236 +0.06% 1.34 → 1.30ms -3.06%

No instruction-count regression above 1%.

Only instruction counts gate. Wall clock is shown for context — on identical hardware it moves 4-20% run to run.

Measured by tak — instruction-counted CLI benchmarks, stored in this repository's git notes.

Shadow comparison

Parsing mise use -g node@20 against a shadow of mise's committed spec.
Reported, not gated: the shadow grows as the derive learns to express more, so
what to watch is the ratio rather than either column.

framework instructions, cold parse vs usage
usage 4325
argh 6292 1.5x
clap 5893640 1362x
bpaf 21917948 5067x
                                              min       p01       p10    median
usage-rs: argv -> struct                      209       212       216       224  ns
argh: argv -> struct                          262       272       277       284  ns
clap: build tree + parse -> struct         475719    477034    479284    483311  ns
bpaf: build parser + parse -> struct      1583720   1583720   1590403   1608912  ns

usage: argv -> struct                             213 ns      0.21 µs
clap: build tree + parse -> struct             490790 ns    490.79 µs
clap: parse -> struct, tree reused              22784 ns     22.78 µs
clap: build tree only                          308219 ns    308.22 µs

2339b2a802ea vs 83ed87bb9e8f · measured on the runner, not pushed to the history.

@jdx
jdx merged commit 1e2ff15 into main Aug 20, 2026
10 checks passed
@jdx
jdx deleted the agent/token-boundary-controls branch August 20, 2026 01:52
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant