Outreachy tracking issue for Rust whitespace check bugs

[teor2345]

The Rust language specification (specifically, the lexer) defines "whitespace" including the "vertical tab" character.

But some tools that process Rust use the standard library's is_ascii_whitespace which doesn't include vertical tab. This is a bug.

This ticket is based on this Zulip discussion

Impacted Functions

Any standard library (or crate) functions that call is_ascii_whitespace are also impacted:

• trim_ascii*
• split_ascii_whitespace
• ascii::Char::is_whitespace
• TODO: are there any others?

How to Fix this Bug

Unfortunately, there is no standard library method that matches on Unicode Pattern_White_Space (or the ASCII or u8 subset). So the fix should call the Rust compiler function, or create a new function with the correct filters by escape or character code.

Steps to Fix these Bugs

You can use AI for your understanding, but don't use it to write messages, comments, or code. We want to talk to you, and we want you to contribute your own work!

You must have a code example PR merged for either the Overloading or Build System project. If you don't, open a PR for Overloading (Build System is closed to new applicants), or work on your open PR until it is finished.

1. Choose one bug from the list of bugs. There is a strict limit of one bug per Outreachy applicant. Claim the issue by commenting on this ticket. If someone else has claimed it, choose another bug.
2. Read the contribution instructions for the Rust tool you're fixing.
3. Checkout the tool's git repository.
4. Create a commit with a test that shows the issue, this test should fail until you fix the bug.
5. Fix the issue in another commit, and update the tests or test outputs so they pass.
6. Open a PR for the Rust tool that explains the issue, and fixes it. Add a link to this issue (Outreachy tracking issue for Rust whitespace check bugs #53) in the PR description.

Unusual whitespace is deleted by rustfmt, so you might need to put #[rustfmt::skip] in your test code. (Or #![rustfmt::skip] at the top of the file.)

For more details, see the Zulip discussion.

List of Bugs that exclude Vertical Tab

These bugs don't include Vertical Tab. They also don't include Unicode Pattern_White_Space, but that might be intentional.

Extra steps for these bugs:
7. On the PR, ask if you should switch from checking ASCII to Unicode.

Clippy

• false negative in needless ifs
• short span in same match arms
• incorrect whitespace check in collapsible_if

rustfmt

• Only splitting on regex ASCII whitespace (including vertical tab), but not Unicode Pattern_White_Space in the line splitting regex - regex docs

Rust Analyzer

• panic in rust-analyzer whitespace token check
• non-macro code miscompare - currently commented out

List of Related Bugs

If all bugs above are claimed, you can do one of these related bugs or cleanups instead.

Rust run-make-support

• implement shell whitespace splitting correctly

Rust Compiler

• Markdown code block parsing accepts trailing non-whitespace (but bans whitespace)
  • Instead, it should accept trailing whitespace, and ban trailing non whitespace. This is roughly !any(!u8::is_ascii_whitespace), or more simply, all(u8::is_ascii_whitespace).

List of Cleanups

Rust Compiler

• Replace custom trim_ascii_start function with the standard library function
  • this is markdown parsing, so the check is correct, but it doesn't need to duplicate standard library code
  • the Rust maintainers might tell you to fix the function to match its documentation instead - that's ok

List of Bugs that include too much Unicode Whitespace

These bugs include all Unicode White_Space using is_whitespace methods. But the Rust language uses Unicode Pattern_White_Space as whitespace.

No confirmed bugs, in most tools (except the lexer), ignoring more whitespace is not a significant bug.

If you think you've found a significant buggy use of is_whitespace, trim or similar in a Rust parser, please comment below.

If there aren't any bugs left

Add tests for the rust compiler, clippy, rustfmt, or other Rust tools, which process Rust source code files containing:

• Unicode Pattern_White_Space, which should be accepted
  • WhatWG ASCII whitespace is a subset of this whitespace
• Non-Pattern Unicode White_Space, which should be rejected

You might need to use a character picker or Rust code to write the Unicode whitespace to the file.

List of tools:

• rustfmt
• clippy
• rustc (rust compiler)
• rust-analyzer
• TODO: other tools?

[Ajay-singh1]

I'll be working on clippy bug false negatives in needless ifs.

[JojoFlex1]

will do rustfmt bug

[teor2345]

I'll be working on clippy bug.

@Ajay-singh1 there are two clippy bugs listed above, please choose one of them. You can copy-paste the name from the ticket 🙂

[Ajay-singh1]

Sure!

[shashu8660]

i will be working on [short span in same match arms] clippy bug listed above

[teor2345]

I'll be working on clippy bug.

@Ajay-singh1 there are two clippy bugs listed above, please choose one of them. You can copy-paste the name from the ticket 🙂

The short span in same match arms clippy bug has been chosen by someone else.

Since you haven't said which one you've chosen, I've selected false negative in needless ifs for you. Please do that one.

Edit: got them the wrong way around originally, but you picked the right one

[TG199]

I'll work on Rust Analyzer: panic in rust-analyzer whitespace token check

[iAmChidiebereh]

Hi @teor2345 kindly de-assign me from rust analyzer [non-macro code miscompare].
I will work on incorrect whitespace check in collapsible_if instead

[Sandijigs]

hi @teor2345 , i would like to work on implement shell whitespace splitting correctly

[yalagadapavankumar]

Hi, I'd like to work on this issue: Markdown code block parsing accepts trailing non-whitespace (but bans whitespace)