consistency rule intermittently drops an alert when a scope: paragraph rule is enabled (v3.18.0+) #1184

Check for existing issues

• Completed

Environment

• OS: macOS (Apple Silicon)
• Install method: Homebrew. I also got the same result with the official vale_3.22.0_macOS_arm64.tar.gz release.
• Vale version: 3.22.0

Describe the bug / provide steps to reproduce it

Running Vale repeatedly on the same file with the same config sometimes gives one alert fewer. Nothing changes between runs. Every version I tried from v3.18.0 to v3.22.0 does this. Every version I tried up to v3.17.1 gives the same output every time.

The smallest repro I could make (five tiny files):

.vale.ini

StylesPath = styles

[*.txt]
BasedOnStyles = Test

styles/Test/Quotes.yml

extends: consistency
message: "Inconsistent quote style '%s'"
level: error
nonword: true
either:
  '"': '”'
  "'": '’'

styles/Test/Para.yml (it matches nothing in this input; it only has to be enabled)

extends: existence
message: "never fires"
level: error
scope: paragraph
tokens:
  - 'zzzqqq'

input.txt is a ", a ' and a ”, then a blank line, then ”’. It needs to be exactly 13 bytes with no trailing newline:

printf '\042\047\342\200\235\n\n\342\200\235\342\200\231' > input.txt

repro.sh runs Vale N times and counts how many different outputs it sees:

#!/bin/sh
n=${1:-200}
i=0
while [ "$i" -lt "$n" ]; do
  vale --output=line input.txt | sort | tr '\n' '|'
  echo
  i=$((i + 1))
done | sort | uniq -c | sort -rn

From the folder that holds .vale.ini, run sh repro.sh 300. I expected one line of output. On 3.22.0 I get two:

 243 input.txt:1:3:Test.Quotes:Inconsistent quote style '”'|input.txt:3:1:Test.Quotes:Inconsistent quote style '”'|input.txt:3:2:Test.Quotes:Inconsistent quote style '’'|
  57 input.txt:1:3:Test.Quotes:Inconsistent quote style '”'|input.txt:3:2:Test.Quotes:Inconsistent quote style '’'|

The second line is the same as the first but without the 3:1 alert. In three batches of 300 runs, 57, 66 and 82 runs came out short, so roughly one run in four or five. I can't tell whether 2 or 3 alerts is the "right" answer here. The problem is only that it changes from run to run.

What I found

Same repro, official release binaries:

In this example it needs both of these at the same time: a consistency rule with two pairs under either, and an enabled rule with scope: paragraph. In each of these variants the output was identical on every run (at least 150 runs each):

• delete Para.yml
• keep only the " pair in Quotes.yml
• split Quotes.yml into two rules with one pair each
• use input.md with [*.md] in .vale.ini instead of .txt

GOMAXPROCS=1 sh repro.sh 300 still varies (57 short runs), so I don't think it is a race between threads, though I haven't ruled out everything.

My guess at the cause. I suspect the order of the two pairs matters. NewConsistency builds its steps with for v1, v2 := range rule.Either, a Go map, and Go doesn't fix the order of a map, so the pairs can end up in either order from one run to the next. Both steps report under the same check name, and the missing alert is always the second ”, which is the only text reported twice under that name. That map loop is also in v3.17.1, so on its own it isn't new. What is new in v3.18.0 is the per-check bookkeeping in File.AddAlert (chkMasked in internal/core/file.go, which I couldn't find in the v3.17.1 tag). I haven't built Vale or bisected, so I don't know if that is the cause.

Related

#476 was a similar problem for substitution, where the advice was to split the patterns into separate files. Splitting also makes this case stable, as listed above. But the two pairs in a consistency rule don't overlap each other, so I'd expect their order not to matter, and this only started in v3.18.0.

I used Claude Code to help investigate this and to write it up. The repro is self-contained.