Improve test reliability

[mbien]

tests are still failing too often. Since we don't have CI stats for github, I tried to crawl workflow results using a rudimentary bash script calling into the github API.

The script scans workflow runs and filters merges to master which failed, then scans re-runs too and prints the failed steps. Rest is just sort and uniq.

     14 "ide/project.ant" - done
     13 "java.hints batch1"
     11 "java.file.launcher" - done
     10 "ide/css.editor"
      7 "groovy/groovy.editor" - done
      6 "webcommon/html.knockout"
      3 "platform/o.n.bootstrap"
      2 "java/maven"
      2 "java.hints batch2"
      1 "web.beans"
      1 "platform/openide.nodes"
      1 "java.editor"

note: this does not take the retry script into account which might be active on some steps

here the results with the links pointing directly to the workflow run:

Details

run: https://github.com/apache/netbeans/actions/runs/12936028241 (4 attempts)

• groovy/groovy.editor
• groovy/groovy.editor
• groovy/groovy.editor

run: https://github.com/apache/netbeans/actions/runs/12934523662 (2 attempts)

• java.file.launcher

run: https://github.com/apache/netbeans/actions/runs/12933263724 (3 attempts)

• ide/project.ant
• ide/project.ant
• java.editor
• java/maven

run: https://github.com/apache/netbeans/actions/runs/12933078681 (4 attempts)

• java.hints batch1
• java.hints batch2
• platform/o.n.bootstrap
• java.hints batch1
• java.hints batch1

run: https://github.com/apache/netbeans/actions/runs/12912321783 (3 attempts)

• java.file.launcher
• java.hints batch1
• platform/o.n.bootstrap
• ide/css.editor
• ide/css.editor

run: https://github.com/apache/netbeans/actions/runs/12872488235 (3 attempts)

• webcommon/html.knockout
• webcommon/html.knockout

run: https://github.com/apache/netbeans/actions/runs/12869051716 (3 attempts)

• java.hints batch1
• ide/css.editor
• webcommon/html.knockout

run: https://github.com/apache/netbeans/actions/runs/12857668868 (2 attempts)

• ide/css.editor

run: https://github.com/apache/netbeans/actions/runs/12851818621 (3 attempts)

• ide/css.editor
• webcommon/html.knockout

run: https://github.com/apache/netbeans/actions/runs/12846158264 (2 attempts)

• ide/project.ant
• ide/css.editor

run: https://github.com/apache/netbeans/actions/runs/12836525165 (2 attempts)

• java.file.launcher

run: https://github.com/apache/netbeans/actions/runs/12836146985 (2 attempts)

• java/maven
• java.file.launcher

run: https://github.com/apache/netbeans/actions/runs/12826345964 (3 attempts)

• ide/project.ant
• platform/o.n.bootstrap
• webcommon/html.knockout
• ide/project.ant

run: https://github.com/apache/netbeans/actions/runs/12825920011 (3 attempts)

• ide/project.ant
• groovy/groovy.editor
• ide/project.ant

run: https://github.com/apache/netbeans/actions/runs/12825908774 (2 attempts)

• ide/project.ant
• java.file.launcher

run: https://github.com/apache/netbeans/actions/runs/12795877604 (2 attempts)

• ide/project.ant
• java.hints batch1
• java.file.launcher
• ide/css.editor

run: https://github.com/apache/netbeans/actions/runs/12794975127 (2 attempts)

• groovy/groovy.editor

run: https://github.com/apache/netbeans/actions/runs/12791936642 (2 attempts)

• ide/project.ant
• groovy/groovy.editor

run: https://github.com/apache/netbeans/actions/runs/12765772367 (2 attempts)

• ide/project.ant
• java.hints batch1

run: https://github.com/apache/netbeans/actions/runs/12764904231 (2 attempts)

• ide/project.ant
• java.file.launcher
• platform/openide.nodes

run: https://github.com/apache/netbeans/actions/runs/12762278637 (3 attempts)

• java.file.launcher
• ide/css.editor
• ide/css.editor

run: https://github.com/apache/netbeans/actions/runs/12762263024 (3 attempts)

• java.hints batch1
• java.file.launcher
• java.file.launcher
• java.hints batch1

run: https://github.com/apache/netbeans/actions/runs/12749609504 (2 attempts)

• java.hints batch1
• java.hints batch2
• java.file.launcher

run: https://github.com/apache/netbeans/actions/runs/12722000825 (4 attempts)

• ide/project.ant
• web.beans
• groovy/groovy.editor
• java.hints batch1
• ide/css.editor
• java.hints batch1
• java.hints batch1

run: https://github.com/apache/netbeans/actions/runs/12721993365 (2 attempts)

• ide/project.ant
• webcommon/html.knockout

[sdedic]

I take java/maven and will try platform/o.n.bootstrap

[mbien]

i take a look at ide/project.ant

(testGetBuildScriptState: org.netbeans.spi.project.support.ant.GeneratedFilesHelperTest)

edit: found the issue - turns out it was a real bug: #8190

[mbien]

for groovy/groovy.editor -> #8191

[mbien]

for java.file.launcher -> #8193

[mbien]

extracted java.hints batch1 failures into separate issue since this might be a real bug/be more difficult to fix -> #8195