Skip to content

Commit d2eb79d

Browse files
authored
[PoC] Parallel PHPUnit runner in Go (~4x faster) + fix phpVersion leak between test classes (#8349)
1 parent f3a8273 commit d2eb79d

6 files changed

Lines changed: 340 additions & 0 deletions

File tree

Lines changed: 56 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,56 @@
1+
name: Benchmark PHPUnit
2+
3+
# A/B wall-time benchmark: serial phpunit vs the Go parallel runner.
4+
# Scheduled only (not on pull requests); runs every 2 hours on ubuntu + windows.
5+
# Reminder: cron fires only from the default branch, so this starts running
6+
# once the file is on `main`.
7+
on:
8+
schedule:
9+
- cron: '0 */2 * * *'
10+
workflow_dispatch: null
11+
12+
env:
13+
COMPOSER_ROOT_VERSION: "dev-main"
14+
15+
jobs:
16+
benchmark:
17+
strategy:
18+
fail-fast: false
19+
matrix:
20+
os: [ubuntu-latest, windows-latest]
21+
php-versions: ['8.4', '8.5']
22+
23+
runs-on: ${{ matrix.os }}
24+
timeout-minutes: 15
25+
26+
name: benchmark (${{ matrix.os }})
27+
steps:
28+
- uses: actions/checkout@v5
29+
30+
-
31+
uses: shivammathur/setup-php@v2
32+
with:
33+
php-version: ${{ matrix.php-versions }}
34+
coverage: none
35+
ini-values: zend.assertions=1
36+
37+
- uses: "ramsey/composer-install@v4"
38+
39+
- uses: actions/setup-go@v5
40+
with:
41+
go-version: 'stable'
42+
43+
- name: Serial phpunit
44+
shell: bash
45+
run: |
46+
printf '| platform | php | mode | wall time |\n| --- | --- | --- | --- |\n' >> "$GITHUB_STEP_SUMMARY"
47+
start=$SECONDS
48+
vendor/bin/phpunit
49+
echo "| ${{ matrix.os }} | ${{ matrix.php-versions }} | serial | $((SECONDS - start))s |" >> "$GITHUB_STEP_SUMMARY"
50+
51+
- name: Go parallel runner
52+
shell: bash
53+
run: |
54+
start=$SECONDS
55+
go run ./utils-tests-runner/main.go
56+
echo "| ${{ matrix.os }} | ${{ matrix.php-versions }} | go-runner | $((SECONDS - start))s |" >> "$GITHUB_STEP_SUMMARY"

‎src/Testing/PHPUnit/AbstractRectorTestCase.php‎

Lines changed: 5 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -30,6 +30,7 @@
3030
use Rector\Testing\Fixture\FixtureSplitter;
3131
use Rector\Testing\PHPUnit\ValueObject\RectorTestResult;
3232
use Rector\Util\Reflection\PrivatesAccessor;
33+
use Rector\ValueObject\PhpVersion;
3334

3435
/**
3536
* @api used by public
@@ -63,6 +64,10 @@ public static function tearDownAfterClass(): void
6364
SimpleParameterProvider::setParameter(Option::NEW_LINE_ON_FLUENT_CALL, false);
6465

6566
SimpleParameterProvider::setParameter(Option::TREAT_CLASSES_AS_FINAL, false);
67+
68+
// reset PHP version to the test default, so a version-bound test class
69+
// does not leak its phpVersion() into the next class in the same process
70+
SimpleParameterProvider::setParameter(Option::PHP_VERSION_FEATURES, PhpVersion::PHP_10);
6671
}
6772

6873
protected function setUp(): void

‎utils-tests-runner/.gitignore‎

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,2 @@
1+
/fast-phpunit
2+
/fast-phpunit.exe

‎utils-tests-runner/README.md‎

Lines changed: 93 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,93 @@
1+
# fast-phpunit (proof of concept)
2+
3+
Run the PHPUnit suite **~4x faster** by splitting test classes across parallel
4+
workers that each boot PHP **once** and run many classes in a single process.
5+
6+
## Speed — CI, per platform (GitHub runners, 4 vCPU, full suite)
7+
8+
Serial `vendor/bin/phpunit` vs this runner, measured on the CI run step:
9+
10+
| Platform | Serial | Go runner | Speedup |
11+
| --- | --- | --- | --- |
12+
| ubuntu-latest | 31s | **16s** | ~1.9x |
13+
| windows-latest | **116s** | **67s** | **~1.7x** |
14+
15+
Windows is the slow platform (~3.7x slower than Ubuntu serial); the runner cuts
16+
~49s off it. Both run the full suite and pass.
17+
18+
## Speed — local (24-core host, full suite)
19+
20+
Full suite: **685 test classes / 5833 fixtures**.
21+
22+
| Mode | Wall time | vs serial |
23+
| --- | --- | --- |
24+
| Serial `vendor/bin/phpunit` (1 process) | **38.3s** | 1.0x |
25+
| `fast-phpunit -p 8` | ~11s | ~3.5x |
26+
| `fast-phpunit -p 12` | **~9s** | **~4.2x** |
27+
| `fast-phpunit -p 24` | ~9.8s | ~3.9x |
28+
29+
Subset — `rules-tests/CodeQuality` (86 classes / 771 fixtures):
30+
31+
| Mode | Wall time |
32+
| --- | --- |
33+
| Serial (1 process) | 5.71s |
34+
| Process-per-class, `xargs -P8` | 8.55s (**slower than serial**) |
35+
| `fast-phpunit -p 8` | **2.43s** |
36+
37+
Sweet spot is ~12 workers; more does not help, because the heaviest chunk
38+
bounds wall time and 20+ concurrent PHP processes start contending.
39+
40+
## Why it is faster
41+
42+
Bootstrap dominates a Rector test class, not the assertions:
43+
44+
| Step | Time |
45+
| --- | --- |
46+
| Boot only (container build, 0 tests) | ~0.23s — fixed, per process |
47+
| One class, 27 fixtures (warm) | 0.92s → ~26ms/fixture |
48+
49+
Average class has ~7 fixtures, so **bootstrap is ~56% of an average class's run
50+
time**. Any runner that spawns a fresh process per class pays that 0.23s boot
51+
685 times — which is why process-per-class parallelism is slower than serial
52+
(see subset table).
53+
54+
This runner splits classes into N chunks balanced by fixture count, and each
55+
worker runs its whole chunk in one warm process — so the container is built N
56+
times, not 685 times.
57+
58+
## Usage
59+
60+
```bash
61+
cd utils-tests-runner && go build -o fast-phpunit .
62+
cd ..
63+
utils-tests-runner/fast-phpunit -p 12 # whole suite
64+
utils-tests-runner/fast-phpunit -p 8 rules-tests/CodeQuality # a subtree
65+
```
66+
67+
Flags: `-p` workers (default = CPU count), `-bin` phpunit path.
68+
69+
## Isolation — required to make it correct
70+
71+
Two shared-state issues surface when many classes share a process; both are
72+
handled so any chunking is safe.
73+
74+
1. **Shared temp cache (cross-process).** Rector caches parsed files under
75+
`sys_get_temp_dir()/rector_cached_files`; parallel processes racing that
76+
directory throw `Failed to open directory` / `Directory not empty`. Each
77+
worker gets its own `TMPDIR`.
78+
79+
2. **Leaked `phpVersion()` (in-process) — a latent bug, fixed here.**
80+
`phpVersion(...)` is stored in the static `SimpleParameterProvider` and was
81+
never reset between classes, so a version-bound class leaks its version into
82+
the next class in the same process — a version-less class then sees, e.g.,
83+
PHP 8.1 instead of the test default (`PhpVersion::PHP_10`) and produces wrong
84+
output. The serial suite passes only because of its class ordering; any
85+
reshuffle (this tool **or** paratest) can trigger it. Fixed in
86+
`AbstractRectorTestCase::tearDownAfterClass()` by resetting
87+
`PHP_VERSION_FEATURES` to the test default.
88+
89+
## Status
90+
91+
Proof of concept. Standalone Go helper that shells out to the existing
92+
`vendor/bin/phpunit`, so it does not change how tests are written or how CI
93+
runs. Pure Go, no `.php`, so it is invisible to ECS / PHPStan / Rector.

‎utils-tests-runner/go.mod‎

Lines changed: 3 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,3 @@
1+
module rector/fast-phpunit
2+
3+
go 1.26.4

‎utils-tests-runner/main.go‎

Lines changed: 181 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,181 @@
1+
// Command fast-phpunit runs the PHPUnit suite in parallel by splitting test
2+
// classes into N balanced "warm" chunks: each worker boots PHP once and runs
3+
// many test classes in a single process, so the container is built N times
4+
// instead of once per class (as tools that spawn a process per chunk do).
5+
//
6+
// Balancing is by fixture count, since Rector rule tests iterate one assertion
7+
// per .php.inc fixture, so fixture count approximates a class's runtime.
8+
package main
9+
10+
import (
11+
"flag"
12+
"fmt"
13+
"os"
14+
"os/exec"
15+
"path/filepath"
16+
"regexp"
17+
"runtime"
18+
"sort"
19+
"strings"
20+
"sync"
21+
"time"
22+
)
23+
24+
type testClass struct {
25+
path string
26+
weight int // fixture count, min 1
27+
}
28+
29+
var fixtureCountRe = regexp.MustCompile(`\.php\.inc$`)
30+
31+
func main() {
32+
workers := flag.Int("p", runtime.NumCPU(), "number of parallel workers")
33+
php := flag.String("php", "php", "php interpreter")
34+
// the real PHP entry script (runs cross-platform via `php`); vendor/bin/phpunit
35+
// is a shell/batch proxy on Windows and cannot be passed to php directly.
36+
phpunit := flag.String("bin", "vendor/phpunit/phpunit/phpunit", "phpunit entry script")
37+
flag.Parse()
38+
39+
dirs := flag.Args()
40+
if len(dirs) == 0 {
41+
dirs = []string{"rules-tests", "tests"}
42+
}
43+
44+
classes := discover(dirs)
45+
if len(classes) == 0 {
46+
fmt.Fprintln(os.Stderr, "no test classes found")
47+
os.Exit(1)
48+
}
49+
50+
chunks := balance(classes, *workers)
51+
52+
start := time.Now()
53+
failed := run(chunks, *php, *phpunit, *workers)
54+
elapsed := time.Since(start)
55+
56+
totalFixtures := 0
57+
for _, c := range classes {
58+
totalFixtures += c.weight
59+
}
60+
fmt.Printf("\n%d classes, %d fixtures, %d chunks, %d workers\n",
61+
len(classes), totalFixtures, len(chunks), *workers)
62+
fmt.Printf("wall time: %.2fs\n", elapsed.Seconds())
63+
64+
if failed > 0 {
65+
fmt.Printf("FAILED chunks: %d\n", failed)
66+
os.Exit(1)
67+
}
68+
fmt.Println("OK")
69+
}
70+
71+
// discover finds *Test.php files and weights each by sibling Fixture/ file count.
72+
func discover(dirs []string) []testClass {
73+
var classes []testClass
74+
for _, dir := range dirs {
75+
_ = filepath.WalkDir(dir, func(path string, d os.DirEntry, err error) error {
76+
if err != nil || d.IsDir() || !strings.HasSuffix(path, "Test.php") {
77+
return nil
78+
}
79+
classes = append(classes, testClass{path: path, weight: fixtureWeight(path)})
80+
return nil
81+
})
82+
}
83+
return classes
84+
}
85+
86+
func fixtureWeight(testPath string) int {
87+
fixtureDir := filepath.Join(filepath.Dir(testPath), "Fixture")
88+
entries, err := os.ReadDir(fixtureDir)
89+
if err != nil {
90+
return 1
91+
}
92+
count := 0
93+
for _, e := range entries {
94+
if !e.IsDir() && fixtureCountRe.MatchString(e.Name()) {
95+
count++
96+
}
97+
}
98+
if count < 1 {
99+
return 1
100+
}
101+
return count
102+
}
103+
104+
// balance greedily packs classes (heaviest first) into n bins, always adding to
105+
// the lightest bin. Minimizes the heaviest chunk, so wall time is bounded by the
106+
// slowest worker rather than by unlucky static splits.
107+
func balance(classes []testClass, n int) [][]testClass {
108+
sort.Slice(classes, func(i, j int) bool {
109+
return classes[i].weight > classes[j].weight
110+
})
111+
bins := make([][]testClass, n)
112+
loads := make([]int, n)
113+
for _, c := range classes {
114+
min := 0
115+
for i := 1; i < n; i++ {
116+
if loads[i] < loads[min] {
117+
min = i
118+
}
119+
}
120+
bins[min] = append(bins[min], c)
121+
loads[min] += c.weight
122+
}
123+
var out [][]testClass
124+
for _, b := range bins {
125+
if len(b) > 0 {
126+
out = append(out, b)
127+
}
128+
}
129+
return out
130+
}
131+
132+
func run(chunks [][]testClass, php, phpunit string, workers int) int {
133+
sem := make(chan struct{}, workers)
134+
var wg sync.WaitGroup
135+
var mu sync.Mutex
136+
failed := 0
137+
138+
for idx, chunk := range chunks {
139+
wg.Add(1)
140+
go func(idx int, chunk []testClass) {
141+
defer wg.Done()
142+
sem <- struct{}{}
143+
defer func() { <-sem }()
144+
145+
// Each worker gets its own temp dir so Rector's file cache
146+
// (sys_get_temp_dir()/rector_cached_files) and the fixture temp
147+
// dumper never race across processes. sys_get_temp_dir() reads
148+
// TMPDIR on Linux/macOS and TMP/TEMP on Windows, so set all three.
149+
tmp := filepath.Join(os.TempDir(), fmt.Sprintf("fast-phpunit-%d", idx))
150+
_ = os.MkdirAll(tmp, 0o755)
151+
defer os.RemoveAll(tmp)
152+
153+
// invoke via `php <phpunit>` so it works uniformly on Windows,
154+
// where vendor/bin/phpunit is not directly executable.
155+
args := make([]string, 0, len(chunk)+1)
156+
args = append(args, phpunit)
157+
for _, c := range chunk {
158+
args = append(args, c.path)
159+
}
160+
cmd := exec.Command(php, args...)
161+
cmd.Env = append(os.Environ(), "TMPDIR="+tmp, "TMP="+tmp, "TEMP="+tmp)
162+
out, err := cmd.CombinedOutput()
163+
if err != nil {
164+
mu.Lock()
165+
failed++
166+
fmt.Printf("chunk FAILED (%d classes): %v\n%s\n", len(chunk), err, tail(string(out), 15))
167+
mu.Unlock()
168+
}
169+
}(idx, chunk)
170+
}
171+
wg.Wait()
172+
return failed
173+
}
174+
175+
func tail(s string, lines int) string {
176+
parts := strings.Split(strings.TrimRight(s, "\n"), "\n")
177+
if len(parts) > lines {
178+
parts = parts[len(parts)-lines:]
179+
}
180+
return strings.Join(parts, "\n")
181+
}

0 commit comments

Comments
 (0)