[5.x] Performance Optimizations for Stache and Query Operations

[hastinbe]

I've implemented performance optimizations for Statamic's Stache and query operations. Our ongoing proprietary CMS migration, combined with the team's need to load our full dataset, created a severe bottleneck. The Stache warm-up time was reaching 6 minutes, a process we were executing frequently during daily development. These changes focus on reducing redundant operations and improving algorithmic efficiency, resulting in a ~6x faster warm time and a ~50% memory reduction across 16k files.

Issues Addressed

Stache Warming

• Items were being loaded and parsed multiple times during warming
• File traversal could be more memory efficient
• Parallel processing available for CLI operations (using 'fork' driver)

Query Operations

• in_array() calls in filterWhereIn() and filterWhereNotIn() were doing linear searches
• Some repository methods weren't handling empty inputs efficiently
• Dictionary search operations could benefit from hash lookups

Changes Made

Stache Optimizations

• Load items once per store instead of once per index
• Cache items during paths() resolution to avoid double parsing
• Use RecursiveDirectoryIterator for more efficient file traversal
• Add parallel processing support for CLI operations (fork driver)
• Add new warming configuration options in config/stache.php

Query/Repository Optimizations

• Replace in_array() with isset(array_flip()) in filterWhereIn() and filterWhereNotIn()
• Add early returns for empty inputs in repository methods
• Optimize array operations to avoid unnecessary intermediate collections
• Improve BasicDictionary::matchesSearchQuery() with hash lookups

Performance Results

I ran benchmarks on a production dataset with 15,996 files to measure the impact:

Baseline:

• Warm Time: 328.45s
• Peak Memory: 829.77MB
• Cleared: 40.02s

After Optimizations:

• Warm Time: 53.07s (~6x faster)
• Peak Memory: 416.92MB (~50% reduction)
• Cleared: 517.46ms (~77x faster)

Query Operations:

• filterWhereIn() with 1000 items: ~78x faster with hash lookups
• BasicDictionary::matchesSearchQuery(): 1.29-1.63x faster depending on dataset size

The parallel processing improvement mainly applies to CLI operations using the 'fork' driver, as web requests can't fork processes.

Detailed Benchmark Breakdown

Metric	Baseline	Optimized	Improvement
Warm Time	328.45s	53.07s	~6x faster
Peak Memory	829.77MB	416.92MB	~50% reduction
Cleared	40.02s	517.46ms	~77x faster
Files	15,996	15,996	Same dataset

Optimization Progression

Step	Warm Time	Improvement
Baseline	328.45s	-
Single Item Load	270.79s	17.6% faster
Eliminate Double Parsing	232.15s	29.3% faster
Optimized File Traversal	172.81s	47.4% faster
Process-based + Redis	202.81s	38.3% faster
Concurrency + File	42.05s	87.2% faster

Total Improvement: 286.40s reduction in warm time

Configuration Options

New warming configuration section added to config/stache.php:

'warming' => [
    // Enable parallel store processing for faster warming on multi-core systems
    'parallel_processing' => env('STATAMIC_STACHE_PARALLEL_WARMING', false),

    // Maximum number of parallel processes (0 = auto-detect CPU cores)
    'max_processes' => env('STATAMIC_STACHE_MAX_PROCESSES', 0),

    // Minimum number of stores required to enable parallel processing
    'min_stores_for_parallel' => env('STATAMIC_STACHE_MIN_STORES_PARALLEL', 3),

    // Concurrency driver: 'process', 'fork', or 'sync'
    'concurrency_driver' => env('STATAMIC_STACHE_CONCURRENCY_DRIVER', 'process'),
],

These options allow fine-tuning of the parallel processing behavior based on your server environment and requirements.

Note: The fork driver is not available on Windows systems. On Windows, the system will automatically fall back to the process driver or sequential processing.

Implementation Notes

• All changes are backwards compatible with no breaking changes
• Existing tests pass without modification
• Proper fallbacks are in place for error conditions

[hastinbe]

Test script https://gist.github.com/hastinbe/d6ec020c019131c7490623ea217d6755

Edit: accidentally cancelled checks below can someone re-run?

[duncanmcclean]

Edit: accidentally cancelled checks below can someone re-run?

Done

[godismyjudge95]

First of all this is exactly what I've been wanting to work on but haven't had time, so thank you for getting a functional PR submitted!

I am going to leave a few comments with things I noted/implemented from my testing - feel free to disregard them.

Also, did you do any testing in regards to using a generator in the Traverser and then passing that all the way down? https://www.php.net/manual/en/language.generators.overview.php
I was thinking this would essentially make the entire thing one big loop rather than looping over the paths multiple times. Let me know and I can provide the code I already have for it.

Thanks again for submitting this.

[ryanmitchell]

love this, thanks for your work

[jasonvarga]

Amazing 🤗

[hastinbe]

Revision

• Loading all items once upfront broke parent-child relationships. So $item->parent() returned null
• RecursiveDirectoryIterator doesn't maintain the display order. Instead of reverting back to Filesystem::allFiles(), I used Symfony's Finder directly to preserve the memory efficiency of an iterator.
• If desired, this also means we can drop the Filesystem dependency for the Traverser (which I did)

Even without the Single Item Load improvement, we're still on par with about the same performance

Metric	Baseline	Optimized (Fixed)	Improvement
Warm Time	328.45s	46.00s	~7.1x faster
Peak Memory	829.77MB	401.5MB	~52% reduction
Clear Time	40.02s	562ms	~71x faster
Files	15,996	15,996	Same dataset

[jasonvarga]

Thanks for all this. Are you still working on it?

[hastinbe]

Thanks for all this. Are you still working on it?

Finished if there are no further suggestions from anyone