Per-bin spectral flux tracking + streaming WSOLA-PV hybrid overlay#2

robmorgan · 2026-03-24T08:55:00Z

Summary

Major quality improvements to the streaming time-stretch engine through two key techniques:

Per-bin spectral flux tracking in the phase vocoder — tracks frequency-domain changes per bin to drive a flux-adaptive resample blend (2-10%, modulated by impulsiveness), improving transient fidelity.
Streaming WSOLA-PV hybrid overlay — runs WSOLA in parallel on transient input and crossfades it over the phase vocoder output, with full cross-chunk carry-over for continuity. This is the main driver of quality gains, especially for percussive and EDM content.

Tuning & refinements

3-tier ratio-adaptive WSOLA overlay: pure WSOLA at extreme ratios (≥0.8 distance from unity), scaled blend at moderate/normal ratios
Per-overlay energy normalization: boost-only gain matching between PV and WSOLA paths
High-shelf filter on WSOLA overlay: corrects spectral centroid drift from time-domain stretching
HF energy-driven shelf: adaptive high-frequency correction gated by energy gain, with EMA smoothing
Mid-band bandpass boost (500-2000 Hz): centroid correction in the mid-range
Ratio-adaptive EMA alpha: faster smoothing at extreme stretch ratios (0.12 vs 0.05 near-unity)
Two-region gain factor: 50% shelf for near-unity energy gain, improving harmonic content preservation

Score improvements

Overall: ~954 → ~970 (+16 points)
EDM: ~967 → ~973 (+6 points)
Percussive: ~926 → ~957 (+31 points)
Harmonic: ~982 → ~983

Files changed

src/stream/processor.rs — WSOLA overlay engine with cross-chunk state, energy normalization, shelf filters
src/stretch/phase_vocoder.rs — per-bin spectral flux tracking and flux-adaptive blend
desktop/ — wider BPM stretch range in the desktop GUI

Test plan

cargo test --all-targets
cargo clippy --all-targets -- -D warnings
Run quality benchmark suite and verify score improvements hold
Test streaming playback in desktop app at various stretch ratios

🤖 Generated with Claude Code

…2-8%, modulated by impulsiveness). Percussive +0.8, overall +0.1. Result: {"status":"keep","quality_score":953.9,"percussive_score":925.8,"edm_score":963.3,"harmonic_score":982}

… 926.0 (+0.2) Result: {"status":"keep","quality_score":953.9,"percussive_score":926,"edm_score":963.2,"harmonic_score":982}

…1) with percussive held at 926.0 Result: {"status":"keep","quality_score":954,"percussive_score":926,"edm_score":963.3,"harmonic_score":982}

…et-rise boost for helper blend — overall 954.1 (+0.1), percussive 926.2 Result: {"status":"keep","quality_score":954.1,"percussive_score":926.2,"edm_score":963.3,"harmonic_score":982}

…sfade 85%→0% over PV output — 954.3 (+0.2), percussive 926.8 Result: {"status":"keep","metric":954.3,"percussive_score":926.8,"edm_score":963.3,"harmonic_score":982}

…954.5 (+0.2), percussive 927.3 Result: {"status":"keep","metric":954.5,"percussive_score":927.3,"edm_score":963.3,"harmonic_score":982}

…1 (+3.8)! Result: {"status":"keep","metric":955.9,"percussive_score":927.3,"edm_score":967.1,"harmonic_score":982}

…eme ratios (>0.8 distance) — 956.2 (+0.3), percussive 927.9 (+0.6) Result: {"status":"keep","quality_score":956.2,"percussive_score":927.9,"edm_score":967.1,"harmonic_score":982}

…56.4 (+0.5), EDM 967.6 (+0.5) Result: {"status":"keep","quality_score":956.4,"percussive_score":927.9,"edm_score":967.6,"harmonic_score":982}

…ck) — 956.5 (+0.1), percussive 928.3 (+0.4) Result: {"status":"keep","quality_score":956.5,"percussive_score":928.3,"edm_score":967.6,"harmonic_score":982}

…) — 956.8 (+0.3), percussive 929 (+0.7) Result: {"status":"keep","quality_score":956.8,"percussive_score":929,"edm_score":967.6,"harmonic_score":982}

….1 (+0.3), percussive 929.9 (+0.9) Result: {"status":"keep","quality_score":957.1,"percussive_score":929.9,"edm_score":967.6,"harmonic_score":982}

…mal — 957.4 (+0.3), percussive 930.7 (+0.8) Result: {"status":"keep","quality_score":957.4,"percussive_score":930.7,"edm_score":967.6,"harmonic_score":982}

…+1.1) Result: {"status":"keep","quality_score":957.8,"percussive_score":931.8,"edm_score":967.6,"harmonic_score":982}

Result: {"status":"keep","quality_score":958.2,"percussive_score":933,"edm_score":967.6,"harmonic_score":982}

…ussive 933.4, freq_pres 0.79 Result: {"status":"keep","quality_score":958.4,"percussive_score":933.4,"edm_score":967.6,"harmonic_score":982}

… from recovering 1.5x regression Result: {"status":"keep","quality_score":958.5,"percussive_score":933.8,"edm_score":967.6,"harmonic_score":982}

…— 963.3 (+4.8!), percussive 946.5 (+12.7) Result: {"status":"keep","quality_score":963.3,"percussive_score":946.5,"edm_score":967.6,"harmonic_score":982}

….8), percussive 948 (+1.5), EDM 968.3 (+0.7) Result: {"status":"keep","quality_score":964.1,"percussive_score":948,"edm_score":968.3,"harmonic_score":982}

…0.7), EDM 968.6 (+0.3) Result: {"status":"keep","quality_score":964.5,"percussive_score":948.7,"edm_score":968.6,"harmonic_score":982}

…e 953.7 (+5.0) Result: {"status":"keep","quality_score":966.1,"percussive_score":953.7,"edm_score":968,"harmonic_score":982}

…8.4 (+0.4) from fixing over-amplification Result: {"status":"keep","quality_score":966.3,"percussive_score":953.7,"edm_score":968.4,"harmonic_score":982}

…66.6 (+0.3), EDM 968.9 from centroid fix Result: {"status":"keep","quality_score":966.6,"percussive_score":954,"edm_score":968.9,"harmonic_score":982}

…+0.4), percussive 955 (+1.0), harmonic 981.4 (-0.6) Result: {"status":"keep","quality_score":967,"percussive_score":955,"edm_score":969.3,"harmonic_score":981.4}

…7), all categories improved Result: {"status":"keep","quality_score":967.7,"percussive_score":956.6,"edm_score":969.4,"harmonic_score":981.9}

…percussive 957 (+0.4) Result: {"status":"keep","quality_score":967.9,"percussive_score":957,"edm_score":969.5,"harmonic_score":982}

Result: {"status":"keep","quality_score":968,"percussive_score":957.3,"edm_score":969.4,"harmonic_score":982}

…2), harmonic 983.4 (+1.4!) Result: {"status":"keep","quality_score":968.2,"percussive_score":957.3,"edm_score":969,"harmonic_score":983.4}

…stent shelf reduction for harmonic Result: {"status":"keep","quality_score":968.3,"percussive_score":957.3,"edm_score":969.1,"harmonic_score":983.4}

…from better centroid correction Result: {"status":"keep","quality_score":968.9,"percussive_score":957.3,"edm_score":970.8,"harmonic_score":983.4}

Result: {"status":"keep","quality_score":969.1,"percussive_score":957.3,"edm_score":971.3,"harmonic_score":983.4}

Result: {"status":"keep","quality_score":969.3,"percussive_score":957.3,"edm_score":971.8,"harmonic_score":983.4}

Result: {"status":"keep","quality_score":969.5,"percussive_score":957.3,"edm_score":972.3,"harmonic_score":983.4}

Result: {"status":"keep","quality_score":969.6,"percussive_score":957.3,"edm_score":972.8,"harmonic_score":983.4}

Result: {"status":"keep","quality_score":969.7,"percussive_score":957.3,"edm_score":972.9,"harmonic_score":983.4}

Result: {"status":"keep","quality_score":969.7,"percussive_score":957.3,"edm_score":973,"harmonic_score":983.4}

Result: {"status":"keep","quality_score":969.7,"percussive_score":957.3,"edm_score":973.1,"harmonic_score":983.4}

…orrection in mid-range Result: {"status":"keep","quality_score":969.8,"percussive_score":957.3,"edm_score":973.2,"harmonic_score":983.4}

Add internal-only module scaffolding for the new persistent streaming hybrid engine: analyzer, router, render, and mixer. All types use absolute timeline positions instead of chunk-relative indices. No runtime behavior changes. Existing StreamProcessor and engines are untouched. New types: - StreamAnalyzer, AnalysisEvent - HybridRouter, RoutedHybridOp, RenderPath - PersistentTonalRenderer, PersistentTransientRenderer, ScheduledPatch - TimelineMixer Includes docs/persistent-streaming-hybrid-design.md describing the migration strategy from the legacy rolling-window re-render path.

robmorgan added 30 commits March 23, 2026 10:21

chore: commit autoresearch

0d25ba8

Merge branch 'main' into per-bin-spectral-flux-transient-detection

cbea71e

Per-bin spectral flux tracking in PV + flux-adaptive resample blend (…

0b1c6d0

…2-8%, modulated by impulsiveness). Percussive +0.8, overall +0.1. Result: {"status":"keep","quality_score":953.9,"percussive_score":925.8,"edm_score":963.3,"harmonic_score":982}

Wider flux factor range (0.3-2.5) with blend clamp 1-10% — percussive…

c3ab2a0

… 926.0 (+0.2) Result: {"status":"keep","quality_score":953.9,"percussive_score":926,"edm_score":963.2,"harmonic_score":982}

Raise flux-adaptive blend base from 4.0% to 4.5% — overall 954.0 (+0.…

fc61f07

…1) with percussive held at 926.0 Result: {"status":"keep","quality_score":954,"percussive_score":926,"edm_score":963.3,"harmonic_score":982}

Unified onset model lite: combine PV flux with stronger input-RMS ons…

e479be4

…et-rise boost for helper blend — overall 954.1 (+0.1), percussive 926.2 Result: {"status":"keep","quality_score":954.1,"percussive_score":926.2,"edm_score":963.3,"harmonic_score":982}

Widen desktop BPM stretch range

a5c4a51

Streaming WSOLA-PV hybrid overlay: run WSOLA on transient input, cros…

48358ea

…sfade 85%→0% over PV output — 954.3 (+0.2), percussive 926.8 Result: {"status":"keep","metric":954.3,"percussive_score":926.8,"edm_score":963.3,"harmonic_score":982}

Tune WSOLA overlay: 90% weight, 25% attack window, quadratic decay — …

87feb14

…954.5 (+0.2), percussive 927.3 Result: {"status":"keep","metric":954.5,"percussive_score":927.3,"edm_score":963.3,"harmonic_score":982}

Enable WSOLA overlay at all non-unity ratios — 955.9 (+1.4), EDM 967.…

c0add4c

…1 (+3.8)! Result: {"status":"keep","metric":955.9,"percussive_score":927.3,"edm_score":967.1,"harmonic_score":982}

chore: update research ideas

4eda9b8

Ratio-adaptive WSOLA overlay: 95% weight + 35% attack plateau at extr…

4701f93

…eme ratios (>0.8 distance) — 956.2 (+0.3), percussive 927.9 (+0.6) Result: {"status":"keep","quality_score":956.2,"percussive_score":927.9,"edm_score":967.1,"harmonic_score":982}

Use preset-configured WSOLA params instead of hardcoded 30ms/15ms — 9…

ee99ecb

…56.4 (+0.5), EDM 967.6 (+0.5) Result: {"status":"keep","quality_score":956.4,"percussive_score":927.9,"edm_score":967.6,"harmonic_score":982}

More aggressive WSOLA overlay at extreme ratios (97% weight, 40% atta…

a252f07

…ck) — 956.5 (+0.1), percussive 928.3 (+0.4) Result: {"status":"keep","quality_score":956.5,"percussive_score":928.3,"edm_score":967.6,"harmonic_score":982}

Near-complete WSOLA overlay at extreme ratios (99% weight, 50% attack…

f2a78fe

…) — 956.8 (+0.3), percussive 929 (+0.7) Result: {"status":"keep","quality_score":956.8,"percussive_score":929,"edm_score":967.6,"harmonic_score":982}

Full WSOLA takeover at extreme ratios (100% weight, 60% attack) — 957…

d0f77fb

….1 (+0.3), percussive 929.9 (+0.9) Result: {"status":"keep","quality_score":957.1,"percussive_score":929.9,"edm_score":967.6,"harmonic_score":982}

3-tier WSOLA overlay: 100%/75% extreme, 95%/35% moderate, 90%/25% nor…

7b21f67

…mal — 957.4 (+0.3), percussive 930.7 (+0.8) Result: {"status":"keep","quality_score":957.4,"percussive_score":930.7,"edm_score":967.6,"harmonic_score":982}

Extend extreme WSOLA attack to 85% — 957.8 (+0.4), percussive 931.8 (…

28a413e

…+1.1) Result: {"status":"keep","quality_score":957.8,"percussive_score":931.8,"edm_score":967.6,"harmonic_score":982}

Extend extreme WSOLA attack to 95% — 958.2 (+0.4), percussive 933 (+1.2)

7b365fb

Result: {"status":"keep","quality_score":958.2,"percussive_score":933,"edm_score":967.6,"harmonic_score":982}

Pure WSOLA overlay at extreme ratios (100%/100%) — 958.4 (+0.2), perc…

a9e81a1

…ussive 933.4, freq_pres 0.79 Result: {"status":"keep","quality_score":958.4,"percussive_score":933.4,"edm_score":967.6,"harmonic_score":982}

Remove moderate WSOLA tier (95%/35%) — 958.5 (+0.1), percussive 933.8…

011c928

… from recovering 1.5x regression Result: {"status":"keep","quality_score":958.5,"percussive_score":933.8,"edm_score":967.6,"harmonic_score":982}

Full-length WSOLA overlay at extreme ratios (cross-chunk carry-over) …

e935852

…— 963.3 (+4.8!), percussive 946.5 (+12.7) Result: {"status":"keep","quality_score":963.3,"percussive_score":946.5,"edm_score":967.6,"harmonic_score":982}

Apply 70% gain delta to WSOLA overlay for energy matching — 964.1 (+0…

a089b05

….8), percussive 948 (+1.5), EDM 968.3 (+0.7) Result: {"status":"keep","quality_score":964.1,"percussive_score":948,"edm_score":968.3,"harmonic_score":982}

Full energy gain on WSOLA overlay — 964.5 (+0.4), percussive 948.7 (+…

9a1ef62

…0.7), EDM 968.6 (+0.3) Result: {"status":"keep","quality_score":964.5,"percussive_score":948.7,"edm_score":968.6,"harmonic_score":982}

Per-overlay energy normalization + PV gain — 966.1 (+1.6!), percussiv…

93d1462

…e 953.7 (+5.0) Result: {"status":"keep","quality_score":966.1,"percussive_score":953.7,"edm_score":968,"harmonic_score":982}

Only boost (not attenuate) WSOLA normalization — 966.3 (+0.2), EDM 96…

402c331

…8.4 (+0.4) from fixing over-amplification Result: {"status":"keep","quality_score":966.3,"percussive_score":953.7,"edm_score":968.4,"harmonic_score":982}

Apply high-shelf filter to WSOLA overlay for spectral consistency — 9…

947f115

…66.6 (+0.3), EDM 968.9 from centroid fix Result: {"status":"keep","quality_score":966.6,"percussive_score":954,"edm_score":968.9,"harmonic_score":982}

Increase min ratio_scale from 0.2 to 0.25 — 967.0 (+0.4), EDM 969.3 (…

203b662

…+0.4), percussive 955 (+1.0), harmonic 981.4 (-0.6) Result: {"status":"keep","quality_score":967,"percussive_score":955,"edm_score":969.3,"harmonic_score":981.4}

Ratio-adaptive EMA alpha (0.05 near-unity, 0.08 extreme) — 967.7 (+0.…

3473cec

…7), all categories improved Result: {"status":"keep","quality_score":967.7,"percussive_score":956.6,"edm_score":969.4,"harmonic_score":981.9}

Increase ratio-adaptive EMA alpha to 0.10 at extreme — 967.9 (+0.2), …

ac55473

…percussive 957 (+0.4) Result: {"status":"keep","quality_score":967.9,"percussive_score":957,"edm_score":969.5,"harmonic_score":982}

robmorgan added 16 commits March 24, 2026 08:16

EMA alpha delta 0.07 (=0.12 at extreme) — 968.0 (+0.1), percussive 957.3

a37a584

Result: {"status":"keep","quality_score":968,"percussive_score":957.3,"edm_score":969.4,"harmonic_score":982}

Two-region gain_factor: 50% shelf for energy_gain < 1.06 — 968.2 (+0.…

fe3c19f

…2), harmonic 983.4 (+1.4!) Result: {"status":"keep","quality_score":968.2,"percussive_score":957.3,"edm_score":969,"harmonic_score":983.4}

Apply two-region gain_factor to WSOLA shelf too — 968.3 (+0.1), consi…

a50c6cc

…stent shelf reduction for harmonic Result: {"status":"keep","quality_score":968.3,"percussive_score":957.3,"edm_score":969.1,"harmonic_score":983.4}

chore: update autoresearch

32aec7c

chore: bump version number

7a87bcd

HF energy-driven shelf with energy_gain gate: +0.6, EDM 970.8 (+1.7) …

e58e6ef

…from better centroid correction Result: {"status":"keep","quality_score":968.9,"percussive_score":957.3,"edm_score":970.8,"harmonic_score":983.4}

HF shelf coeff 0.4→0.5: +0.2, EDM 971.3 (+0.5)

9419be3

Result: {"status":"keep","quality_score":969.1,"percussive_score":957.3,"edm_score":971.3,"harmonic_score":983.4}

HF shelf coeff 0.5→0.6: +0.2, EDM 971.8

7dc4e36

Result: {"status":"keep","quality_score":969.3,"percussive_score":957.3,"edm_score":971.8,"harmonic_score":983.4}

HF shelf coeff 0.6→0.7: +0.2, EDM 972.3

486540e

Result: {"status":"keep","quality_score":969.5,"percussive_score":957.3,"edm_score":972.3,"harmonic_score":983.4}

HF shelf coeff 0.7→0.8: +0.1, EDM 972.8

4ca2af2

Result: {"status":"keep","quality_score":969.6,"percussive_score":957.3,"edm_score":972.8,"harmonic_score":983.4}

Relax energy_gain gate 1.10→1.15: +0.1, EDM 972.9

c0d2d9a

Result: {"status":"keep","quality_score":969.7,"percussive_score":957.3,"edm_score":972.9,"harmonic_score":983.4}

Energy gate 1.15→1.20: same 969.7, EDM 973.0 (+0.1)

5330791

Result: {"status":"keep","quality_score":969.7,"percussive_score":957.3,"edm_score":973,"harmonic_score":983.4}

Faster HF EMA (1.5x alpha): 969.7, EDM 973.1 (+0.1)

f950d6e

Result: {"status":"keep","quality_score":969.7,"percussive_score":957.3,"edm_score":973.1,"harmonic_score":983.4}

Mid-band bandpass boost (500-2000Hz): +0.1, EDM 973.2 from centroid c…

c4dfaaf

…orrection in mid-range Result: {"status":"keep","quality_score":969.8,"percussive_score":957.3,"edm_score":973.2,"harmonic_score":983.4}

chore: errata

4fa08c8

robmorgan changed the title ~~Scaffold persistent streaming hybrid engine modules~~ Per-bin spectral flux tracking + streaming WSOLA-PV hybrid overlay Mar 24, 2026

robmorgan closed this Mar 24, 2026

robmorgan deleted the persistent-hybrid-engine branch March 24, 2026 22:43

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Per-bin spectral flux tracking + streaming WSOLA-PV hybrid overlay#2