TL;DR

I extended DAF Online with real-time Frequency Altered Feedback (FAF). FAF pitch-shifts your voice in your headphones while you speak, triggering the brain’s choral effect to reduce stuttering. Getting it right took three algorithm iterations, some careful latency accounting, and a custom jitter benchmark that works around a non-obvious limitation of AudioWorkletGlobalScope. This post covers all of it.

What is Frequency Altered Feedback?

Delayed Auditory Feedback (DAF) slows speech by creating a timing mismatch between articulation and perception. I covered DAF in the previous post.

Frequency Altered Feedback (FAF) attacks the same problem through a different mechanism. It shifts the pitch of your voice in your headphones by a ratio $r$, where

$$r = 2^{s/12}$$

and $s$ is the semitone shift. A shift of 3-6 semitones is enough to trigger the choral effect: your brain perceives itself as speaking alongside another voice and drops into a group-speaker processing mode, which disengages the feedback loop that drives stuttering.

The two mechanisms are neurally independent. DAF acts on auditory-motor timing pathways. FAF acts on the choral speech cortical network. Clinical devices like SpeechEasy combine both, which is why DAF Online now supports them together.

Clinical semitone ranges

The ±300-cent range mode in the app’s cents UI exists specifically for PSR experiments.

FAF and DAF engage different neural pathways and their effects add up. In trials, the combination outperforms either alone, which is why clinical hardware uses both.

The Wrong Algorithm First: WSOLA

My initial implementation used WSOLA (Waveform Similarity Overlap-Add). The algorithm advances its input read head at a rate proportional to $r$:

this._inR += H * r;

For $r > 1$ (pitch up), the read head races ahead of incoming audio. The input buffer drains faster than new samples arrive. For $r < 1$, it falls behind. In both cases pitch and delay are coupled: setting $+8$ semitones audibly shortened the DAF delay; $-8$ semitones stretched it. This is unsuitable.

Fixed-Anchor OLA: Decoupling Pitch from Delay

The fix is to remove the drifting read head. Every synthesis hop, the analysis position is anchored at a fixed offset behind the write head:

$$\text{anchor} = w_{in} - L$$

where $w_{in}$ is the input write head and $L$ is the lookback depth. Pitch is shifted by resampling within the grain: a span of $G \cdot r$ input samples is linearly interpolated onto $G$ output samples.

$$\text{srcPos}(i) = \text{anchor} + \frac{i}{G - 1} \cdot G \cdot r, \quad i \in [0, G-1]$$

• $r > 1:\quad$ $G \cdot r > G$ — more input samples compressed into $G$ outputs → higher frequency
• $r = 1:\quad$ $G \cdot r = G$ — identity
• $r < 1:\quad$ $G \cdot r < G$ — fewer input samples stretched across $G$ outputs → lower frequency

The anchor is fixed relative to the write head. The delay is invariant under changes to $r$.

Choosing $R_{max}$ and the lookback depth

The readSpan $G \cdot r$ grows with $r$. At maximum shift ($r = R_{max} = 2.0$, i.e. +12 semitones), the grain reads $512$ input samples. The lookback must guarantee all of them are already in the ring buffer:

$$L = \lceil G \cdot R_{max} \rceil + 2 = \lceil 256 \times 2.0 \rceil + 2 = 514 \text{ samples} \approx 10.7 \text{ ms at 48 kHz}$$

The $+2$ guard ensures the linear interpolation never reads past the write head at exactly $r = R_{max}$. With anchor at $w_{in} - 514$, the maximum source position is:

$$\text{srcPos}_{max} = (w_{in} - 514) + 256 \times 2.0 = w_{in} - 2$$

leaving a two-sample margin before the write head. Note the sizing criterion flipped from the old design: formerly the lookback was sized for the smallest reachable ratio ($R_{min}$, governing how far back the read head could stray); now it is sized for the largest reachable ratio ($R_{max}$, governing how wide the read span can grow).

Amplitude normalisation

With fixed grain size $G = 256$ and fixed synthesis hop $H = 128$ (50% overlap), the Hann COLA sum is approximately 1 regardless of pitch ratio $r$. No $r$-dependent gain correction is required; grain amplitude is left at unity (1.0).

This differs from variable-hop schemes — including the PSOLA processor below — where the synthesis hop $\lfloor T_0 / r \rfloor$ varies with both $T_0$ and $r$, causing the COLA sum to vary with $r$ and requiring an explicit $1/r$ correction to maintain consistent output level.

Why Switch to PSOLA?

Fixed-anchor OLA solves the delay-coupling problem and runs cheaply. But it has a fundamental quality limit: grain boundaries don’t align to the signal’s periodicity.

A 256-sample grain cut from a 150 Hz voice at 48 kHz captures 0.8 pitch periods. The next grain starts at a random phase of the next cycle. The Hann window smooths the transition but cannot eliminate the phase discontinuity, which manifests as metallic flutter on sustained vowels, audible at shifts above ±2 semitones.

For stuttering therapy, sustained vowels at 3-6 semitones are exactly the use case.

TD-PSOLA (Time-Domain Pitch Synchronous Overlap-Add) fixes this by aligning grain boundaries to the signal’s own pitch periods. Each grain is exactly $2T_0$ samples, centered on a pitch mark (glottal closure instant). The synthesis hop is $T_0 / r$, which is what changes the perceived fundamental frequency. Because adjacent grains start at the same phase of their respective pitch cycles, overlap-add is phase-coherent and the output is perceptually smooth.

FD-PSOLA (Frequency-Domain PSOLA) is a more complex variant that applies a Fourier transform to each grain, manipulates the spectrum, then inverse transforms back to time domain. It can achieve higher quality at extreme shifts but is more computationally expensive and has higher latency due to the FFT window size. This latency makes it unsuitable for real-time FAF in the browser.

PSOLA Implementation

YIN pitch detection

YIN (de Cheveigné & Kawahara, 2002) estimates $T_0$ via the cumulative mean normalised difference function. The raw difference function is:

$$d(\tau) = \sum_{j=0}^{W-1} \left( x[j] - x[j - \tau] \right)^2$$

Normalised by the running cumulative mean:

$$d'(\tau) = \frac{d(\tau) \cdot \tau}{\sum_{k=1}^{\tau} d(k)}$$

$d'(\tau)$ tends toward 1.0 for aperiodic signals and dips below a threshold (0.15 here) at the true period. The running-sum formulation keeps normalisation at $O(1)$ per $\tau$. We re-run every 4 grains (~20 ms) since $T_0$ doesn’t change faster in natural speech, and smooth the estimate with a first-order IIR:

$$\hat{T}_0 \leftarrow 0.85 \cdot \hat{T}_0 + 0.15 \cdot T_0^{raw}$$

This prevents grain-size discontinuities at voiced/voiced boundaries.

Pitch mark detection

Once $T_0$ is estimated, the processor locates a pitch mark — the specific sample to centre the grain on — by searching near the analysis anchor for a negative-to-positive zero-crossing (a directed crossing). The requirement for directionality is non-obvious but important: a plain minimum-$|$sample$|$ search is direction-ambiguous and can alternate between positive- and negative-going crossings on successive grains, effectively halving the perceived pitch. The directed search eliminates this. If no qualifying crossing is found within the search window (which occurs on frames where the signal is locally flat), the processor falls back to the sample of minimum absolute value.

Unvoiced fallback

Fricatives, plosive releases, and silence have no $T_0$. YIN returns _voiced = false and the processor falls back to a 128-sample direct passthrough: input samples are copied to the output buffer unwindowed, with a short crossfade at voiced/unvoiced boundaries to preserve phase continuity. There is no $T_0$ to shift; attempting PSOLA on /s/ produces artefacts. Unlike a Hann-windowed OLA fallback, the direct copy adds negligible algorithmic overhead.

PSOLA latency and the voice type selector

The processor lookback covers $T_{max}$ samples — one maximum pitch period — which is the minimum history required for the grain-centred synthesis to access the full $2T_0$ analysis window:

$$T_{max} = \left\lfloor \frac{f_s}{f_{floor}} \right\rfloor, \quad L_{PSOLA} = \frac{T_{max}}{f_s} \times 1000 \text{ ms}$$

pitchFloor is passed at node construction via processorOptions, the correct mechanism for init-time configuration that cannot be updated via AudioParam automation:

new AudioWorkletNode(ctx, 'pitch-shifter-psola', {
  processorOptions: { pitchFloor: this._pitchFloor },
});

The voice type selector adjusts this value:

Matching voice type to your actual pitch range is not cosmetic; it directly controls the algorithm’s latency floor.

Loading Both Processors in One Blob

Both OLA and PSOLA are available in the app. Since each processor is a self-contained IIFE string, they are concatenated into a single blob and loaded with one addModule() call, with no extra Vite config and no public/ files:

const blob = new Blob([COMBINED_CODE], { type: 'application/javascript' });
await ctx.audioWorklet.addModule(URL.createObjectURL(blob));

The shared Hann table (1024 floats, 4 KB) is allocated once at module scope, not per node instance.

Switching modes (OLA ↔ PSOLA) requires tearing down existing nodes because processorOptions.pitchFloor is init-time only and cannot be changed post-construction. The “same count fast-path” (which updates only pitchRatio AudioParams without graph surgery) is explicitly bypassed on mode switches.

Multi-FAF: N Parallel Signals

The app supports $N$ simultaneous pitch-shifted signals. The graph topology:

$$\text{source} \rightarrow \left\{ \begin{array}{c} \text{fafNode}_1 \\ \vdots \\ \text{fafNode}_N \end{array} \right\} \rightarrow \text{sumGain}\!\left(\tfrac{1}{N}\right) \rightarrow \text{delayNode} \rightarrow \text{out}$$

The normalising gain at $1/N$ keeps perceived loudness constant as $N$ increases. Parallel worklet nodes don’t stack latency; they all process within the same 128-sample scheduler pass.

PSOLA nodes apply an internal $1/r$ gain correction (their synthesis hop $\lfloor T_0 / r \rfloor$ varies, so the COLA sum varies with $r$). OLA nodes use unity gain, since their hop and grain size are fixed. In both cases the $1/N$ sumGain composes correctly on top.

The Effective Delay Display

Total user-perceived delay follows the same equation as DAF:

$$k_{eff} = k_{user} + k_{sys}$$

where $k_{user}$ is the slider value and

$$k_{sys} = k_{floor} + k_{FAF}$$

with $k_{floor}$ = baseLatency + outputLatency + inputLatency from the AudioContext, and $k_{FAF}$ the algorithmic latency of whichever FAF mode is active. The display recomputes whenever any component changes: slider moved, FAF toggled, session started, graph resumed, voice type changed.

Chrome reports outputLatency = 0 immediately after context creation and updates asynchronously. A 200 ms delayed re-read of measureLatencyFloor catches the stabilised value without a full benchmark run.

Jitter Measurement

Accurate jitter measurement from an AudioWorklet required solving a non-obvious problem.

The naive approach of measuring intervals between consecutive process() calls using currentTime produces stddev = 0 by definition. currentTime is the audio clock: it advances in exactly $128 / f_s$ seconds per quantum as guaranteed by the spec. It measures ideal scheduling, not actual scheduling.

performance.now() is not available in AudioWorkletGlobalScope. Date.now() is, but its 1 ms resolution causes its own problem: at 48 kHz the quantum is 2.67 ms, so two consecutive quanta can land on the same millisecond tick, producing 0 ms intervals that accumulate into spurious spikes. Raw intervals from Date.now() have correct mean but meaningless stddev.

The fix is to stop measuring intervals and measure drift from expected time instead. Anchor to both currentFrame (exact integer sample count) and Date.now() (coarse wall clock) at the first quantum:

$$\text{drift}[n] = t_{wall}[n] - \left( t_0 + \frac{f[n] - f_0}{f_s} \times 1000 \right)$$

where $f[n]$ is currentFrame at quantum $n$ and $f_0$, $t_0$ are the anchor values. currentFrame advances in exact samples with no resolution loss. Even with 1 ms Date.now() resolution, drift is meaningful: an 8 ms CPU preemption produces ~8 ms drift regardless of rounding. The standard deviation of drift samples is the canonical jitter metric.

An interval measures the gap between two coarse timestamps. A drift measures the deviation of one coarse timestamp from a precise prediction. The prediction is exact (derived from currentFrame), so the noise is one-sided: only Date.now() contributes coarsening, not both endpoints. This halves the effective resolution noise compared to raw intervals.

Non-Obvious Bugs

onStateChange null dereference. The original listener:

this._ctx?.addEventListener('statechange', () => cb(this._ctx!.state));

AudioContext.close() sets this._ctx = null, then the context fires a final statechange event with state "closed". The ! dereferences null and throws. Fix: read state from the event target, not the instance field:

this._ctx?.addEventListener('statechange', (e) =>
  cb((e.target as AudioContext).state)
);

this._graph! after await in teardown/rebuild chain.

this._graph.setFAFNodes([])
  .then(() => this._graph!.setFAFNodes(semitones)) // throws if DAF stopped mid-await

The user can stop DAF between teardown and rebuild. The ! assertion fires on null. Fix: this._graph?.setFAFNodes(semitones), which silently no-ops if the session ended.

activateModeBtn called before declaration. Chrome hoists block-scoped function declarations within a block; Firefox and Safari do not (ES2015 strict mode). The initialisation call to activateModeBtn('faf-type', 'single') preceded the function declaration by 10 lines within the same DOMContentLoaded callback. This works on Chrome but throws a TypeError on Firefox. Fixed by reordering the declaration above the call.

OLA vs PSOLA: When Each is Appropriate

High-Fidelity (PSOLA) is the default and the correct choice for most therapy use. The latency advantage that once favoured OLA no longer exists — at 120 Hz voice floor PSOLA is now marginally faster. Standard (OLA) is appropriate when a fixed, voice-type-independent latency is needed without any configuration, or when the semitone shift is small ($\leq 2$ st) and phase coherence is less critical.

• You can try it yourself at DAF Online.

References

• Natke, U., et al. (2001). Fluency, fundamental frequency, and speech rate under frequency-shifted auditory feedback in stuttering and nonstuttering persons. Journal of the Acoustical Society of America.
• Kalinowski, J., et al. (1996). Effect of alterations in auditory feedback and speech rate on stuttering frequency. Journal of Speech and Hearing Research, 39, 396–407.

Thanks for reading! If you have questions, feedback, or want to discuss a project, feel free to drop me an emailor message.

Today marks the 1000th sheet of A4 paper I’ve used since I began university back in 2020.¹ My studying technique is writing what I’ve learned using pen and paper. All the theoretical knowledge, math, and most stuff that isn’t code belong on paper. Typing these ideas is less memorable compared to writing with pen and paper. Despite this, flashcards belong to Anki.

I was close to running out of paper, so I bought another 500-sheet stack. When buying paper, you’ll find there are a lot of options if you look closely. I want to write briefly about A4 paper types for writing with a pen from a consumer perspective. Quality Class A is really nice to write on. Quality Class C is okay at the end of the day. Since I don’t use that much paper, I think using Class A would be a good choice to write on. Here are the details:

• Paper Weight (GSM): $80~\text{g/m}^2$ is your standard copy paper, while $90~\text{g/m}^2$ is heavier and feels nicer. $80~\text{g/m}^2$ works fine.
• Whiteness (CIE): Higher CIE means brighter paper. I prefer around 170 CIE for better contrast. When writing with a pen, I really like 170 CIE. 145 CIE is your average white paper, and when you go down to 100 CIE, it’s noticeably duller. I thought about including an comparisonimage here but digital screens doesn’t do it justice.
  It is measured via a spectrophotometer that shines light on the paper.
• Smoothness: Well, you guessed it right—it’s how smooth it is. The smoother the paper, the more time it can take to absorb ink, which might lead to smudging if you’re not careful. 220 ml/min is good for ballpoint pens, but 170 ml/min feels nicer to touch.
  It’s measured via the Bendtsen method, where a ring is pressed against the paper and the air volume passing between the ring and paper is measured.
• Opacity (Ghosting): Higher opacity means less ink bleed-through. 93% is good for single-sided writing, while 95% opacity is good for double-sided writing. Usually, this isn’t a big issue as I mostly write using a pencil.
• Glossy vs. Matte: For writing, matte is usually better, and glossy is more for printing photos.

If you want to dig deeper, there is literature about its history, manufacturing, measurement, and more. But it’s not my expertise.

Footnotes

1. It's probably more than that; I use the uni's printers, some copy shops, and other A4 stacks that are not in my home.↩

Thanks for reading! If you have questions, feedback, or want to discuss a project, feel free to drop me an emailor message.

I built this website in April 2026 using Astro 6, Tailwind CSS 4 and MDX. My stack centers on a Vite 7-powered pipeline optimized with Terser and astro-compress. The $\LaTeX$ equations are rendered using remark-math and rehype-katex. I also have a RSS feed with a beautiful xsl.

In late April, I’ve added /photos with some tweaks to css, along with a thumbnail renderer using the good old Canvas API for OpenGraph images and JSON-LD. This allows me to add a “share as image” button to my posts, which generates a thumbnail on the fly with the post title and a nice background. Of course /photos has its own RSS feed as well.

The entire lifecycle, from transforming content collections to final minification and deployment to GitHub Pages, is orchestrated via make utilizing exiftool and gh to ensure a lean, production-ready build. The unsung heroes include awk, xargs, find, and grep. I’ve opted to ship the site as a compressed binary artifact rather than loose files to the GitHub repo to provide a layer of mild obfuscation to deter casual scrapers and bots.

Rather than inheriting the legacy overhead of al-folio or similar, I spent three days engineering a custom solution tailored to my specific needs. The second iteration took 2 to 3 days as well. Building from scratch ensured total control over the stack, and while it took a few more days to iron out the finer details, I’m happy with the result. I hope you like it, too!

Thanks for reading! If you have questions, feedback, or want to discuss a project, feel free to drop me an emailor message.

TL;DR

I built DAF Online, a free, browser-based tool for speech therapy that helps people who stutter and Parkinson’s patients find fluency. While native apps and $1000 hardware exist, I used the Web Audio API to achieve sub-6ms latency in the browser by aggressively optimizing the audio graph.

Update: I’ve added Frequency Altered Feedback (FAF) to the existing tool. It’s more complex to implement due to buffering requirements, but it opens up another therapeutic avenue. You can read more about it here.

What is Delayed Auditory Feedback?

Delayed Auditory Feedback (DAF) is simple: you hear your own voice played back with a short delay. What’s less obvious is what that tiny lag does to your brain.

For people who stutter, speaking while hearing a slightly delayed version of your own voice can induce near-instant fluency. It’s called the Chorus Effect. Your brain perceives a second speaker and shifts into a different, more fluid processing mode. The same principle is used by speech-language pathologists (SLPs) for Parkinson’s patients, where the delay acts as a natural “speed limit,” forcing slower, more deliberate speech.

The tool has three core audiences: people who stutter, individuals with Parkinson’s Disease, and SLPs running remote telehealth sessions who need a quick, zero-friction way to get a patient practicing from home.

The Landscape Before I Built This (Early 2025)

When I went looking for a free, browser-based DAF tool, I found: nothing that actually worked.

The market looked roughly like this:

• Dedicated hardware (e.g., Casa Futuro, SpeechEasy): $1000-$2500+. Clinically validated, but you need to order, wait, and pay.
• Native mobile apps (e.g., DAF Pro): A handful exist on iOS and Android. Some are free-tier, most push you toward a subscription. They work reasonably well on modern phones.
• Web-based pages: The few I found were marketing funnels pointing back to the native apps, or had limited functionality, or long delays. No one had built an actual working web implementation you could just… open and use.
• The “Developer Gap”: I found a few GitHub repositories that implemented DAF logic. Some used the Web Audio API, while others were native C++ or Python implementations. Their problem was that they weren’t hosted. Just code sitting in a repo.

The implementation isn’t complex. The Web Audio API has had a DelayNode for years. The gap wasn’t technical; nobody had simply bothered to close it.

The Math: What’s Actually Happening

The feedback loop is simple. The output signal is the input signal shifted in time:

$$y[n] = \alpha \cdot x[n - (k_{user} + k_{sys})]$$

Where:

• $y[n]$: The signal the user hears at time $n$
• $x[n]$: The user’s voice entering the microphone
• $k_{user}$: Intentional Lag is the delay you dial in
• $k_{sys}$: System Floor is the device lag. the hidden hardware/OS latency floor
• $\alpha$: Gain (volume)

Then the effective delay $k_{eff}$ is the sum of the intentional lag and the device lag:

$$k_{eff} = k_{user} + k_{sys}$$

The variable most people ignore is $k_{sys}$. It’s not zero. And if it’s high, your “50ms delay” is actually 100ms, which is a qualitatively different therapeutic experience and potentially useless.

Why Latency Is the Whole Game

For DAF to work therapeutically, the internal device latency needs to stay under 15-20ms. That is the time your hardware and software spend processing audio before your intentional delay is added.

Here’s why it matters: if $k_{sys}$ is already 50ms and you set a 50ms intentional delay, the user hears a 100ms echo. Worse, high internal latency usually comes with jitter (timing variance), which breaks the chorus effect entirely. Jitter makes the delay feel unstable. The brain doesn’t settle into choral mode, it just gets confused.

The Latency Landscape by Device

NB: My implementation achieves latency around 6 to 13ms on modern hardware. The exact number depends on the device and browser.

This is why native apps have historically had an edge over web tools. iOS and Android give native audio code direct access to the hardware buffer. The browser sits a layer above that but with the right flags, you can close most of the gap.

How to Test Your Own Floor

Set the software delay to 0 ms. Speak a sharp “P” or “K” sound. If it sounds like one sound, your floor is likely under 15ms. If it sounds like a double-hit or a slap-back echo, your internal latency is above 30ms and you should switch to a wired headset or a better audio driver before using the tool therapeutically.

Bluetooth headphones are incompatible with DAF therapy. Their 150-250ms hardware latency dwarfs any intentional delay you’d set, making the total delay unpredictable and therapeutically ineffective. Always use wired headphones.

How It’s Built for Speed

To be a legitimate alternative to dedicated hardware, the implementation needed to minimize $k_{sys}$ as aggressively as possible. Three things matter most.

1. Minimal Audio Graph Topology

Every node in the Web Audio API graph adds overhead. The final implementation uses a lean, four-node linear chain. No branches, no unnecessary processing.

// Minimal 2-hop topology for maximum performance
_connectAudioNodes() {
    const nodes = this.audioNodes;

    // source (Mic) -> delay (DAF) -> gain (Vol) -> destination (Output)
    nodes.source.connect(nodes.delayNode);
    nodes.delayNode.connect(nodes.gainNode);
    nodes.gainNode.connect(this.audioContext.destination);
}

2. Requesting Hardware-Level Latency

Browsers default to an “interactive” latency mode (~50ms buffer). Setting latencyHint: 0 tells the browser to request the minimum buffer size the hardware allows. Matching the native device sample rate eliminates resampling lag.

_createAudioContext() {
    const contextOptions = {
        // Request absolute minimum buffer size from hardware
        latencyHint: 0,
    };

    // Match native hardware sample rate to bypass resampling lag
    if (this.deviceSampleRate) {
        contextOptions.sampleRate = this.deviceSampleRate;
    }

    this.audioContext = new (window.AudioContext || window.webkitAudioContext)(contextOptions);
}

3. Honest Latency Measurement

The tool reads baseLatency and outputLatency directly from the AudioContext and adds them to the display so the user always sees their effective delay, not just the slider value.

// Measuring the true hardware "floor"
const outputMs = (this.audioContext.baseLatency + (this.audioContext.outputLatency || 0)) * 1000;
this.measuredFloorMs = outputMs;

// UI shows both the target and the honest effective delay
const effective = Math.round(targetDelay + measuredFloorMs);
this.displayLabel = `${targetDelay} ms (~${effective} ms effective)`;

This matters for trust. A user who sees “50ms (est. 56ms effective)” understands their setup. A user who sees “50ms” and hears 100ms thinks the tool is broken.

Frequency Altered Feedback (FAF)

I havent gotten around implementing FAF yet, but its also used in speech therapy. Instead of delaying the signal, it shifts the pitch up or down. The effect is similar: it disrupts the brain’s normal feedback loop and can improve fluency for some users. It’s on the roadmap for a future update, but DAF was the priority since it’s more widely used and has a clearer latency requirement.

Implementing FAF is a challenge because it requires buffering audio to analyze and shift the frequency content. The buffering means added latency, which can break the therapeutic effect if it exceeds the 15-20ms threshold. More on that in the deep dive below.

FAF shifts your voice up or down in pitch rather than delaying it. The therapeutic mechanism is similar, but the implementation is messier.

The problem is that you can’t shift pitch without first buffering a chunk of audio to analyze. A simple delay line just holds samples and replays them. Pitch shifting has to look at a window of the signal before it can do anything, which means latency before your intentional delay is even added.

At 44.1 kHz, the relationship is straightforward:

$$L_{ms} = \frac{N}{f_s} \cdot 1000$$

Why you can’t skip the buffer

The naive fix is sample-by-sample processing: shift pitch like a sped-up record. That works for about half a second until the playback outruns the input and you get a gap. To keep the feedback in sync with actual speech rate, you need time-domain splicing (SOLA): small grains of audio, cross-faded together. That requires an AudioWorklet and a minimum window size.

Phase vocoders vs. granular synthesis

Phase vocoders do this better perceptually. They use FFTs to shift pitch cleanly with no metallic artifacts. The catch is they need large buffers for frequency resolution, typically 1024 samples or more, which puts you at 23ms of algorithmic latency before anything else. That’s already past the cutoff.

Granular synthesis sounds rougher, but it runs on 128 or 256 samples. For this use case, a slightly robotic voice at 10ms beats a natural-sounding one at 40ms.

SEO: Why the Body Text Is Long (On Purpose)

“Delayed Auditory Feedback” is an incredibly niche topic. If you compare it to a broader term like “Stuttering” in Google Trends, you can see how small the specific search market is for the tool itself compared to the condition it treats.

Building the tool was the easy part. Getting it in front of people who need it took just as long.

Most users arrive via high-intent functional queries. They know what a DAF tool is. They just need to find one that works. A minimal landing page with a slider and a button would rank for nothing. In the first months, the tool was invisible to these high-intent users, stalled at 11th-15th in the rankings while hardware retailers and native apps claimed the top spots.

By writing thorough, accurate content about the science and the use cases, the site achieved a 2.4 weighted average position for core keywords, capturing 85% of organic traffic from the top 3 results with a 75% CTR on primary search intent (Mar 2026).

Fun Fact: As it turns out, “DAF” is a congested acronym dominated by Dutch heavy-duty truck manufacturer DAF Trucks N.V. If you search “DAF” without context, Google assumes you’re looking for a 7.5-ton hauler, not a speech aid.

Conclusion

With the right initialization flags and a minimal graph topology, a browser-based DAF tool can match native app latency on decent hardware, with no install, no account, and no payment required.

The gap wasn’t a hard engineering problem; it was just an ignored one. Speech therapy is a small market, and most developers aren’t building for people who stutter or have Parkinson’s. Which is why it was worth doing.

If you want to look at the implementation or try the tool yourself:

• DAF Online — Try it here!
• Source code on GitHub

Footnotes

1. My implementation.↩
2. My implementation.↩

Thanks for reading! If you have questions, feedback, or want to discuss a project, feel free to drop me an emailor message.

This post explores research originally presented at the CVPR 2025 Workshop on Synthetic Data for Computer Vision (SynData4CV).

TL;DR

Instead of using “classical” image tweaks like flipping or rotating, which actually distort facial identity, this research proves that using InstantID to generate high-quality synthetic portraits as training data significantly improves a DreamBooth model’s ability to produce realistic, professional-grade headshots.

The Problem with Few-Shot AI Portraits

You have five casual phone photos and want a polished LinkedIn headshot. Sounds like a job for a text-to-image model, right? In theory, yes. In practice, personalized diffusion models like DreamBooth struggle with a bottleneck of identity retention: they need to learn who you are from a tiny handful of images, then generalize that identity to entirely new scenes and styles.

This is the few-shot personalization problem. It sits at the tension between two competing goals: identity retention (the output should actually look like you) and recontextualization (you should be placeable in any scene the user prompts). Most standard training pipelines lean hard in one direction or the other. My research, published as “Generating Synthetic Data via Augmentations for Improved Facial Resemblance in DreamBooth and InstantID”, investigates a third path: using one generative model to improve the training of another.

Why Classical Augmentations Backfire

When deep learning practitioners want more training data, the first instinct is to reach for classical augmentations: random flips, crops, rotations, colour jitter. These are reliable staples for large-scale classification tasks. For few-shot face personalization, they are a trap.

Geometric traps

Random Horizontal Flip seems harmless, but faces are subtly asymmetric. A mole on the left cheek, a slightly crooked smile, the direction of a part in the hair: flipping these teaches the model a second identity that contradicts the first. Rather than generalizing, the model averages them into an uncanny composite.

Random Rotation introduces black padding bars around the frame, which the model dutifully learns as part of the subject’s visual signature. It also misaligns facial landmarks, undermining the spatial consistency that makes face generation coherent.

Colour confusion

Colour Jittering (tweaking brightness, contrast, saturation, and hue) causes the model to incorrectly associate those shifts with the rare token representing your subject. The result is erratic generations where the subject might appear with an alien skin tone or under lighting that was never in any real photograph.

Segmentation imperfections

Replacing backgrounds using a segmentation model like $U^2$-Net sounds like a clean solution to background leakage. In practice, the segmentation boundary around fine hair creates a blended halo artifact. The model then learns that wispy, semi-transparent fringe is part of the subject’s identity, making clean background swaps nearly impossible downstream.

The pattern is the same across all of these: classical augmentations introduce distributional artifacts, and the model, with no other signal to reject them, faithfully memorizes those artifacts as identity-defining features.

Classical augmentations introduce distributional artifacts. With no other signal to reject them, the model faithfully memorizes these artifacts as identity-defining features.

A New Approach: GenAI Improving GenAI

Instead of perturbing real images in ways that corrupt facial structure, the approach explored in this paper asks a different question: what if the augmented images were themselves high-quality generations of the person, produced by a model that already understands faces?

The answer is generative augmentation via InstantID. By conditioning InstantID on a subject’s facial landmarks and a set of reference images, we can synthesize diverse, photo-realistic portraits of that person across varied poses, lighting conditions, and contexts, all while preserving the structural integrity of their face. These synthetic images already live in the diffusion model’s feature space, so DreamBooth does not have to reconcile the domain gap that plagues classical augmentations.

The result is measurably better facial resemblance in the fine-tuned DreamBooth model, with the full range of recontextualization still intact.

Practical Takeaways

These findings translate directly into actionable recommendations for anyone building portrait personalization pipelines.

1. Balance real and synthetic images

The most important constraint for preventing overfitting is dataset diversity. No single concept (a specific background, a particular outfit, a generated style) should represent more than 25% of your training set. When synthetic images crowd out real ones, the model loses its grip on genuine identity and begins to replicate InstantID’s stylistic fingerprint rather than the subject’s actual face.

2. The Rule of Four

When generating synthetic training data with InstantID, providing four reference images offers the best trade-off between usability and facial similarity. Fewer references produce inconsistent identity across generations; more references yield diminishing returns and increase annotation overhead.

3. Resolution matters

Images around 1 megapixel align with the native training resolution of SDXL and deliver the best qualitative results. Upscaling smaller images introduces compression artefacts; downscaling large images discards high-frequency facial detail. If your source photos are from a phone camera, a light centre-crop to roughly 1024 × 1024 is ideal.

4. Skip the flips, rotations, and jitter

Given the evidence above: do not use Random Horizontal Flip, Random Rotation, or Colour Jitter in the fine-tuning pipeline. Their well-known benefits for large-scale classification tasks do not transfer to few-shot face personalization.

Measuring Resemblance: The FaceDistance Metric

Qualitative “vibes” are a start, but human intuition is subjective. To systematically rank checkpoints and understand how synthetic data actually moves the needle, we needed a reproducible, automated metric. This led to the development of FaceDistance, a validation tool built on FaceNet embeddings.

Rather than looking at pixels, FaceDistance looks at geometry. It projects facial images into a 128-dimensional hyperspherical space where the distance between points reflects perceptual similarity. Specifically, the metric calculates the average cosine distance between a generated image $G_i$ and the set of original reference images $\{R_j\}$:

Definition: Given batches of generated images $G = \{G_i\}_{i=1}^m$ and original reference images $R = \{R_j\}_{j=1}^n$, the FaceDistance is defined as:

$$\bigl[\operatorname{FaceDistance}(G, R)\bigr]_i := \frac{1}{n} \sum_{j=1}^n \delta^{[0,2]}_{\text{cos}}\!\bigl(f(G_i),\, f(R_j)\bigr)$$

Breaking down the logic:

• The Encoder ($f$): We use MTCNN for precise face detection, followed by FaceNet to extract the identity embedding.
• The Distance ($\delta_{\cos}$): We use cosine distance, clipped to a $[0, 2]$ range for numerical stability.

A lower FaceDistance score indicates a stronger mathematical resemblance to the subject.

Pro Tip: FaceDistance acts more like a high-pass filter than a perfect judge. It is excellent for identifying “catastrophic drift” (where the model loses the subject entirely) but it isn’t sensitive enough to decide if a “good” image is “great.”

In our pipeline, we found that simply discarding the top 15% of highest-distance embeddings from the training set (in cases with $n \geq 8$ references) consistently led to cleaner, more recognizable results.

The Human Test: Does It Fool Real People?

Metrics only go so far. To validate that these portraits actually pass muster in professional contexts, the study recruited 97 white-collar workers to evaluate the generated headshots. Both DreamBooth and InstantID produced portraits that were frequently indistinguishable from genuine professional photographs.

Participants’ preferences split along an interesting fault line. Those who valued identity accuracy (“does this actually look like the person?”) tended to prefer DreamBooth outputs. Those drawn to overall aesthetics favoured InstantID for its polished, retouched quality. Neither model dominated on all dimensions, which points to a useful practical heuristic: use DreamBooth-with-generative-augmentation when fidelity to a specific individual is paramount, and use InstantID directly when a studio-quality aesthetic matters more than strict identity retention.

Conclusion

Classical augmentations are not universally beneficial. For few-shot face personalization, several common techniques actively degrade output quality. Replacing them with generative augmentation, where InstantID synthesizes diverse but identity-consistent training images, closes the gap between a handful of casual snapshots and a high-fidelity professional portrait.

The broader takeaway extends beyond portraits: synthetic data is not just a fallback for when real data is scarce. It is a tool for shaping precisely what a model learns. As generative models improve, training pipelines that use one model to curate data for another will become increasingly common.

Acknowledgments

This work would not have been possible without the dedicated mentorship of Benjamin Kiefer. Beyond steering the technical direction of this research, Benjamin was a constant guide through the often-turbulent process of publishing my first paper. His attentiveness during our weekly meetings and his rigorous feedback were fundamental to the success of this project. I am deeply grateful for his support in turning these initial ideas into a peer-reviewed publication.

My thanks also go to Naman Deep Singh for valuable insights into image copyright. His guidance helped clarify the complexities of dataset curation and provided a clear understanding of the legal boundaries for image usage.

I am also grateful to the CVPR SynData4CV workshop reviewers for their constructive comments.

Citation

If you build on this work or wish to explore the full list of references and literature supporting this research, please refer to the formal paper:

Ulusan, K., & Kiefer, B. (2025, May). Generating synthetic data via augmentations for improved facial resemblance in DreamBooth and InstantID [Paper presentation]. CVPR Workshop on Synthetic Data for Computer Vision (SynData4CV), Nashville, TN, United States. https://arxiv.org/abs/2505.03557

@inproceedings{ulusan2025generating,
  author        = {Ulusan, Koray and Kiefer, Benjamin},
  title         = {Generating Synthetic Data via Augmentations for
                   Improved Facial Resemblance in {DreamBooth} and {InstantID}},
  booktitle     = {Synthetic Data for Computer Vision Workshop at {CVPR} 2025},
  year          = {2025},
  url           = {https://openreview.net/forum?id=2o0RxrcV23},
  eprint        = {2505.03557},
  archivePrefix = {arXiv},
  primaryClass  = {cs.CV},
  doi           = {10.48550/arXiv.2505.03557}
}

• The full paper is available at arXiv:2505.03557
• Project webpage is synthetic-face-augmentation.github.io

Thanks for reading! If you have questions, feedback, or want to discuss a project, feel free to drop me an emailor message.

TL;DR

I modified TD3 with three reinforcement learning techniques: Prioritized Experience Replay (PER), potential-based reward shaping, and multi-step returns. I used these to train an agent in a simulated air hockey game. Most modifications made things worse. What actually worked was a curriculum: pre-training on shooting and defending modes before facing the real opponent. The final agent wins 98.3% of games against the strong built-in opponent.

The Problem: Sparse Rewards in a Competitive Environment

Air hockey is a hard environment for RL. Goals are rare and delayed, preceded by a long sequence of positioning decisions that receive no direct reward signal. The agent needs to learn to move toward the puck, hit it in the right direction, and coordinate defense and offense, all from a reward that stays at zero until something decisive happens.

The environment I used is HockeyEnv (a.k.a. “Laser Hockey”), a Box2D/Gymnasium simulation of a two-player air hockey game. The observation space is 18-dimensional (positions, velocities, angles of both player and puck), and the action space is a 4-dimensional continuous vector covering movement and shooting. Each episode runs for up to 250 timesteps in normal mode, or a shorter 80-step window in dedicated shooting/defending training modes.

The standard “run a good off-policy algorithm and wait” approach struggles here. The agent’s first instinct is to stand still and draw, because drawing is better than the random-action baseline it gets penalized against. Getting past that local optimum requires deliberate intervention.

Base Algorithm: Twin Delayed DDPG (TD3)

TD3 is an actor-critic algorithm that addresses the well-known overestimation bias of DDPG by maintaining two critics and taking the minimum of their Q-value estimates when computing targets:

$$y = r_t + \gamma \min_{k=1,2} Q_{\theta_k'}(s_{t+1},\, \pi_{\phi'}(s_{t+1}) + \epsilon), \quad \epsilon \sim \text{clip}(\mathcal{N}(0,\sigma), -c, c)$$

It also delays actor updates relative to critic updates (hence “Twin Delayed”), which gives the critics time to stabilize before the policy starts chasing them. I built UlusanTD3 on top of Stable Baselines 3, extending the base TD3 class to support the three techniques described below.

The name UlusanTD3 is chosen purely for convenience of the project graders and easy identification in the codebase. It doesn’t imply any fundamental change to the TD3 algorithm itself, but rather serves as a container for the specific modifications and experiments conducted in this project.

Technique 1: Prioritized Experience Replay

Standard experience replay samples uniformly from a FIFO buffer. PER [Schaul et al., 2015] argues that transitions where the agent was wrong (those with high temporal-difference TD error) are more informative and should be sampled more often. The sampling probability for a transition is:

$$P(i) = \frac{p_i^\alpha}{\sum_k p_k^\alpha}$$

where $\alpha$ controls prioritization strength and $p_i$ is the priority of transition $i$. For TD3 with two critics, I define the priority as the average absolute TD error across both:

$$p_i = \left|\frac{1}{2}(\delta_1 + \delta_2)_i\right| + \epsilon$$

To keep priorities tractable and prevent divergence, I clip TD errors to the range $[\epsilon, 1]$. Without this upper bound, a single catastrophic prediction early in training can dominate the buffer forever and destabilize the actor.

Sampling more from high-error transitions introduces a bias, which is corrected with importance-sampling (IS) weights:

$$w_i = \left(N \cdot P(i)\right)^{-\beta}, \quad \text{normalized by } \max_i w_i$$

These weights are folded into the critic loss, replacing the standard MSE:

$$\mathcal{L}(\theta_k) = \mathbb{E}\left[w \cdot \delta_k^2\right]$$

Because I have two critics with potentially different scales, I give each its own optimizer rather than summing their losses:

# In UlusanTD3.__init__
if isinstance(self.replay_buffer, PrioritizedExperienceReplayBuffer):
    self.critic1_optimizer = th.optim.Adam(
        self.critic.q_networks[0].parameters(), lr=learning_rate
    )
    self.critic2_optimizer = th.optim.Adam(
        self.critic.q_networks[1].parameters(), lr=learning_rate
    )

The PER buffer itself is backed by a SumSegmentTree, which supports O(log N) priority updates and O(log N) stratified sampling. This is essential when the buffer holds a million transitions:

def _sample_indicies_proportional(self, batch_size: int) -> np.ndarray:
    p_total = self._td_errors.sum(end=self.size())
    segment_length = p_total / batch_size
    elem_at_segment_prefixsum = (
        np.arange(batch_size) + np.random.uniform(0, 1, batch_size)
    ) * segment_length
    return [
        self._td_errors.find_prefixsum_idx(p)
        for p in elem_at_segment_prefixsum
    ]

Using a NumPy array in SegmentTree was important because a Python list was too slow for the large buffer size and high update frequency.

After each gradient step, priorities are updated to reflect the latest TD errors:

# Back in the train loop
td_errors = (td_error1 + td_error2) / 2.0
self.replay_buffer.set_priorities(
    batch_inds,
    td_errors.abs().squeeze().detach().cpu().numpy()
)

What actually happened: PER made performance worse in every configuration I tested. More on why below.

Technique 2: Potential-Based Reward Shaping

In environments with sparse rewards, auxiliary signals that encode domain knowledge can accelerate learning without changing the optimal policy. Potential-based reward shaping [Ng et al., 1999] adds a shaping term:

$$F(s_t, s_{t+1}) = \gamma \phi(s_{t+1}) - \phi(s_t)$$

The key property is that this never changes the optimal policy. It only changes how quickly the agent converges to it. The potential function $\phi: S \to \mathbb{R}$ can encode whatever domain knowledge you have.

HockeyEnv conveniently exposes sub-reward components in its info dict. I used a combination of:

• closeness_to_puck — reward for staying near the puck
• touch_puck — bonus for making contact
• puck_direction — reward for hitting the puck toward the opponent’s goal

Two components I tried and removed: centered_puck introduced noise and slowed training, and game_length inadvertently taught the agent to step aside and let in own goals.

def shaped_reward(rewards, infos):
    phis = [
        (info.get("prev_potential_reward", 0),
         info.get("current_potential_reward", 0))
        for info in infos
    ]
    # F(s, s') = gamma * phi(s') - phi(s)
    return [
        r + self.gamma * phi - phi_prev
        for r, (phi_prev, phi) in zip(rewards, phis)
    ]

Because HockeyEnv is a fully observable MDP, I compute $\phi$ directly from the environment state on every step, so no approximation is needed.

Technique 3: Multi-Step Returns

Standard TD3 bootstraps one step into the future. In hockey, the decisive action (the puck shot) is made many timesteps before the goal is actually scored, so the one-step target has no way to credit that shot with the eventual reward.

The truncated n-step return addresses this by accumulating rewards forward:

$$R_t^{(n)} = \sum_{k=0}^{n-1} \gamma^k \cdot r_{t+k}$$

and substituting it into the TD3 target:

$$y = R_t^{(n)} + \gamma^n \min_{k=1,2} Q_{\theta_k'}(s_{t+n},\, \pi_{\phi'}(s_{t+n}) + \epsilon)$$

With reward shaping combined, the telescoping sum over the potential terms simplifies, and the target becomes:

$$y = \sum_{k=0}^{n-1} \gamma^k r_{t+k} + \gamma^n \Phi(s_{t+n}) - \Phi(s_t)$$

Implementation-wise, this requires buffering the last $n$ transitions before committing any of them to the replay buffer. The buffer is flushed early when a terminal state is reached:

def add(self, obs, next_obs, action, reward, done, infos):
    self._reward_info_buffer.append((reward, infos))

    if any(done):
        # flush remaining transitions on episode end
        while len(self._reward_info_buffer) > 0:
            self._add_n_step_return(obs, next_obs, action, done, infos)
        return

    if len(self._reward_info_buffer) < self.n_step_return_num:
        return  # keep buffering

    self._add_n_step_return(obs, next_obs, action, done, infos)

The discount exponent in the Bellman target also needs updating to account for the extended horizon:

target_q_values = (
    replay_data.rewards
    + (1 - replay_data.dones)
    * self.gamma ** self.n_step_return_num  # γⁿ instead of γ in TD3
    * next_q_values
).detach()

The Ablation Study: Most Things Didn’t Help

I ran combinations of all three techniques (26 configurations in total, sweeping $n \in \{2, 3, 4, 5, 10, 20, 30\}$ for multi-step returns) and evaluated each against the weak built-in opponent. The results were not great.

Most modifications performed worse than the TD3 baseline. The 3-step return variant was the only technique that consistently outperformed the baseline, and even that improvement was modest.

PER failed systematically. Training with IS weights disabled diverged immediately: without correction, the buffer fills with high-error transitions and the critic chases a badly biased distribution. With IS weights enabled, training was stable but still underperformed the baseline.

The failure of PER wasn’t a bug, it was informative. PER’s design assumes a stationary data distribution. When the environment or the opponent changes, that assumption breaks.

Curriculum Learning: The Thing That Actually Worked

The real insight was about changing how the agent was trained rather than what algorithm it used.

Without guidance, an agent facing a strong opponent quickly figures out that drawing (never scoring, never conceding) is safer than attempting to score. Once it settles into that strategy, it’s hard to unlearn, because the risk of a failed shot (giving the opponent a chance to score) outweighs any expected benefit from trying.

The curriculum I designed breaks this trap in two phases.

Phase 1 (steps 0 to 440k): Alternate every episode between the dedicated shooting mode and defending mode of HockeyEnv. These stripped-down scenarios cut out the full-game complexity and force the agent to develop fundamental skills: aim and shoot; track and block. The episode horizon is only 80 steps, which enables much faster iteration.

Phase 2 (steps 440k+): Empty the replay buffer entirely and switch to training against the strong BasicOpponent in full normal-mode games. The clean buffer prevents old experiences from contaminating the new distribution.

The agent that had been stuck below 50% win rate against the strong opponent reached 98.3% win rate within 110k steps of Phase 2 training. Notably, the win rate against the weak opponent also climbed during Phase 1, even though the agent had never played full games during that phase.

Why PER and Curriculum Don’t Mix

When the curriculum switches from Phase 1 to Phase 2, the replay buffer gets emptied. For standard TD3, this is a clean reset. For PER, it causes problems.

The new transitions in Phase 2 initially have high TD errors (the agent has never seen full-game states before). These saturate the buffer with maximum-priority entries. The IS weights assigned to these transitions drop to near zero, because $w_i = (N \cdot P(i))^{-\beta}$ becomes tiny when a large fraction of transitions share the same maximum priority. The critic is updated on high-error samples with effectively zero weight, which means it barely updates at all. The actor loss then diverges.

As shown in the logic below, when new high-error transitions dominate the distribution, the probability $P(i)$ of selecting a new sample becomes very large relative to the small buffer size $N$ during the reset, causing the weight to vanish:

$w_i = (N \cdot P(i))^{-\beta}$

When $P(i_{new}) \gg P(i_{old})$, then $w_{i\_new} \to 0$.

The critic is updated on high-error samples with effectively zero weight, which means it barely updates at all. Consequently, the actor loss diverges because it is receiving gradients from an unmoving, inaccurate critic.

This is why PER was dropped from the final curriculum configuration entirely.

Self-Play

I also explored self-play, training the agent against a pool of its own past checkpoints. The hope was to develop generalization beyond the scripted BasicOpponent.

It didn’t work. The agents converged to a Nash equilibrium of mutual avoidance: both players positioning themselves to not touch the puck rather than risk conceding a goal. Episode lengths climbed toward the 250-step maximum. Once discovered, this drawing strategy was self-reinforcing, because any agent that tried to attack would get punished by an opponent that had learned to exploit aggressive positioning.

Risk aversion dominates in self-play when the stakes are symmetric. The agent’s value function correctly estimates that the expected return from “don’t touch the puck” is higher than the noisy expected return from “attempt a shot.” Injecting BasicOpponent episodes or clearing the buffer when switching opponents did not fix this.

The approach that actually works, from what I heard from peers, is to mix self-play with skill-based training against an easy opponent throughout the whole training run. That way the agent never completely forgets that scoring goals is the point.

Self-play can lead to degenerate equilibria if not carefully structured. In competitive environments, it’s crucial to maintain a curriculum that keeps the agent focused on the ultimate goal rather than settling for safe but unproductive strategies.

Tournament Results

The final agent (3-step return, curriculum, no PER) competed in the 2025 RL course tournament at the University of Tübingen, ranking 131/146 (including stale accounts) with a 40% win rate against other students’ agents. Some students chose not to join the tournament.

This is a more honest number than the 98.3% against BasicOpponent. My self-play loop didn’t worked because the rewards were very sparse. My peers included other modes in their curriculum, to increase reward signal and prevent the drawing deadlock. In the tournament, RL agents with world models didn’t performed much better than ones like TD3. The real winner move was to train on tournament data to learn the specific quirks of the opponents, which is a form of overfitting but effective in a competitive setting. Yes, it was allowed.

If I included basic defending and shooting modes throughout the self-play phase, tournament performance would have been noticeably better. The agent was robust but never had a complete training setup.

Practical Takeaways

These findings aren’t specific to air hockey.

On PER: It works well in stationary, single-distribution settings. In non-stationary environments (curriculum training, population-based training, anything that changes the data distribution mid-training), the mismatch between stored priorities and the current distribution becomes a liability. Either clear the buffer on every regime change or skip PER in this setting.

On reward shaping: Be selective about what you encode in $\phi$. Subcomponents that make intuitive sense (stay near the puck) can introduce perverse incentives at the MDP level (game_length rewarding own goals). The sufficiency theorem guarantees no harm asymptotically, but finite training is far from asymptotic.

On multi-step returns: Modest $n \in [2, 5]$ is almost always better than large $n \in [10, 30]$ in continuous control. Large $n$ introduces high variance in the return estimate and makes the bootstrap target less reliable.

On self-play: Combine it with skill-based training modes from day one. Self-play alone, starting from scratch, finds the drawing equilibrium before it finds the scoring one.

Conclusion

Adding algorithmic improvements to TD3 mostly made things worse. What actually unlocked real performance was a carefully structured training curriculum: a decision about what the agent practices rather than how it learns.

In sparse-reward environments, the hardest problem is not the algorithm, it’s the training setup. PER, multi-step returns, and reward shaping are all principled ideas, but they operate on data. Curriculum learning shapes what data is generated in the first place.

A stable self-play loop that combines pool-based opponent selection with dedicated skill modes is the most promising direction for pushing these agents further.

Here are the resources if you want to dive deeper:

• Technical Report
• Presentation
• GitHub Repo
• RL Agent’s Code

Acknowledgments

This project was completed as part of the RL Course 2024/25 taught by Prof. Georg Martius at the University of Tübingen, in collaboration with Elia Frederick Reppchen (Rainbow DQN) and ChandraLekha Ramireddy (SAC). Compute was provided by the TCML cluster offered by the Cognitive Systems Group (of Prof. Andreas Zell) at the University of Tübingen.

Thanks for reading! If you have questions, feedback, or want to discuss a project, feel free to drop me an emailor message.

TL;DR

Three homework sets from Trustworthy Machine Learning. The assignments covered OOD generalisation (ensemble diversification, IRM, adversarial attacks on LLMs), explainability (feature attribution, training data attribution), and uncertainty (calibration, epistemic/aleatoric decomposition, deep ensembles). They were quite the uphill battle at the time, but the insights have stuck with me and regularly shape how I approach ML problems today.

OOD Generalisation

Domain generalisation vs. the actual setup

Domain generalisation trains on labelled source domains $D_1, \dots, D_n$ with zero access to the target domain at all. The exercise used a slightly different setup: $n$ source domains plus a batch of unlabelled deployment samples from $D_{n+1}$. That unlabelled data opens the door to semi-supervised adaptation techniques that pure domain generalisation cannot use.

Learning from Failure

The paper Learning from Failure (LfF, NeurIPS 2020) handles the feature-selection problem without labelling which features are spurious. The trick is to train a biased model first and read off the task-relevant features from where it fails: samples where spurious cues (e.g. colour) point the wrong way are exactly the samples the biased model gets wrong. A second, debiased model is then up-weighted on those failures. No bias annotation needed; the model’s own failure mode is the signal.

The implicit assumption is that spurious correlations are easier to learn than task-relevant ones, so the biased model really does latch on to them first. That holds on synthetic benchmarks and is less certain in the wild.

dSprites and the diagonal setup

All experiments used dSprites: 64x64 images of shapes (square, ellipse, heart) with latent factors for colour, shape, scale, orientation, and position. Labels are uniformly binned into 3 classes per latent.

The diagonal dataset is the core construct: bias cue and task cue are perfectly correlated in training (off_diag_proportion = 0), so colour == shape always. A model that learns colour achieves 100% training accuracy without ever touching shape. On the unbiased test set, colour accuracy collapses to chance (33%) and shape accuracy reveals whether anything real was learned. Setting off_diag_proportion = 0.1 adds 10% bias-conflicting samples, which slightly dilutes the spurious signal.

Product of Experts

The PoE loss adds member logits before the softmax, which is equivalent to taking the product of their softmax distributions up to normalisation:

$$p_{\text{PoE}}(y \mid x) \propto \prod_{m=1}^{M} p_m(y \mid x) = \text{softmax}\!\left(\sum_{m=1}^{M} \ell_m(x)\right)$$

class ModelTrainerPoE(ModelTrainer):
    def train_loop(self, images_batch, labels_batch, epoch=None):
        labels_batch = labels_batch[self.task_label_index]
        logits_dict = self.model(images_batch)
        for idx, (_, logits) in enumerate(logits_dict.items()):
            combined = logits if idx == 0 else combined + logits
        return F.cross_entropy(combined, labels_batch)

Two identical ResNet18s trained jointly on the fully diagonal dataset both specialise in colour. They share the same architecture, the same data, and receive symmetric gradients through the summed logits. Nothing breaks the symmetry.

Three changes together produced actual diversification:

1. Heterogeneous architectures: ResNet18 for member 0, ResNet101 for member 1.
2. Greyscale input for member 1: it physically cannot use colour.
3. Sequential optimisation: train only member 0 for epoch 1, then switch to member 1.

A cosine-similarity diversity penalty also helps at the margins:

diversity_penalty = 0
for i, i_outs in enumerate(logits_dict.values()):
    for j, j_outs in enumerate(logits_dict.values()):
        if i != j:
            diversity_penalty += F.cosine_similarity(i_outs, j_outs).mean()
loss += 0.1 * diversity_penalty

The greyscale intervention is architecturally clean; the diversity penalty is a soft nudge without guarantees about what the two members disagree on.

PoE alone does not diversify a homogeneous ensemble. You need either different inputs, different architectures, or explicit gradient-level repulsion.

HSIC regularisation

ReBias (ICML 2020) uses the Hilbert-Schmidt Independence Criterion to decorrelate a biased model’s embeddings from an unbiased one’s. HSIC measures statistical dependence via centred kernel matrices:

$$\text{HSIC}(K, L) = \frac{1}{m^2} \operatorname{tr}(\tilde{K} \tilde{L}), \quad \tilde{K} = HKH, \quad H = I - \tfrac{1}{m}\mathbf{1}\mathbf{1}^\top$$

def _hsic_loss(self, embeddings):
    em1, em2 = embeddings
    def rbf_kernel(x, sigma=1.0):
        return torch.exp(-torch.cdist(x, x, p=2)**2 / sigma**2)
    K, L = rbf_kernel(em1), rbf_kernel(em2)
    m = K.shape[0]
    H = (torch.eye(m) - torch.ones(m, m)/m).to(K.device)
    return (1/m**2) * torch.trace(H @ K @ H @ (H @ L @ H))

The training loss is $\mathcal{L}_{CE} + \alpha \cdot \text{HSIC}(z_{\text{biased}}, z_{\text{unbiased}})$.

Running the heterogeneous ensemble (BiasedNet + ResNet18) and varying ALPHA across five orders of magnitude did not materially change the accuracy breakdown. The HSIC value itself was small and stable, which pointed at the kernel bandwidth: with $\sigma = 1$ and 512-dimensional penultimate embeddings, pairwise distances saturate the RBF and the gradient signal effectively disappears. A data-adaptive bandwidth (e.g. the median heuristic) would be the right fix, but I didn’t pursue it.¹

The off_diag_proportion mattered more than HSIC in these experiments: going from 0% to 10% off-diagonal pushed ResNet18 shape accuracy from 33% to 41% regardless of HSIC regularisation.

IRM

Invariant Risk Minimisation (Arjovsky et al., 2019) takes a different approach to domain generalisation: instead of diversifying the ensemble, it learns a single representation where the optimal linear classifier on top of it is the same across all training environments. The penalty measures this by checking the gradient of each environment’s loss with respect to a dummy scalar $w$ clamped at 1: if $\nabla_w \mathcal{L}_e(\Phi \cdot w)\big|_{w=1} \approx 0$ for all $e$, then $w = 1$ is simultaneously optimal everywhere. The implementation splits each batch by environment label, computes the ERM loss and the squared gradient norm per environment separately, then combines them as $\mathcal{L}_{\text{IRM}} = \sum_e \mathcal{L}_e + \lambda \sum_e \lVert \nabla_{w|w=1} \mathcal{L}_e \rVert^2$. The gradient norm is computed by attaching a scalar dummy multiplier to the representation, doing a backward pass per environment, and squaring the resulting grad. On dSprites, IRM did not clearly outperform ERM in early training, which is consistent with its known sensitivity to finite-sample noise: the penalty can be trivially satisfied when environments are not sufficiently diverse to force a genuinely invariant solution.

Adversarial attacks on LLMs

GCG (Zou et al., 2023) showed that aligned LLMs can be jailbroken by appending an adversarial suffix to a harmful prompt. The suffix is found by a greedy coordinate search: compute the gradient of the target string’s cross-entropy with respect to one-hot token embeddings, sample the top-$k$ candidate replacements for each suffix position, evaluate all candidates in a forward pass, and keep the best. get_loss is cross-entropy between the model’s logits at the target positions and the target token ids. compute_token_gradient embeds the input as one-hot vectors, runs a forward pass with those embeddings, calls get_loss, and returns the gradient of the loss with respect to the one-hot matrix. sample_ids_from_grad uses the negated gradient at each suffix position as a proxy score for candidate tokens and samples without replacement from the top-$k$ by score, producing a set of candidate replacement tokens per position. The run_gcg outer loop and evaluation were provided.

XAI

Remove-and-classify

Evaluating a feature attribution map by eye is not very rigorous. Remove-and-classify (RAC) gives a model-agnostic metric: sort pixels by attribution score, remove the top $k$ at a time, rerun the model, measure accuracy drop. A lower area under the accuracy curve (lower RAC score) means the method correctly identifies which pixels the model actually relies on.

def remove_and_classify(model, dataset, explanations, k=100):
    sorted_ranks = sort_explanations_by_importance(explanations)
    performances = []
    for i in range(0, width * height, k):
        for idx, coords in enumerate([sorted_ranks[j][i:i+k]
                                      for j in range(len(explanations))]):
            rows, cols = zip(*coords)
            dataset.data[idx][rows, cols] = 128  # replace with grey
        performances.append(model_accuracy(model, dataset))
    return performances

The centred Gaussian baseline assigns highest importance to pixels near the image centre regardless of content. For CUB-200 bird images, birds often appear near the centre, so this baseline is not as naive as it sounds.

Saliency

Saliency (Simonyan et al., 2013) is the gradient of the predicted class score with respect to the input:

$$M_{ij} = \max_c \left| \frac{\partial f_c(x)}{\partial x_{ij,c}} \right|$$

def saliency(model, dataset, sample_idx):
    image, _ = dataset[sample_idx]
    image = image.to(device).unsqueeze(0).requires_grad_(True)
    output = model(image)[0]
    output[output.argmax(dim=0)].backward()
    return image.grad.abs().max(dim=1)[0].squeeze().cpu()

One backward pass. The maps tend to be noisy because gradients saturate in many regions of a deep network, highlighting edges more than semantically meaningful areas.

CAM

CAM (Zhou et al., 2016) works only on architectures where the classifier is a global average pooling followed by a linear layer (or a 1x1 convolution). The activation map for class $c$ is a weighted sum of final-layer feature maps:

$$\text{CAM}_c(x, y) = \sum_k w_k^c f_k(x, y)$$

def class_activation_map(model, dataset, sample_idx):
    activation = {}
    model.layer4[-1].relu.register_forward_hook(
        lambda m, i, o: activation.update({'f': o.detach()})
    )
    image = dataset[sample_idx][0].to(device).unsqueeze(0)
    img_class = model(image).argmax(dim=1)
    cam = (model.conv_last.weight[img_class].squeeze(0)
           * activation['f'].squeeze(0)).sum(dim=0)
    cam = F.interpolate(cam[None, None], size=image.shape[-2:], mode='bicubic')
    return cam.squeeze().detach().cpu()

CAM localises well because it uses the network’s own classification weights as spatial importance. The trade-off is that you need GAP before the classifier, which rules out most standard architectures without modification.

Training data attribution

Where feature attribution asks which pixels drove a prediction, training data attribution asks which training examples drove it.

The influence function of training example $z_j$ on test example $z$ approximates the change in test loss from removing $z_j$ and retraining:

$$\text{IF}(z_j, z) \approx -\nabla_\theta \mathcal{L}(z, \hat\theta)^\top H_{\hat\theta}^{-1} \nabla_\theta \mathcal{L}(z_j, \hat\theta)$$

Computing $H^{-1}v$ is the expensive step. LiSSA (used in FastIF) estimates it iteratively over minibatches using the recurrence:

$$H^{-1}v \approx \sum_{t=0}^{T}(I - H)^t v$$

new_estimate = [
    a + (1 - damp) * b - c / scale
    for a, b, c in zip(v, last_estimate, this_estimate)
]

The evaluation reruns training after removing the top-$p\%$ most helpful, most harmful, or random training examples, then measures confidence change on the test example. Removing helpful examples should decrease confidence; removing harmful ones should increase it.

kNN speedup. Full influence computation is $O(n)$ over training examples. FastIF restricts gradient dot products to the $k$ nearest neighbours in embedding space, using the [CLS] token from the final layer:

def embed_one_example(model, example):
    with torch.no_grad():
        out = model(**example, output_hidden_states=True)
        return out.hidden_states[-1][0, 0]

def get_knn_indices(model, embedded_train, test_input, k=100):
    test_emb = embed_one_example(model, test_input)
    dists = torch.cdist(torch.stack(embedded_train), test_emb.unsqueeze(0))
    return torch.topk(dists.squeeze(1), k=k).indices

The IHVP is still computed over the full training set since the Hessian is a global quantity. Only the final gradient dot products are restricted to neighbours.

Discussion

For feature attribution: the centred Gaussian baseline performing reasonably well on CUB-200 means the model is itself centre-biased, since birds appear near the centre in most images. Any attribution method that just highlights the centre will therefore look decent on RAC, which is a dataset artefact rather than a property of the attribution. Replacing removed pixels with a constant (128 / grey) is also not a sound evaluation because the inpainting itself can be informative about where features were removed, and it shifts the test distribution outside what the model trained on. A more principled approach would use learned in-painting or marginalise over a conditional distribution.

For training data attribution: LiSSA converges when the Hessian’s spectral radius is less than 1, which requires the scale parameter to be large enough to contract the recurrence. Near-zero eigenvalues inflate the corresponding components of $H^{-1}v$, making the approximation unstable; in practice this is handled by the damp term that adds a small diagonal. params_filter restricts gradient computation to a named subset of parameters, typically just the final layer. For large pretrained models this is a good trade-off: the last layer’s gradients capture most of the task-specific signal, and computing full-model Jacobians would be prohibitively expensive. For the kNN speedup, computing influence scores for the full training set vs. only the $k$ nearest neighbours and sweeping $k$ shows that quality (correlation with full influence) degrades slowly while compute drops roughly as $k/n$.

Calibration

BCE as a proper scoring rule

A scoring rule is strictly proper if the forecaster’s expected score is minimised by reporting the true probability. For binary cross-entropy, let $p = P(Y=1)$. The expected loss is:

$$\mathbb{E}[\mathcal{L}(q, Y)] = -(p\log q + (1-p)\log(1-q))$$

Setting $\frac{d}{dq} = 0$ gives $p/q = (1-p)/(1-q)$, which simplifies to $p = q$. The second derivative is $p/q^2 + (1-p)/(1-q)^2 > 0$, so the function is strictly convex and $q = p$ is the unique minimiser.

For contrast, the linear loss $\mathcal{L}(q,y) = -yq + (1-y)(1-q)$ has expected value $1 - p - q$, minimised at $q \to 1$ regardless of $p$. It is not a proper scoring rule.

Calibration metrics

Expected Calibration Error bins predictions by confidence and measures the weighted average gap between confidence and accuracy:

$$\text{ECE} = \sum_{b=1}^{B} \frac{|B_b|}{n} \left| \text{acc}(B_b) - \text{conf}(B_b) \right|$$

NLL evaluates calibration as a proper scoring rule directly. AUROC measures discrimination, which is independent of calibration: a model that always predicts 0.51 for positives and 0.49 for negatives achieves perfect AUROC but terrible calibration.

def ece_score(pred_prob, y, n_bins=10):
    bin_edges = torch.linspace(0, 1, n_bins + 1)
    ece = torch.tensor(0.0)
    for i in range(n_bins):
        mask = (bin_edges[i] <= pred_prob) & (pred_prob < bin_edges[i+1])
        if mask.any():
            acc = (pred_prob[mask].round() == y[mask]).float().mean()
            conf = pred_prob[mask].mean()
            ece += (mask.sum() / len(pred_prob)) * (acc - conf).abs()
    return ece

def nll_score(pred_prob, y):
    return -torch.mean(y * torch.log(pred_prob) + (1 - y) * torch.log(1 - pred_prob))

Which model is which

Four anonymous binary classifiers, evaluated on ECE, NLL, and AUROC:

Model 3’s reliability diagram looks unusual at 10 bins: tight calibration on average but high per-bin variance. Zooming to 50 bins shows the samples are not uniformly distributed across confidence levels. The model is extremely confident on most predictions and rarely uses intermediate probabilities.

Cheating ECE

Model 5 had a bad ECE of 0.25. The ground-truth labels were shuffled (preserving the marginal class distribution but losing per-sample alignment), so there was no way to actually improve the model. The task was to make the score look better anyway.

ECE bins predictions, which creates tolerance: predictions of 0.48 and 0.52 land in different bins but are nearly identical values. You can exploit this by gradient-descending the predictions against the shuffled labels while re-shuffling at each step:

def increase_ece(pred_prob, shuffled_y):
    pred_prob = pred_prob.clone().detach().requires_grad_(True)
    optimizer = torch.optim.Adam([pred_prob])
    for step in range(1000):
        optimizer.zero_grad()
        loss = differentiable_ece(pred_prob, shuffled_y)
        loss.backward()
        optimizer.step()
        pred_prob.data.clamp_(0, 1)
        shuffled_y = shuffled_y[torch.randperm(shuffled_y.size(0))]
    return pred_prob.detach()

The re-shuffling prevents the optimiser from fitting any particular label assignment. In practice, it collapses predictions toward 0.5, which looks calibrated in the 10-bin ECE because each bin gets roughly equal confidence and accuracy values.

NLL cannot be cheated this way. NLL is strictly proper: any transformation of $\hat{p}$ that does not move it closer to the true $P(Y=1)$ increases the expected NLL by definition. Collapsing to 0.5 gives NLL $\geq \log 2 \approx 0.693$, which is worse than any model that knows anything about the task. The binning tolerance ECE relies on simply does not exist in NLL.

ECE can be gamed without improving the model. NLL cannot, which is part of why it is a preferred calibration metric in the literature.

Epistemic uncertainty: ensembles

The second part of the uncertainty homework covered three ensemble methods for estimating epistemic uncertainty. Bagging trains $N$ models on bootstrapped subsets (sampling $n$ examples with replacement, giving each model ~63% unique examples) and averages their predicted distributions. MC Dropout (Gal & Ghahramani, 2016) keeps dropout active at test time and draws $M$ stochastic forward passes; epistemic uncertainty is the Jensen-Shannon divergence between the $M$ predictive distributions and aleatoric uncertainty is their average entropy. Fast Geometric Ensembling (Garipov et al., 2018) continues training from a converged checkpoint with cosine annealing and snapshots the model at each cycle minimum, giving ensemble diversity at a fraction of the cost of training from scratch $N$ times.

All three methods should show higher epistemic uncertainty on OOD inputs (the models disagree more when the input is unlike training data) and stable aleatoric uncertainty (since aleatoric noise is a property of the data, not the model). In practice, MC Dropout tends to underestimate epistemic uncertainty because the dropout approximation is a weak posterior; FGE snapshots sit in different basins and tend to disagree more meaningfully on hard examples.

Aleatoric uncertainty: ELK and EDL

Total predictive uncertainty decomposes into epistemic and aleatoric components. Under an ensemble posterior the analytical result is: total entropy of the mean prediction equals mean entropy across members (aleatoric) plus mutual information between the prediction and the model parameters (epistemic). The Monte Carlo estimate replaces the intractable integrals with averages over $M$ samples: $\hat{U}_{\text{aleatoric}} = \frac{1}{M}\sum_m H[p_m]$ and $\hat{U}_{\text{epistemic}} = H[\bar{p}] - \hat{U}_{\text{aleatoric}}$.

Evidential Deep Learning (Sensoy et al., 2018) places a Dirichlet prior over class probabilities, so a single forward pass returns both a prediction and separate uncertainty estimates. Without a regulariser, the network learns to make the Dirichlet concentration parameters very large on training data, collapsing uncertainty to near zero even when predictions are wrong. The regulariser penalises evidence on incorrect predictions by scaling the KL divergence from the posterior to a flat Dirichlet by the prediction error, forcing the model to retain uncertainty proportional to how much it’s been wrong.

LLM uncertainty

MC Dropout on GPT-2: keep the model in train() mode at test time, run $M$ forward passes, and measure the entropy of the mean next-token distribution (total uncertainty) and the JSD across pass distributions (epistemic). Tokens where the model consistently assigns high probability to a single token should have low epistemic uncertainty; rare or syntactically ambiguous positions should have high uncertainty. For the BERT NER task, uncertainty should be highest at entity boundaries and on entity types that appear rarely in training, and lowest on tokens clearly labelled O.

Closing Thoughts

A theme across all three exercises: evaluation metrics have blind spots that matter. ECE can be gamed, AUROC ignores calibration, RAC depends on how you inpaint removed pixels, and influence functions evaluated on the test set are implicitly using that test set for model selection. None of this makes the metrics useless, but it does mean you should not rely on a single number.

The HSIC experiment was a good example of a subtler version of the same issue. Varying $\alpha$ across several orders of magnitude produced no visible effect on accuracy because the kernel bandwidth was too small for the embedding dimensionality, making the loss term effectively zero. The metric looked like it was doing something (the loss value changed) but it was not actually doing anything (the gradients were negligible). Getting null results from a regulariser is easy to mistake for a confirmation that the method is not needed, when really the implementation was quietly broken.

The most transferable lesson from the PoE section is the greyscale intervention: instead of adding a loss term to discourage colour, removing colour from the input entirely is a stronger and more interpretable guarantee. Architectural constraints beat soft penalties when you can afford them.

Acknowledgments

These tasks were completed as part of the Trustworthy Machine Learning Course 2024/25 taught by Prof. Seong Joon Oh at the University of Tübingen, in collaboration with Ali Zhunis, and Esma Doğrusözlü. Team name Aldebaran (an orange giant star) was assigned by the course organization. Compute was provided by Kaggle.

Footnotes

1. Evaluating HSIC's contribution rigorously also requires ablating α=0 on identical training runs, not just sweeping α on the test set. Any hyperparameter choice tuned against the test set is quietly using test information.↩

Thanks for reading! If you have questions, feedback, or want to discuss a project, feel free to drop me an emailor message.