Historico de resultados
Pagina principal
Analisis de las pistas
Logo

Yasu Analytics (Desk 48)

Inspiration

Fraudulent calls using synthetic voices are escalating fast in call centers, and the people most exposed are the ones who bank by phone because they can't or don't use an app. We didn't want to build "yet another audio classifier" that just listens for spectral artifacts. Altur's challenge pointed at something more interesting: every call in their dataset already has an AI agent on one side, asking questions, interrupting, going silent. That's a second, mostly untapped signal, not just how a voice sounds, but how it behaves in conversation. We wanted a system that uses both.

What it does

Yasu Analytics takes a recorded phone call, caller on one channel, Altur's AI agent on the other, and returns a verdict: is the caller a real human, or a synthetic voice, with a calibrated confidence score. Under the hood it splits the channels, detects exactly where each side is speaking, and extracts two families of features: acoustic (how the voice itself sounds) and conversational (how the caller responds: latency, overlaps, and how they recover when the agent interrupts or goes silent). Both feed into a trained classifier ensemble exposed over a single POST /detect endpoint, matching Altur's exact spec, ready to sit in front of a real contact center line.

How we built it

We treated this less like "train a classifier" and more like "earn the right to trust a signal," in five gated phases:

Phase 0: audit the data before trusting it. Before writing a single feature, we checked whether the dataset had shortcuts that had nothing to do with being an AI voice. We found two: synthetic calls run ~7dB louder on average, and about two-thirds of them sit on a completely silent background channel — a fingerprint of how the audio was likely injected rather than captured over a real line. We flagged both to Altur instead of quietly building on top of them, and made sure our real features go beyond these tells.
Phase 1: build our own ears. The dataset ships with turn-level metadata (turns/), but it undercounted caller speech and, worse, misread the agent's own echo bleeding into the human caller's channel as if the caller were speaking. We trained and tuned our own voice-activity detector (Silero, threshold and silence gap picked only on train) and used it everywhere — training and serving — so there's no mismatch between what the model learned on and what it sees live.
Phases 2–3: two signals, not one. Acoustic features captured how the voice sounds; conversational features captured latency medians, turn-taking, and recovery from the agent's interruptions and silences — the "humans recover messily, machines recover consistently" idea from the brief.
Phase 4: model + calibration. We trained CatBoost on the combined feature set and layered a weighted soft-voting ensemble on top, tuning everything on train and only ever scoring on val/held-out data to keep ourselves honest.
Phase 5: ship it. A POST /detect API, deployed on a GPU-free Vultr instance, since none of our models need one.

We also scoped, but deliberately gated, two further ideas: a semantic layer (asking the agent's "does this detail exist?" questions and checking whether the caller says "I don't have that" versus inventing an answer, like an LLM would) held pending Altur's sign-off on sending audio to a third-party model, and voice fingerprinting of repeat synthetic callers, as a stretch goal.

Challenges we ran into

The biggest one was resisting shortcuts that looked like great results. Our very first single-rule baseline — "if the caller takes more than 1.6s to respond, it's AI", already hit 63/71 on val, and the volume/silence artifacts we found in Phase 0 could have pushed accuracy near-perfect without the model actually learning anything about being synthetic. We had to consciously separate "this feature is real signal" (caller latency held up at AUC 0.85–0.88 even after controlling for the artifacts) from "this feature is a dataset artifact we shouldn't ship." The agent-echo bleed-through in Phase 1 was a similar trap — it would have quietly corrupted our latency features if we hadn't built our own VAD to catch it.

Accomplishments that we're proud of

Going from a 63/71 hand-written rule to a calibrated ensemble that hits 0.99 accuracy and 0.9999 ROC-AUC on clean data (2 errors out of 353 calls), and still holds up at 0.92 accuracy and 0.975+ ROC-AUC on a much harder, 1,128-call stress set with noise and augmentation. We're just as proud of the process behind those numbers: we caught and documented two data-leakage risks before they could inflate our score, and kept a strict train/val separation the entire project instead of tuning against the same set we reported on.

What we learned

That in fraud/deepfake detection, the biggest risk usually isn't an underpowered model, it's an overpowered shortcut. The most valuable hour we spent wasn't training the classifier, it was auditing the dataset for anything that could let a model "cheat." We also learned that combining a behavioral signal (conversation dynamics) with an acoustic one is genuinely complementary: under stress-test noise, the acoustic-heavy ensemble lost a bit of recall on human callers, while the conversational timing signal kept holding up, which is exactly the kind of robustness a bank would need on real, messy phone lines.

What's next for Yasu Analytics

Closing the human-recall gap under noisy/degraded audio, most likely by making the conversational features more noise-robust rather than leaning harder on acoustic ones. Longer term: turning on the semantic layer once data-sharing terms with Altur are worked out, and voiceprinting detected synthetic voices so a bank can flag the same cloned voice reappearing across different calls or even different institutions.