Inspiration
My grandmother got a text saying her account would close in 24 hours. She almost clicked it. What struck me afterwards wasn't that a filter had missed it, rather, even if a filter had caught it, she'd have learned nothing. Every anti-scam tool makes a silent yes/no decision inside a mail server, and the person actually holding the phone gets no reason, no explanation, nothing they could use next time. The people targeted hardest (older adults, anyone reading in their second language) are the least served by an English-only tool that needs an account.
What it does
Paste a suspicious text, email, or DM, or drop in a screenshot, and Skeptic tells you how dangerous it is and exactly why. Every point of the risk score traces to a named signal, the triggering words are highlighted inside the message, and the link breakdown shows which part of the address actually matters. Then it tells you what to do, tailored to the kind of scam, plus a plain-language warning you can forward to whoever received it. It runs entirely in the browser: no account, no server, no network requests, works offline, English and Spanish, three text sizes and a high-contrast mode.
How we built it
Four independent detectors vote on every message. A trained model (Naive Bayes + logistic regression over TF-IDF, stacked so a third model learns how far to trust each [which also calibrates the output]). A rule engine for the tactics a scammer can't drop: urgency, secrecy, untraceable payment, credential harvesting. A link analyser doing homoglyph detection, punycode, brand-in-wrong-position, edit distance to real domains. And a header analyser reading SPF/DKIM/DMARC when you paste a raw email. Scores are a plain sum, so every reason is attributable, and so is every counterfactual. React + TypeScript + Vite, Tesseract in WebAssembly for screenshots, all served from our own origin so nothing leaks.
Challenges we ran into
The big one: I pointed it at 5,572 real SMS messages and it fired on one legitimate message in three. It was scoring "Good night my dear" as romance fraud. The cause wasn't the rules, my training data had appointments and delivery notices and not one warm message between people who love each other, so the model had never seen how people actually talk. Separately, an ablation study reported 0.000 recall for the header layer, which looked like a broken detector but was actually a broken test: my holdout was almost all SMS with no headers to read.
Accomplishments that we're proud of
Publishing my own bad numbers. False alarms went 32.8% -> 5.5% after the fix, and the remaining 5.5% is in the app rather than hidden. The ablation table shows what every layer is worth, including that the model alone only reaches 84.4%. Zero false alarms across all 71 legitimate holdout messages. And the whole thing is honest about being a second opinion, not an authority.
What we learned
That a false alarm costs more than a miss. A scam you miss that still says "Caution" leaves someone warned; a false alarm teaches them to ignore every warning after it. So I deliberately traded three points of recall for a 6× reduction in false alarms. Also that you don't know what your model learned until you point it at data you didn't make. Everything I believed about it was true right up until it met a real inbox.
What's next for Skeptic
Real labelled training data instead of synthetic. That 5.5% is the measurable cost of not having it. More languages beyond English and Spanish, starting where scam volume is highest. Shipping the browser extension properly instead of load-unpacked. And getting the practice drill in front of the people it's for, because the detector helps with the message in front of you, and the drill is the part that's still working a year later.
Log in or sign up for Devpost to join the conversation.