Heewon

A fine-tuned Qwen3.5-9B (LoRA) that reads a patched V8 bug, writes a JS/WASM proof-of-concept, runs it against the real vulnerable build, and iterates until something actually crashes.

We built this for a hackathon to answer a simple question: frontier models like Fable look strong at cyber work — can a much smaller open-source model get meaningfully closer with the right fine-tuning?

Short answer: yes. On our held-out browser-exploit tasks, Heewon improved Qwen3.5-9B by about 2.3× over the untuned base.

Give the model a patched Chrome V8 N-day (bug context + localized source). It:

1. Reasons about the vulnerability
2. Emits a self-contained JS or WASM PoC
3. Runs that PoC against the real vulnerable d8 binary
4. Reads the crash / coverage feedback
5. Tries again

Training uses the same execution-graded reward the benchmark uses. The model hill-climbs “did this actually crash the engine?”, not a proxy like BLEU or a judge LLM.

Everything is on already-patched N-days. The goal is measuring exploit-writing capability — not finding anything new.

mine real gold PoCs  →  SFT (capability)  →  GRPO against live oracle  →  eval
(SEC-bench / ExploitGym / V8 regress)        (vuln vs fixed d8)

The training environment (environment.py) is a multi-turn loop: write PoC → grade → observe → iterate. The reward lives on a hardened Modal oracle (oracle/), not inside the training worker.

The base Qwen already “knows” a lot of public N-days from pretraining (~68% memorization on the eval set). Absolute scores are misleading. We report delta over base.

On our 30-bug probe against an untuned baseline:

On the wider 88-bug fleet, the scaled SFT adapter crashes 10 / 88 bugs on a differential grader (up from 8 on the earlier backbone), including reliable crashes and ASAN-corruption-class hits on a few targets.

GRPO against a sparse reward mostly matched SFT rather than beating it — which told us the bottleneck was reward density and data ceiling, not the RL method. Scaling real exploit demos in SFT was the lever that moved the needle.

A gameable cyber reward is worse than useless — the model just learns to cheat. Ours is built so the easy cheats don’t work:

• Differential grading — PoC must behave differently on vuln vs fixed builds
• Crash = exit signal (rc < 0), never print("DCHECK failed")
• No JS-readable nonce to echo for a fake leak
• File I/O disabled during grading — PoCs must be self-contained, like the real harness
• Leak control — train set scrubbed against ExploitBench’s 41 eval bugs by crbug id and fix-commit SHA

We tried the shortcuts. The oracle killed them.

environment.py / environment_v2.py   # Freesolo multi-turn env + reward
configs/                             # SFT / GRPO TOMLs (sft_v3, rl_curriculum, …)
oracle/                              # Modal d8 fleet (vuln vs fixed grading)
data/                                # mining, leak control, curriculum, RFT
dataset/                             # SFT + RL jsonl
eval/                                # fast vLLM-merge eval + grading scripts
train/                               # harness / diagnostics helpers
demo/                                # deck + terminal demo
docs/                                # status, findings, dataset catalog

More detail lives in docs/STATUS.md, docs/DATASETS.md, and TRAINING.md.

# deps
pip install -e ".[dev]"
cp .env.example .env   # add FREESOLO_API_KEY (and oracle keys if grading)

# unit tests (local harness)
pytest

# train on Freesolo Flash (needs flash CLI + published env)
flash env push --name exploitbench .
flash train configs/sft_v3.toml --cost   # binding preflight
flash train configs/sft_v3.toml

# GRPO against the live oracle (set POC_EXEC_URL / POC_EXEC_KEY)
flash train configs/rl_curriculum.toml --cost
flash train configs/rl_curriculum.toml

Oracle deploy (Modal): see oracle/modal_fleet_v2.py / modal_fleet_v3.py and docs/DATASETS.md.

• OOM / context — 9B + long PoCs + GRPO group rollouts is VRAM-tight; configs are pinned to empirically safe shapes (batch_size=1, group_size=4, context capped).
• Sparse reward — most rollouts land rung 0–1; half of early GRPO steps had zero within-group variance → no gradient. Curriculum + denser ladders fixed the signal.
• Data scarcity — real verified V8 exploits are rare once you leak-gate the eval set. Mining V8’s regression corpus (and teacher-rationalizing it) was what raised the ceiling.
• Reward hacking — every shortcut we left in got farmed. Differential + exit-signal grading was non-negotiable.

Teaching a 9B model to write working browser exploits — not just sound like it can — and measuring that with a grader you can’t fake. Small model, real crashes, honest delta.

• ExploitBench — capability-ladder benchmark for V8 exploit writing
• SEC-bench / ExploitGym — patched vuln/fixed build sources
• Training via Freesolo Flash

Hackathon project. Authorized research on already-patched N-days only.