Skip to content

Fix GED scoring for large isomorphic CFGs - #57

Merged
mahaloz merged 6 commits into
mainfrom
fix/ged-isomorphism-large-cfgs
Aug 8, 2026
Merged

mahaloz merged 6 commits into
mainfrom
fix/ged-isomorphism-large-cfgs

Conversation

@mahaloz

@mahaloz mahaloz commented Jul 29, 2026

Copy link
Copy Markdown
Collaborator

Summary

Fix GED evaluation so graph size can never prevent a perfect structural match:

  • run directed, entry/exit-role-aware nx.is_isomorphic for every CFG pair, with no size limit
  • return GED 0 immediately for isomorphic graphs
  • raise the VJ-GED limit from 60 to 200 nodes
  • use a SciPy assignment solver equivalent to cfgutils' VJ cost model
  • use a nonzero size lower bound only for non-isomorphic graphs above 200 nodes
  • bump the GED cache version and strengthen graph cache inputs
  • expand locally defined macros in generated decompiler C before Joern parsing
  • make reevaluation provenance-bound, resumable, slice-complete, and safe for external decompiler IDs
  • add a standalone audit that literally replays every historical >60-node pair through NetworkX isomorphism

Claude Code fallbackSort

Joern still gets source CFGs exclusively from compiled .i files. DecBench strips system headers and writes that same .i text to a temporary file with a .c suffix because Joern's C frontend expects that suffix; it does not read or fall back to the project's original .c.

For bzip2/O2/bzip2::fallbackSort, Claude recovered the source code, including its local macros. The historical candidate parse did not expand those macros:

  • source CFG: 78 nodes / 115 edges
  • historical candidate CFG: 103 nodes / 159 edges
  • old >60 fallback: |78-103| + |115-159| = 69

After local macro expansion, the candidate CFG is also 78 nodes / 115 edges and is role-aware graph-isomorphic to the source, so its corrected GED is 0.

Full-dataset impact

The completed reevaluation covered all 6,219 current artifact slices:

  • frozen published scores: 453,391
  • corrected scores: 455,711
  • frozen scores missing afterward: 0
  • new scores: 2,320
  • changed comparable scores: 19,658
    • lower/improved: 2,048
    • higher: 17,610
    • became perfect: 1,148
    • lost perfect: 567
  • corrected perfect scores: 155,822 / 455,711 (34.2%)

The historical 60-node census found 18,002 affected CFG pairs containing 29,977 individual graphs over the old limit. A pair contributes two graphs when both source and candidate exceeded 60 nodes.

Decompiler >60 pairs >60 graphs Frozen scores Changed scores
angr 3,050 5,076 3,046 2,701
binja 2,652 4,471 2,652 2,344
claude-code 26 48 25 22
codex 28 48 28 24
dewolf 202 202 202 195
ghidra 3,122 5,258 3,122 2,793
ida 3,090 5,186 3,090 2,761
kuna 3,223 5,527 3,223 2,848
manifold 26 38 0 0
r2dec 2,583 4,123 2,577 2,326
Total 18,002 29,977 17,965 16,014

Among historical-large scores, 52 became lower and 15,962 became higher. The predominantly higher distances are expected: the old node/edge-count fallback was only a weak lower bound, whereas graphs up to 200 nodes now receive VJ-GED after exact isomorphism fails.

The literal historical replay passed all 18,002 pairs, including unequal-count
pairs and both raw/sanitized variants for ambiguous legacy slices:

  • 26,325 literal NetworkX calls over 3,100 artifact slices
  • 57 historically isomorphic / 17,945 non-isomorphic / 0 unavailable
  • 17,965 published fallback scores reconstructed exactly, 37 scoreless records,
    and 0 reconstruction mismatches
  • 70 historical false-perfect scores: non-isomorphic graphs whose equal node/edge
    counts made the old fallback return 0
  • under the corrected evaluator: 59 isomorphic / 17,933 non-isomorphic / 10
    unavailable historical-large pairs

Site-update preparation

An isolated candidate was built at
/tmp/decbench-ged-site-candidate-eccsksw3; no live result, site, or dataset tree
was changed.

  • all 455,711 promoted overlay records exactly match schema-6 checkpoints
  • 6,219 promoted slices and all 18,259 large-graph audit records are
    provenance-bound
  • the original nine gain 2,179 GED scores and lose 0
  • default unoptimized GED ranks are unchanged
  • major-dataset Union rank swaps are among near-ties with at most 0.70 percentage
    points of movement
  • Claude Code moves from Add the kuna decompiler backend (+ kuna SLEIGH type aliases) #2 to decbench v2: declib-free backends, multi-version, caching, datasets, … #1 on sample-set GED (+11.38 percentage points;
    +11.59 with normalized failures), explained by 18 newly-perfect corrected
    functions including fallbackSort
  • larger rank movements in the large preset occur among rates at or below
    1.34%, where a handful of functions changes several tied positions

The candidate also records one non-score metadata delta: current main adds its
existing LLM-recall warning to the sample-set description, while the frozen live
aggregate predates that prose commit. The guard permits exactly that suffix and
requires all other preset metadata to match.

Validation

  • 188 focused GED, CFG, cache, result-store, reevaluation, audit, site, and publishing tests pass
  • full suite: 472 passed, 7 skipped; the five tests/test_content.py failures reproduce unchanged on main
  • accelerated VJ matched cfgutils across 16,708 exhaustive small-graph comparisons
  • refined role-aware isomorphism matched plain role-aware NetworkX across randomized checks
  • Ruff, Black, and git diff --check pass
  • all 7,052 current decompiled C artifacts were checked; none contains a raw control byte affected by the parser sanitizer added in current main

Publication scope

This PR contains the metric, parser, reevaluation/audit tooling, tests, and explanatory source documentation. It does not commit regenerated site/ output, live score overlays, or Hugging Face dataset data. Those remain a separate synchronized publication step after review of this PR and the isolated candidate report.

@mahaloz
mahaloz merged commit bd49c83 into main Aug 8, 2026
5 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant