Repository navigation
Fix GED scoring for large isomorphic CFGs - #57
Merged
Merged
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Fix GED evaluation so graph size can never prevent a perfect structural match:
nx.is_isomorphicfor every CFG pair, with no size limitClaude Code
fallbackSortJoern still gets source CFGs exclusively from compiled
.ifiles. DecBench strips system headers and writes that same.itext to a temporary file with a.csuffix because Joern's C frontend expects that suffix; it does not read or fall back to the project's original.c.For
bzip2/O2/bzip2::fallbackSort, Claude recovered the source code, including its local macros. The historical candidate parse did not expand those macros:|78-103| + |115-159| = 69After local macro expansion, the candidate CFG is also 78 nodes / 115 edges and is role-aware graph-isomorphic to the source, so its corrected GED is 0.
Full-dataset impact
The completed reevaluation covered all 6,219 current artifact slices:
The historical 60-node census found 18,002 affected CFG pairs containing 29,977 individual graphs over the old limit. A pair contributes two graphs when both source and candidate exceeded 60 nodes.
Among historical-large scores, 52 became lower and 15,962 became higher. The predominantly higher distances are expected: the old node/edge-count fallback was only a weak lower bound, whereas graphs up to 200 nodes now receive VJ-GED after exact isomorphism fails.
The literal historical replay passed all 18,002 pairs, including unequal-count
pairs and both raw/sanitized variants for ambiguous legacy slices:
and 0 reconstruction mismatches
counts made the old fallback return 0
unavailable historical-large pairs
Site-update preparation
An isolated candidate was built at
/tmp/decbench-ged-site-candidate-eccsksw3; no live result, site, or dataset treewas changed.
provenance-bound
points of movement
kunadecompiler backend (+ kuna SLEIGH type aliases) #2 to decbench v2: declib-free backends, multi-version, caching, datasets, … #1 on sample-set GED (+11.38 percentage points;+11.59 with normalized failures), explained by 18 newly-perfect corrected
functions including
fallbackSortlargepreset occur among rates at or below1.34%, where a handful of functions changes several tied positions
The candidate also records one non-score metadata delta: current
mainadds itsexisting LLM-recall warning to the sample-set description, while the frozen live
aggregate predates that prose commit. The guard permits exactly that suffix and
requires all other preset metadata to match.
Validation
tests/test_content.pyfailures reproduce unchanged onmaingit diff --checkpassmainPublication scope
This PR contains the metric, parser, reevaluation/audit tooling, tests, and explanatory source documentation. It does not commit regenerated
site/output, live score overlays, or Hugging Face dataset data. Those remain a separate synchronized publication step after review of this PR and the isolated candidate report.