Code Graph (MCP)
This benchmark measures what an agent spends to answer a real question, with and without @ttsc/graph. The same prompts are run against three comparators: codegraph, codebase-memory-mcp, and serena.
Bar labels such as 486k tokens are token usage. Graph or comparator rows also show the token reduction versus the no-MCP baseline for that case.
Common Prompt
The common lane uses my real repository-onboarding prompt. It is the one I use when I enter a new codebase.
The same user prompt runs across every project. This tests whether a tool keeps broad orientation cost flat as the codebase changes.
Switch models with the tabs. Each chart row is one repository.
Loading graph benchmark results...
Dedicated Prompts
The dedicated lane uses repository-specific user prompts from the codegraph benchmark prompt family. This is the borrowed-prompt lane.
It checks all four tools on another tool’s per-repo questions. It is not the shared onboarding prompt above.
Switch projects with the tabs. Each chart row is one measured model.
Loading graph benchmark results...
Time to an Answer
The charts above measure what a question costs once a tool is ready. This one adds the readiness itself. Each bar is the two waits a developer actually sits through, in the order they are paid: the cold index build (the faded segment), then the LLM answering with that index in hand (the solid one). A tool that cuts an agent’s token bill and then spends four minutes indexing, and three more re-searching what it indexed, has moved the cost rather than removed it.
The wait is not evenly distributed. On VS Code’s three million lines, codegraph spends twelve minutes building and serena almost five, where @ttsc/graph, which reads the graph the TypeScript compiler has to build anyway, takes under thirty seconds.
Every tool is given the index its own documentation prescribes: codegraph init, codebase-memory-mcp index_repository in its default full mode (the mode the token charts above also ran), and serena project index, which serena recommends for larger projects. Repositories are ordered by the size of the program each index was built from.
Loading answer times...
How to read it
@ttsc/graph holds a flat, low median token cost as the repository grows. The comparators swing with repository size.
On some cases they land above the no-MCP baseline. That means the agent spends more with the tool than without it.
This is one person’s benchmark over a bounded set of scenarios. The launch post covers the method in full and where the numbers hold or break down. The Comparison page explains the design reasons behind the gap.
What is compared
@ttsc/graph exposes one typed MCP tool. Its request and result contract lives in the typed sources under packages/graph/src/structures.
The three comparators run from their own published entry points. They stay separate arms rather than folding into one story, since each resolves and returns a graph differently.
The Comparison page lays those differences out.
Reproduce it
Start from a clean checkout. Install the workspace, build @ttsc/graph, then run the same prompt once without MCP and once with the graph arm.
git clone https://github.com/samchon/ttsc.git
cd ttsc
corepack enable
pnpm install
pnpm --filter @ttsc/graph build
pnpm --dir experimental/benchmark graph:agent:codex -- --repo=typeorm --repo-dir=/abs/path/to/project --tsconfig=tsconfig.json --question "Trace the main runtime path I should understand before editing." --arm=baseline --runs=1
pnpm --dir experimental/benchmark graph:agent:codex -- --repo=typeorm --repo-dir=/abs/path/to/project --tsconfig=tsconfig.json --question "Trace the main runtime path I should understand before editing." --arm=graph --runs=1The chart reads raw samples from website/public/benchmark/graph.json. For the full model matrix, comparator setup commands, and publication flow, see experimental/benchmark/README.md.
