Provenance | Devpost

Inspiration

The most expensive bugs are often not caused by bad code. They happen when someone makes a reasonable change without knowing the reason the system looks the way it does.

Picture a developer cleaning up an old checkout flow. They find an awkward confirmation step that slows the request down, so they remove it. What they cannot see is the conversation from a year ago: the payment provider sometimes processed a charge but timed out before responding. That extra step was added after customers were charged twice.

The decision was never lost on purpose. It was spread across a Slack thread, an incident review, and a PR that only explained the immediate fix. By the time someone touches the code again, nobody knows where to look.

This happens constantly in growing teams. The code survives, but the lessons behind it do not.

Provenance connects code to those lessons, so teams can build on what they have already learned instead of learning it again the hard way.

What It Does

Provenance is a VS Code extension that gives developers the context behind the code they are changing.

A developer highlights a block of code in VS Code and asks Provenance to explain it. Provenance reads the selected code, nearby function context, and Git history to identify the commits and pull requests that shaped it. From there, it finds the Slack conversations, tickets, and, when available, Sentry incidents connected to those changes.

Provenance does more than search for similar words. If a Slack thread directly mentions the relevant pull request, commit, or file, it is returned as direct evidence. It also finds related conversations that may use completely different language than the code, such as a design debate, a customer requirement, or a production incident.

The result is a short, cited explanation inside VS Code, along with links to the original sources and a visual timeline showing how the code evolved. Developers can see the PR that introduced a change, the Slack discussion where the team debated it, and the Sentry incident that may have changed the team’s approach later. If Provenance cannot find reliable evidence, it says so instead of creating a plausible but unsupported answer.

How We Built It

Provenance is a VS Code extension for answering a simple question: why is this code here?

A developer highlights code in their editor, and Provenance traces it back through Git history, pull requests, Slack conversations, andSentry incidents. It returns a short answer with links to the original evidence and a timeline showing how the code changed over time.

There are two parts to the system: one prepares Slack data for search, and the other answers questions from VS Code. Both use Elasticsearch.

Slack Ingestion

Before Provenance can answer questions, it turns Slack history into a searchable knowledge base.

It reads approved Slack messages and groups related messages together. This includes normal Slack threads and unthreaded conversations separated by time gaps.
It extracts useful references such as PR numbers, commit SHAs, file paths, and code symbols.
OpenAI creates a short summary of each conversation and embeds that summary for semantic search.
Elasticsearch stores the original messages, summary, embedding, and metadata together.

We embed the summary instead of the raw Slack messages. That makes it easier to search for the idea behind a discussion, even if the code and the conversation use different words.

Each Slack conversation gets a stable ID, so running ingestion again updates existing data instead of duplicating it.

Answering a Question in VS Code

When someone highlights code, FastAPI starts two jobs at the same time:

Git blame and line history find the commits that touched the selected code, including older versions that were later replaced.
OpenAI turns the selected code into a plain-English engineering description—the kind of language someone would use in Slack.

This matters because Slack conversations rarely contain a pasted block of code. They usually explain the bug, feature, tradeoff, or incident behind it.

Provenance then searches Elasticsearch in three ways:

Exact matches for PR numbers, commits, file paths, and symbols.
Keyword search for relevant technical terms.
Semantic search for related discussions that use different wording.

The results are combined and filtered before OpenAI writes a short answer with inline citations.

Evidence, Not Just an Answer

Not every result is equally reliable.

A Slack thread that directly names the relevant PR or commit is treated as verified evidence. It is an exact match, not just a similarity score, and it stays in the result even if later ranking steps disagree.

Semantic matches can still be useful, but they are marked as inferred context. In the timeline, those connections use dashed edges so developers can tell them apart.

Putting It Together

FastAPI combines the Git history, Slack evidence, pull request details, optional Sentry context, generated explanation, and timeline into one response for VS Code.

The result is not just “here is what this code does.” It shows where the code came from, who changed it, what the team discussed, and whether later changes replaced an earlier decision.

If Provenance cannot find solid evidence, it says “no relevant context” rather than making up an answer.

Challenges we ran into

The main challenge was joining information that was never designed to connect. A line of code might point to a commit, then a PR, while the explanation only exists in a Slack thread.
Slack is also messy source material. We had to group threads and message bursts, pull out PRs, SHAs, file paths, and symbols, then separate useful technical discussion from noise.
Search could not be allowed to guess. We needed a way to show exact Git-backed matches differently from semantic matches, and return “no relevant context” when neither was strong enough.

Accomplishments that we're proud of

We built the full path from a highlighted line in VS Code to Git blame, PR history, Slack evidence, and optional Sentry incidents.
The search combines direct references with semantic retrieval. A thread that names the right PR is treated as verified evidence, while related discussions are still useful but clearly marked as inferred.
We turned the result into a visual evidence graph. Developers can follow the chain from code to commits, PRs, people, Slack threads, and incidents instead of trusting a summary alone.
We kept the system local-first. Elasticsearch stores the searchable Slack context on the developer’s machine, and the VS Code extension, CLI, and MCP server all use the same FastAPI service.

What we learned

The best results come from combining AI with hard evidence. Git and explicit IDs provide the anchors; OpenAI helps translate code into the language used in Slack and explain the results.
Exact and semantic search solve different problems. Exact matches are great for PRs and commits, while embeddings help find a discussion when the team used different words than the code.
Failure handling matters as much as the happy path. Git history may be missing, a service may be down, or there may simply be no useful discussion. Returning a partial answer is better than inventing one.
We also learned to test retrieval directly. Our evaluation cases cover known matches, no-context queries, and older decisions that have been superseded by later code.

What's next for Provenance

Improve code-aware matching with stronger symbol parsing and better links between files, functions, and related changes.
Make the evidence graph more useful for long-lived code by showing conflicts and superseded decisions more clearly.
Expand the project beyond a single local repository with stronger access controls, multi-repo support, and safer shared deployments.
Keep polishing the VS Code experience, especially CodeLens and status-bar hints, so developers can notice relevant context before they start changing code.