Code-Witness Bot

Long form comment feature.
Inline comment feature.

Inspiration

Code reviews are essential, but they can be time-consuming. Whether you're maintaining a busy open-source project, collaborating on a team, or working solo, the burden adds up. Contributors don't always receive clear guidance, and even experienced developers can benefit from a second opinion. That's why we built Code-Witness, to ease the review process with thoughtful, automated feedback. It acts as an extra set of eyes, helping you spend less time reviewing and more time building.

What it does

Code-Witness is a GitHub bot that uses Perplexity's Sonar Deep Research model to automatically analyze pull requests and post review comments. It serves to quickly understand code quality, adherence to best practices, and alignment with industry standards. It can post these reviews in two modes: (1) inline comments or (2) a large unified review comment, which can be toggled by a configuration. Optionally, a commit message directive (code-witness: <custom instructions here>) allows developers to steer the bot's focus for even more precision. Also, the bot filters out non-source code diffs such as development configs, text files, images, etc., in order to be cost-effective with its calls to Sonar.

How we built it

We developed Code-Witness using a full Python stack, with FastAPI serving as the backbone of the webhook server. For deployment, we used Render to ensure a fast and reliable cloud environment. The GitHub app handles authentication and event triggers, while HTTPX was used for asynchronous calls to both the GitHub and Sonar APIs. We built a custom parsing pipeline using regular expressions to extract additions and deletions from raw git diff output. This parser filters out binary blobs and non-source code files so that only the most relevant code changes are sent to the Sonar Deep Research model, keeping token usage low and analysis focused.

Challenges we ran into

We knew that we didn't want to overload Sonar's context window with irrelevant diffs (e.g. text, image, dev environment configs, etc.), so we set out to look for existing diff parsers that could filter this out. None of them met our needs. Unsatisfied with those existing solutions for one reason or another, we set out to build our own parser. Our parser keeps track of file names, line numbers, additions/deletions, and enforces character/token caps, and source-code-only filters. This was also our first time building a GitHub bot, and navigating the extensive documentation wasn't always straightforward. Some endpoints looked similar but behaved differently in subtle ways, which led to a few misfires that were only solved by revisiting the docs in more depth. On top of that, we had to structure Sonar's responses in a way that would translate cleanly into GitHub's review UI. Getting all of these pieces to work smoothly took a lot of iteration.

Accomplishments that we're proud of

We're proud of how practical and refined Code-Witness has become:

Installation is frictionless. You can add Code-Witness to any GitHub repository in minutes with a simple GitHub app install and a one-line .code-witness.yml config
Implementing both review comment modes
Coming up with ingenious ways to add configurability to a GitHub app:
- Having the bot read a YAML config to switch review comment modes
- Embedding custom instructions into commit message descriptions, keeping developers in control of analysis
Always getting clean responses from Sonar's structured outputs
Parser optimizes token usage
Successfully deploying Code-Witness from a local development environment to a dedicated server and making it publicly available for installation

What we learned

Throughout this process, we learned more about real-world development workflows, specifically the importance of cost management, error resilience, and user-centric feedback design. Collaboration was key in solving the most important tasks. We also felt that we struck a balance between automation and human oversight in software development workflows.