StudySherpa

An agentic study tutor that lives in your screen and keeps you in your flow. It notices when you're stuck and makes sure you don't forget it again.

Comment

Study Sherpa

An experiment in building a context-aware AI study companion with specification-driven AI coding.

Studying from a screen means constantly breaking concentration to ask for help. You switch tabs, copy a problem into a chatbot, explain your context, get an answer, and then try to reconstruct where you left off.

We wanted to see if an AI agent could instead behave like a study partner sitting beside you: always available, aware of what you're looking at, able to remember what you've struggled with, and capable of noticing when you've been stuck for too long.

So we built Study Sherpa, a desktop study companion designed around a simple interaction:

Press a hotkey → Study Sherpa looks at your screen → an agent figures out what you need → help appears without leaving your work.

What we built

Study Sherpa is an Electron application with three main pieces:

  • Desktop companion: A global hotkey launches the study flow without requiring a tab switch.
  • Agentic explanations: Claude can reason over the current context and decide which tools it needs rather than following a fixed chain of API calls.
  • Persistent memory: Explanations are stored so future questions can be informed by previous struggles.
  • Voice input: A push-to-talk hotkey was designed to feed spoken questions into the same agent loop using a local Whisper model through whisper.cpp.
  • Spaced repetition: Saved stuck moments enter a Leitner-style review queue with 1, 3, 7, 14, and 30-day intervals.
  • Proactive stuck detection: A lightweight background process watches for long periods of inactivity and unchanged screens, allowing Study Sherpa to offer help before the user asks.
  • MCP-first architecture: Core capabilities are exposed through a standalone local MCP server backed by SQLite, rather than being coupled directly to the Electron application.

The intended architecture looks roughly like this:

Electron → Agent → MCP tools → SQLite

The Electron app never accesses the database directly. The MCP server owns persistence, meaning the same study context could theoretically be accessed by other MCP-compatible clients.

The experiment

The most interesting part of Study Sherpa wasn't just the application itself.

We wanted to test a question:

How far can you get building a genuinely non-trivial application almost entirely through specification-driven AI coding?

Instead of manually implementing the application feature by feature, we gave the coding agent a detailed PRD, architecture documentation, component specifications, tool contracts, implementation constraints, and a prescribed build order.

We deliberately built in risk order:

  1. Get the basic hotkey → screenshot → Claude → response loop working.
  2. Introduce MCP as the persistence boundary.
  3. Build the overlay interface.
  4. Add voice as another entry point into the same agent loop.
  5. Add spaced repetition to the existing notes model.
  6. Add proactive stuck detection as an independent background system.

This approach produced a surprisingly large amount of functional architecture in a short period of time.

It also exposed exactly where the approach started to fall apart.

Where we ended up

Study Sherpa is an experimental prototype, not a finished application.

The UI, application architecture, MCP layer, review system, and individual pieces of the agent workflow were built, but we did not get the entire system to a reliable end-to-end state by the deadline.

In particular, integration issues around the desktop screen-capture pipeline prevented the core interaction from consistently completing. Some downstream error handling also exposed assumptions about the shape of responses from the agent/tool layer.

Rather than hiding that, we think it's one of the most interesting results of the project.

The experiment demonstrated that an AI coding agent can produce a surprisingly sophisticated architecture from a sufficiently detailed specification—but that generating the pieces is not the same as integrating and validating the whole system.

Technical challenges

Designing the tool boundary

One of our biggest architectural decisions was determining which actions should be controlled by the agent and which should remain deterministic application behavior.

The agent controls tools such as:

  • capture_screen
  • save_note
  • search_past_explanations

Meanwhile, deterministic UI actions such as retrieving due reviews and recording a review outcome are handled directly rather than asking an LLM to make those decisions.

That distinction ended up being more important than we expected.

MCP from the beginning

We intentionally introduced MCP early instead of retrofitting it later.

The MCP server became the single owner of the SQLite database, while the Electron application communicated with it through the protocol.

Partway through development, spaced repetition exposed a gap in our original four-tool design: there was no clean way to record whether a review was successful without violating our database boundary.

We chose to introduce a dedicated review_note tool rather than turning save_note into a generic upsert operation.

It was a small change, but it reinforced an important lesson: tool contracts are architecture.

Local Whisper instead of an API

We originally planned to use the OpenAI Whisper API for voice input.

We eventually rejected that approach because we wanted the application to have zero per-use transcription cost and to work without a cloud dependency.

Browser speech recognition also wasn't reliable inside Electron, so we moved to a local Whisper model through whisper.cpp.

That introduced its own trade-offs—model size and a native dependency—but gave us the offline, zero-marginal-cost behavior we wanted.

Keeping proactive detection cheap

We didn't want Study Sherpa continuously sending screenshots to a vision model.

Instead, the stuck detector uses local signals—idle time and screenshot hashing—to determine whether the user has been staring at an unchanged screen.

Only after the user accepts the resulting nudge does the expensive agent workflow run.

What we learned

The biggest lesson wasn't about Electron, Claude, or MCP individually.

It was about the difference between architectural generation and software engineering.

A coding agent can generate an impressive amount of code from a good specification. It can create interfaces between components, implement protocols, construct UI, and follow architectural constraints.

But as the number of independently moving pieces increases, integration becomes the hard part.

Study Sherpa ended up being a useful stress test of that boundary.

We started with a detailed specification and an ambitious architecture. We ended with a prototype that demonstrates much of the architecture, but not a reliable finished product.

And that's exactly what we wanted to learn.

What's next

Study Sherpa was designed as the first vertical of a broader personal-agent platform we call Jasmine.

The MCP-first architecture was intended to allow the study tools to eventually plug into a larger agent router without rebuilding the persistence layer.

If we continued the project, the next priorities would be:

  • Stabilizing the screen-capture → agent pipeline
  • Improving end-to-end error handling
  • Syncing with an Obsidian vault
  • Making the review system more adaptive
  • Expanding beyond screen-based study contexts

Study Sherpa isn't finished. But it gave us a pretty good answer to our original question: AI coding agents can get remarkably far—but getting from “all the pieces exist” to “the whole thing reliably works” is still a very different problem.

Built With

Share this project:

Updates