Arbor

The self-organizing memory graph that eliminates context drift and reduces LLM token costs

Comment

Inspiration

The inspiration for Arbor came from a challenge every developer faces: The longer a conversation with an LLM continues, the more expensive and less focused it becomes.

Standard LLM architectures treat conversation history as a flat, append-only list. This creates two critical failures: Exponential Cost (re-processing the same history endlessly) and Context Drift (irrelevant details from previous turns diluting the current answer).

We realized that effective memory works like a Graph, not a text file. We built Arbor to prove that treating memory as a hierarchical file system rather than a linear stream could reduce token costs by 90% while simultaneously improving response precision.

What it does

Arbor is a Self-Organizing Memory Graph for LLMs. Instead of a flat chat history, it organizes conversations into a dynamic tree structure.

  • Context Isolation: It automatically detects topic shifts (e.g., switching from "Coding" to "Project Planning") and branches off into a new node, preserving the clarity of the current context.
  • Pruning: It constructs a "Context Window" by traversing only the Active Lineage (Root → Parent → Child), effectively removing irrelevant sibling branches from the API call.
  • Visual Topology: It provides a real-time graph visualization of the AI's memory state, allowing users to visually "Time Travel" between different threads of thought without losing their place.

How we built it

We built Arbor using a Recursive Graph Architecture powered by Google Vertex AI.

  • Intelligence: Gemini Flash handles generation, while Gecko provides high-speed vector embeddings for state routing.
  • Routing Logic: We implemented a custom "Router Agent" that calculates the cosine similarity between the user's prompt and all available memory nodes to decide whether to STAY, SWITCH, or CREATE a branch.
  • Visualization: We used Graphviz to render the dynamic memory topology in real-time.
  • Frontend: The interactive chat interface is built with Streamlit.
  • Observability: We integrated Datadog LLM Observability to track token usage, stability scores, and latency, proving our efficiency gains in real-time.

Challenges we ran into

Building a "Self-Organizing Memory" presented three major engineering hurdles:

  1. The "Sticky Router" Problem: Early versions were too tolerant of topic changes. If a user switched from Camping to Cooking, semantic overlap caused the router to stay in the incorrect branch. We resolved this with a "Compete, Don't Settle" algorithm that compares the Current Branch against the Best Sibling before making a routing decision.
  2. The "Context Saturation" Dilemma: Users would sometimes drift slowly off-topic within a single node, diluting the context quality. We addressed this with Tiered Confidence, adding a UI layer that displays a "Stability Warning" (Yellow Bar) when semantic relevance drops, encouraging the user to create a new branch.
  3. UI/Logic Decoupling: We initially used smoothed UI metrics for logic decisions, which caused synchronization issues due to lag. We learned to decouple the Visual Experience (Smoothed Momentum) from the Router Logic (Instant Raw Scores).

Accomplishments that we're proud of

  • The "Sawtooth" Cost Graph: We successfully demonstrated that Arbor's token usage drops drastically when switching topics, whereas standard chatbots experience linear cost growth.
  • True Context Isolation: We can discuss distinct topics in separate branches without the AI conflating them—a capability that standard long-context models often struggle to maintain over time.
  • Self-Aware UX: We built a system that provides transparency into the AI's understanding via our "Context Stability" health bar, giving users immediate feedback on the relevance of their current context.

What we learned

The most profound lesson was that Precision is more powerful than Scale. In an era of massive context windows, it is tempting to feed all available data to the model. However, Arbor demonstrated that intelligently Pruning irrelevant context creates a "Clean Room" environment where the AI can focus entirely on the task at hand, resulting in higher quality outputs and lower costs.

What's next for Arbor

  • Semantic Re-parenting: Automatically detecting when two sibling nodes share a latent abstract topic and restructuring them under a new parent node.
  • IDE Integration: Bringing Arbor's branching memory directly into development environments like VS Code for managing complex refactoring tasks.
  • Multi-Modal Branching: Enabling branches to store specific images or file references relevant only to that thread.

Built With

Share this project:

Updates