Loop

An agent that closes the loop on its own mistakes, remembering what worked and skipping what didn't, run after run.

Comment

Inspiration

Most agent demos look smart on the first call, but fall apart on the second. Real tools are messy. They can have missing IDs, ambiguous search results, or even return "success" responses that are actually unscoped or unusable data.

We wanted to build an agent that can fail in a teachable way and then get better when it encounters a differently worded task. The improvement should not come from hard coding another prompt. Instead, the agent should store evidence from its previous failure and reuse what it learned.

That felt like the heart of Syndicate's Track 1 brief: automated agent engineering that learns how to use tools over repeated runs, with measurable improvements in success, latency, and tool hygiene.

What it does

Loop is a support and CRM agent that has to learn one important dependency. It needs to resolve the customer_id by using search_customers before it can correctly call list_orders.

The demo is built around three runs. We refer to them as Agent v1, Agent v2, and Agent v3, based on the points where the lesson is promoted.

  1. seen_a (v1, naive)

The agent calls list_orders first and receives a teaching miss because the customer_id is missing. This creates a candidate lesson. The run takes approximately 4278 ms, has a speed score of 0.70, and has 0% task success.

  1. seen_b (v2, promotion)

The agent receives the same intent with slightly different wording. Reflection sees the same failure pattern and promotes a usable Neo4j lesson after receiving at least two pieces of evidence. The naive path still fails, so task success remains 0%, but the run takes approximately 564 ms and has a speed score of 0.95.

  1. unseen (v3, post lesson)

The agent receives completely new wording, such as "Morgan Blake… delayed delivery." Strategy injection retrieves the learned lesson through shared factors. The agent now starts with search_customers, then calls list_orders, and finally calls create_ticket.

The task succeeds with 100% task success, takes approximately 329 ms, and has a speed score of 0.97. The failed list_orders calls drop to 0.

The improvement is supported by explicit signals. Success increases on the unseen task, and lessons are only retrieved after they have passed the promotion threshold.

The live AGENT_SCORECARD on the dashboard shows task_success_rate, speed, robustness, latency, average failure and drift signals, and the dominant first tool for each version.

How we built it

We built the system around Agent Orchestrator, or AO, as an explicit learning loop.

Tools

We use fixture CRM webhooks through a Node mock or n8n. These tools intentionally contain the kinds of quirks that can cause an agent to make mistakes.

Planner

We use TensorMux and Ollama when they are available, along with a deterministic offline planner for the demo.

Working memory

SQLite stores the information needed during the current run.

Episodic memory

We use SQLite vectors with Ollama's nomic-embed-text, with a hash based bag of words fallback when needed.

Analyzer

The analyzer identifies bad tool usage using Neatlogs spans or working memory logs.

Reflection and semantic memory

Candidate lessons go through an evidence gate before becoming usable lessons in Neo4j Aura. We also have a local JSON fallback.

Strategy injection

Usable lessons are injected into the next run. This supports both direct retrieval and retrieval through shared factors.

Evaluation metrics

Every run produces loop_eval metrics for speed, robustness, latency, drift, and task_success. We then build the Agent v1 to v2 to v3 scorecard around the first usable lesson promotion, without needing a separate instrumentation pass.

Replay and dashboard

The dashboard provides a cyberpunk interface with a live RUN DEMO, a trajectory spine, a lessons panel, and the scorecard. It is also deployed on Vercel, with an in process offline mock CRM so judges can run the demo without needing localhost.

What we learned

"Learning" is only believable when the first failure is intentional and the transfer task is worded differently.

Offline paths matter just as much as the LLM path. Using hash embeddings and an offline planner made the demo deterministic when we were working under hackathon time pressure.

Observability through Neatlogs and OpenTelemetry is not just decoration. It feeds the analyzer, which means reflection is working from actual evidence rather than vibes.

Evaluation also needs to sit on the same spine as the demo. When we bucket the results by lesson promotion boundaries, we can show the progression from v1 to v2 to v3 without creating a separate benchmark.

Shipping to Vercel also forced a real architectural decision. Serverless functions cannot host n8n or Ollama, so we moved the offline stack in process while keeping Neo4j Aura as the remote semantic memory.

Challenges

We had to deal with several technical challenges during development.

Windows, spawn, and SSE

We ran into issues with npx EINVAL and process wiring while getting the live dashboard working on Windows.

Neo4j Aura authentication

Neo4j Aura authentication was another challenge because the instance ID needs to be used as the username in some configurations rather than the default neo4j.

Stacked AO PRs

Our stacked AO PRs diverged from master after the first merge. We resolved this by retargeting the stack and merging the conflicts so that tip PR #10 remained mergeable.

Vercel versus localhost

TensorMux and n8n cannot run directly inside a Vercel function. We therefore made RUN DEMO work completely offline and in process, while keeping a path to connect to hosted n8n later through TOOLS_BASE_URL.

Scorecard UX

There was a case where the trajectory rendered correctly while the AGENT_SCORECARD remained empty because the session cache did not include the scorecard. We fixed this by rebuilding the scorecard from versioned rows on the client side.

UTF 8 BOM

A UTF 8 BOM in package.json briefly broke tsx and RUN DEMO on Windows. We fixed this by stripping the BOM.

What's next

Wire a hosted n8n deployment through Railway or Render into the live dashboard.

Add an optional cloud LLM behind the same TensorMux shaped /v1 interface.

Build stronger Neatlogs evaluation campaigns and support multi run scorecard aggregation, with more than one run per version.

Make the CRM fixtures deterministic, especially the create_ticket rate limit noise, so robustness scores measure learning rather than fixture failures.

Add richer trajectory analytics, including factor heatmaps and lesson provenance graphs from Neo4j.

Built With

Share this project:

Updates

Submission history