ConnectorForge | Devpost

Forging
Forge
Problem
Architecture
Landing Page

Inspiration

A few months ago, a friend on a data team asked me how long it would take to "just pull some weather data into our warehouse." His company already had Fivetran. The problem was that Open-Meteo - the API he wanted - wasn't one of Fivetran's 700+ prebuilt connectors. The answer was: get an engineer to write a custom connector, test it, deploy it, monitor it. A week of work, minimum. He gave up.

That story stuck with me. Fivetran has built an incredible platform - but the world has millions of APIs, and only a few hundred have official connectors. Every analyst at every company hits this wall. The data they need exists. It's reachable. They just can't get to it without an engineer.

I wanted to find out: can an AI agent close this gap?

What it does

ConnectorForge takes two inputs - a documentation URL and one sentence of intent - and produces a working Fivetran custom connector with real data in BigQuery. End to end. Autonomous.

You paste https://open-meteo.com/en/docs/historical-weather-api and write "daily weather for Mumbai from June 2024." The agent does the rest:

Reads the API docs. Gemini parses the documentation into a structured connector plan - base URL, auth, pagination, response shape.
Resolves intent → blueprint. Your sentence becomes configuration.json: start dates, city coordinates, sync spec.
Probes the live API. Before writing code, the agent makes real HTTP calls, negotiates working parameters, and runs happy/empty/null probes. Ground truth is the API response, not the docs' description of it.
Assembles the connector. Schema is extracted from real responses; connector.py is rendered from Jinja templates (Fivetran Connector SDK - schema, incremental update, checkpointing, null-safe coercions). Not free-form LLM codegen.
Self-heals through debug. Runs fivetran debug locally. On failure, Gemini 2.5 Pro reads stderr + the live API response and rewrites connector.py. Up to three iterations.
Deploys and verifies. Runs fivetran deploy, triggers sync via Fivetran REST API, polls until complete, samples BigQuery to confirm rows landed in the requested date range.

The UI streams all four phases live over Server-Sent Events - doc read, codegen, debug loop, deploy.

Timing (honest): codegen + debug often finishes in 1–3 minutes. Full deploy through BigQuery verification is typically 4–10 minutes depending on API and Fivetran sync latency. One verified E2E run (Open Brewery DB) landed 500 rows in BigQuery.

How I built it

Architecture. Google ADK orchestrates five sequential sub-agents: DocInterpreter, SpecResolver, CodeGenerator, DebugLoop, and Deployer. State flows through ADK session keys - each phase writes structured JSON, the next reads it. The forge UI collapses this into four visible steps.

The LLM strategy. Gemini 2.5 Flash for doc parsing, parameter correction on 4xx errors, and schema extraction - fast, structured. Gemini 2.5 Pro only for debug patches, where reasoning over real Python and stderr matters. Every call uses Pydantic response_schema on Vertex AI - no free-text parsing in the pipeline. Optional Groq fallback and exponential backoff handle Vertex 429s; GCP_LOCATION=global reduces rate-limit pain.

Where deterministic code wins. Connector assembly is Jinja templates, not LLM-generated Python. URL/auth/pagination normalization (url_normalize.py, demo_presets.py) fixes common LLM mistakes for known APIs without per-API hardcoding. This eliminated an earlier "rescue chain" that patched broken LLM output after the fact.

Fivetran integration (accurate split).

Connector SDK CLI: fivetran debug and fivetran deploy - connector creation and local validation.
Fivetran REST API: sync trigger, unpause, and status polling in the deploy phase (direct HTTP - not LLM-mediated).
Fivetran MCP (stdio): available for connection listing and orchestration patterns; deploy polling intentionally uses REST for tight loops without burning tokens.

MCP does not create connectors. Creation is always fivetran deploy.

Frontend. Next.js 14 on Vercel, FastAPI on Railway/Cloud Run. SSE connects directly to the backend (NEXT_PUBLIC_BACKEND_URL) so the forge stream isn't buffered. Acid Electric palette - chartreuse on forest black, IBM Plex Mono.

Challenges I ran into

Architecture v1 failed. Gemini writing connector.py directly from docs worked ~40% of the time. Wrong URLs, invented params, missing null guards. I rebuilt: probe live API first → extract schema from evidence → template assembly. Reliability jumped.

Vertex rate limits. Three parallel doc calls hit 429 instantly. Fixed with backoff, global endpoint, optional Groq for doc parsing, and throttling between probes.

Fivetran API quirks. fivetran deploy --api-key wants base64 key:secret. Destination "ID" is the destination name in the UI. Connections may need unpause before sync. Each cost hours to discover.

Stale plan cache & demo drift. Cached connector plans could serve wrong pagination (none instead of date_range for Open-Meteo). refresh_connector_plan() re-applies normalization on every load.

Reused connections & BigQuery validation. Redeploying to the same connection name leaves old rows in BigQuery. Validation now checks rows in the requested date window, not global table MIN date.

Accomplishments I'm proud of

The pipeline works end-to-end on multiple REST JSON APIs: Open-Meteo, Frankfurter, Open Brewery DB, USGS Earthquakes, and REST Countries (with API-key normalization).

I scoped it honestly: REST JSON APIs, not GraphQL or arbitrary auth flows. The architecture generalizes - new APIs don't need new pipeline code, but unknown APIs can still fail.

Every required Fivetran surface is real: Connector SDK, CLI deploy/debug, REST sync orchestration, BigQuery verification. No mocks in the deploy path.

What I learned

Push the LLM into narrow, evidence-based decisions: "given this real response, what's the schema?" and "given this stderr, what's the fix?" Let deterministic code handle templates, HTTP, CLI, and polling.

The Fivetran Connector SDK is well-suited to agentic codegen - incremental, checkpointed, idempotent once you match its patterns.

Pydantic + response_schema is the right contract. Malformed output fails cleanly instead of corrupting downstream state.