Paste an API docs URL + one sentence of intent → get a deployed Fivetran connector syncing into BigQuery.

ConnectorForge is an AI agent pipeline built for the Google Cloud Rapid Agent Hackathon (Fivetran track). It reads API documentation, probes the live API, generates a Fivetran Connector SDK package, self-heals with fivetran debug, deploys via fivetran deploy, and verifies rows in BigQuery.

Built with Google ADK, Gemini 2.5 on Vertex AI, and the Fivetran Connector SDK + MCP.

Live UI: paste docs URL → describe what you want → watch 4 phases stream in real time.

Browser (Next.js)
    │  SSE  GET /api/forge/stream
    ▼
FastAPI + ADK SequentialAgent
    │
    ├─ 1. DocInterpreter     docs URL → connector_plan (JSON)
    ├─ 2. SpecResolver       plan + intent → blueprint + configuration
    ├─ 3. CodeGenerator      probe API → schema → Jinja connector.py
    ├─ 4. DebugLoop          fivetran debug → Gemini Pro patches (≤3 tries)
    └─ 5. Deployer           fivetran deploy → sync → BigQuery sample

MCP does not create connectors — only the SDK CLI deploys them.

• Agent: Google ADK SequentialAgent, deterministic phase callbacks
• LLM: Gemini 2.5 Flash (extract/schema) + Pro (debug patches) on Vertex AI
• Optional: Groq fallback for rate-limit relief (LLM_PROVIDER=groq)
• Backend: Python 3.11, FastAPI, Uvicorn, Pydantic structured outputs
• Frontend: Next.js 14, TypeScript, SSE live pipeline UI
• Infra: Fivetran Connector SDK CLI, uvx fivetran-mcp, BigQuery client

• Python 3.11+
• uv (pip install uv)
• Node.js 18+
• Fivetran account (API key + secret + BigQuery destination)
• GCP project with Vertex AI and BigQuery enabled
• Service account JSON with Vertex AI User + BigQuery Data Viewer

git clone https://github.com/shaileshdev4/connectorforge
cd connectorforge

cp .env.example .env
# Edit .env — see table below

# Place service account at infra/sa-key.json (or set GOOGLE_APPLICATION_CREDENTIALS)

make setup          # pip install -e .[dev] + fivetran-connector-sdk
make dev            # http://localhost:8080

Verify: curl http://localhost:8080/healthz → {"status":"ok"}

cd frontend
npm install
npm run dev         # http://localhost:3000

Open http://localhost:3000/forge, pick a preset, click Forge Connector.

.venv/Scripts/python.exe -m pytest tests/ -q    # Windows
# pytest tests/ -q                               # macOS/Linux

Vercel env (required):

NEXT_PUBLIC_BACKEND_URL=https://your-railway-app.up.railway.app

The UI opens SSE directly against the backend URL so the forge stream is not buffered by Vercel.

Railway: mount GCP credentials as GCP_SERVICE_ACCOUNT_JSON and write to infra/sa-key.json at container start. Allocate 2 GB+ RAM; forge runs can take 5–15 minutes.

agent/
  main.py              FastAPI entrypoint
  orchestrator.py      ADK pipeline
  forge_stream.py      SSE streaming
  pipeline/            V2: negotiate, probe, schema, assemble, debug
  sub_agents/          Phase wrappers
  templates/           Jinja connector templates
  tools/               Fivetran CLI, BigQuery, deploy helpers
frontend/              Next.js UI
tests/                 pytest suite (52 tests)
scripts/               E2E and isolation runners
connector_workspace/   Generated connectors (runtime, gitignored)

1. Structured doc extraction — 3 Flash calls with Pydantic schemas, no free-text parsing.
2. Live API negotiation — HTTP probe with retries; LLM fixes params on 4xx.
3. Multi-probe — happy / empty / null responses feed schema extraction.
4. Pattern-based Jinja — date_range, offset, cursor, geojson, single_shot templates.
5. Known-API normalization — Open-Meteo, Frankfurter, REST Countries, USGS get URL/auth/pagination fixes without per-API hardcoding in templates.
6. Self-heal — Gemini Pro rewrites full connector.py from stderr + real API response.

• REST JSON APIs only — no GraphQL/SOAP.
• No authentication on forge endpoints (demo/hackathon).
• Arbitrary unknown APIs may fail; demo presets cover the quick-start APIs.
• Reused Fivetran connection names keep historical BigQuery rows.
• Forge runs are long (minutes) and need a backend with generous timeouts.

MIT — see LICENSE.