A Self-Auditing Agentic RAG System for Customer Support Automation

SupportFlow AI doesn't just answer customer questions with RAG — it audits its own answer before sending it, scores confidence and business risk, detects contradictions in the knowledge base, and routes every request to exactly one of four actions: auto-send, save as draft, ask a follow-up question, or escalate to a human with a complete handoff packet. Human corrections are stored and reused as style examples for future replies (no fine-tuning needed).

Customer question
  → Query Understanding Agent      (intent, entities, missing info)
  → RAG Retrieval Engine           (Gemini embeddings + FAISS)
  → Policy Conflict Detector       (do retrieved docs contradict each other?)
  → Risk & Emotion Detection Agent (emotion + business-risk matrix)
  → Response Generation Agent      (drafts a reply, grounded in retrieved policy
                                     + past human corrections)
  → Self-Audit Agent               (independent QA pass on the draft)
  → Decision Engine                (auto_send | draft | follow_up | escalate)
  → Human Handoff Agent            (if draft/escalate: builds a full packet +
                                     Handoff Quality Score)
  → Learning Feedback Module       (stores human corrections + embeds them for
                                     future few-shot retrieval)

supportflow-ai/
├── backend/
│   ├── main.py                  FastAPI app (all REST endpoints)
│   ├── pipeline.py              Orchestrates the full agent pipeline
│   ├── config.py                Env-driven settings
│   ├── database.py              SQLite persistence (no ORM)
│   ├── schemas.py                Pydantic request/response models
│   ├── seed_kb.py                CLI script to (re)build the FAISS index
│   ├── core/
│   │   ├── embeddings.py        Gemini embedding wrapper
│   │   ├── vector_store.py      FAISS wrapper (persistent, cosine via IP)
│   │   ├── llm_client.py        Groq chat/JSON wrapper
│   │   ├── document_loader.py   .txt/.md/.pdf loading + chunking
│   │   └── email_agent.py       Sandbox/live email sending
│   └── agents/
│       ├── knowledge_base.py
│       ├── query_understanding.py
│       ├── retrieval.py
│       ├── policy_conflict.py
│       ├── risk_emotion.py
│       ├── response_generation.py
│       ├── self_audit.py
│       ├── decision_engine.py
│       ├── human_handoff.py
│       └── learning_feedback.py
├── frontend/
│   └── dashboard.py              Streamlit manager dashboard (6 tabs)
├── data/policies/                 8 sample fake policy docs (incl. an
│                                   intentional FAQ-vs-Refund-Policy conflict)
├── tests/
│   ├── smoke_test.py             Offline test, mocked LLM+embeddings, $0 cost
│   └── test_cases.py             Live test runner against the real APIs
├── vector_index/                  FAISS index files (created at runtime)
├── storage/                       SQLite DB (created at runtime)
├── requirements.txt
├── .env.example
├── run_backend.sh / .bat
├── run_dashboard.sh / .bat
└── seed_kb.sh / .bat

python -m venv .venv
source .venv/bin/activate        # Windows: .venv\Scripts\activate
pip install -r requirements.txt

• Groq (LLM): https://console.groq.com/keys
• Google Gemini (embeddings): https://aistudio.google.com/apikey

cp .env.example .env
# then edit .env and paste in GROQ_API_KEY and GOOGLE_API_KEY

This proves the whole pipeline (decision logic, FAISS, SQLite, handoff packets, learning loop) is wired correctly, using mocked LLM/embedding calls:

python -m tests.smoke_test

You should see all four decision actions (auto_send, follow_up, escalate from risk, and escalate from a detected KB conflict) appear across the sample queries.

Once your real API keys are in .env:

./seed_kb.sh         # Windows: seed_kb.bat

This embeds the 8 sample policy documents in data/policies/ with Gemini and stores them in the FAISS index at vector_index/.

./run_backend.sh      # Windows: run_backend.bat

FastAPI will be live at http://localhost:8000 (docs at /docs).

In a second terminal:

./run_dashboard.sh    # Windows: run_dashboard.bat

Streamlit opens at http://localhost:8501 with 6 tabs: Knowledge Base, Test Console, Handoff Queue, Conversation History, Learning Log, Email Log.

python -m tests.test_cases

Runs the exact sample queries from the spec (Where is my order?, Can I return shoes I wore once?, I was charged twice., etc.) against your live backend and prints the intent/emotion/risk/confidence/action for each.

The Decision Engine (backend/agents/decision_engine.py) combines four signals into exactly one action:

1. Self-Audit confidence (0-100) — is the draft reply fully supported by retrieved policy, complete, and on-tone?
2. Business risk (low/medium/high/critical) — from the deterministic emotion × intent risk matrix in risk_emotion.py (e.g. payment disputes and legal threats are always escalated, regardless of how calm the customer sounds).
3. Policy conflict — if retrieved chunks from different source documents contradict each other, the system never guesses; it always escalates so an admin can fix the knowledge base.
4. Missing information — if the request can't be answered correctly without more details (e.g. an order ID), the system asks a targeted follow-up question instead of guessing or escalating everything.

Priority order: conflict → critical risk → missing info → high confidence/low risk (auto-send) → high risk (escalate) → medium confidence (draft) → everything else (escalate).

backend/agents/self_audit.py makes a second, independent Groq call that never saw the generation prompt — it re-reads the draft reply against the retrieved policy text like a compliance reviewer, flagging:

• claims not literally supported by the retrieved context
• incomplete answers
• inappropriate tone
• a confidence score and risk level used directly by the Decision Engine

This is what prevents the system from confidently hallucinating a refund window or shipping promise that isn't actually in your policy documents.

Whenever a human edits/approves a handoff in the dashboard with a different final reply than the AI's suggestion, learning_feedback.py:

1. Stores (query, original_ai_reply, corrected_reply, reason, policy) in SQLite (visible in the Learning Log tab).
2. Embeds the customer query with Gemini and stores it in a second small FAISS index.
3. On future queries, the most similar past corrections are retrieved and passed to the Response Generation Agent as style/tone examples.

No fine-tuning required — this is retrieval-based few-shot learning from real human feedback.

The 8 sample policy documents in data/policies/ are intentionally written so the following customer messages each exercise a different part of the system:

• APP_MODE=sandbox (default) — the Email Automation Agent only logs what it would send; nothing is actually emailed. Switch to live and configure SMTP_* in .env to actually send mail.
• SDK drift: this project pins to the groq and google-genai Python SDKs as of early 2026. If Google or Groq change their embedding/response schema in a future SDK release, check backend/core/embeddings.py (the _embed_batch method) and backend/core/llm_client.py and adjust the response-parsing accordingly — the rest of the system is decoupled from SDK specifics via these two thin wrapper files.
• Switching to gemini-embedding-001: set GEMINI_EMBEDDING_MODEL=gemini-embedding-001 and EMBEDDING_DIM to your desired dimension (e.g. 768 or 1536) in .env, then re-seed the KB with ./seed_kb.sh (embeddings from different models are not compatible with each other, so always re-seed after changing the embedding model or dimension).
• Adding your own policy documents: drop .txt/.md/.pdf files into data/policies/ and click "Seed sample policy documents" in the dashboard (or re-run ./seed_kb.sh), or upload them directly from the dashboard's Knowledge Base tab.