Inspiration

Home-health agencies spend roughly 27% of their operating budget on non-clinical administrative labor, most of it manual phone coordination. The worst moment is the 6:00 AM call-out: when a caregiver phones in sick, a human scheduler loses about 45 minutes dialing backups one by one while billable shifts go unstaffed and patients wait. The same office also drowns in patient intake matching, end-of-day medication paperwork, and a language gap that forces them to pay outside medical interpreters by the minute. We wanted to know if a single voice AI could absorb that entire category of work, so a human never has to run the phone tree again.

What it does

The Healthcare Omni-Agent answers every inbound call for a home-health agency and handles the four jobs that eat scheduler time:

  • Shift-Backfill Cascade. A call-out triggers a Neo4j query for compliant, nearby backups, then fires an SMS and voice cascade. The first caregiver to say "YES" locks the shift live, turning a 45-minute scramble into about 5 minutes of oversight.
  • Intake and Graph Matching. New patient intakes are captured by voice and matched against the caregiver roster using graph-relational intelligence (certifications, languages, proximity), so the agency captures revenue faster and shrinks its waitlist.
  • Med-Log and Compliance Sync. An end-of-shift verbal questionnaire is turned into structured JSON and appended to an audit-ready log, removing manual compliance paperwork.
  • Interpreter Bridge and Escalation. A distress phrase hot-dials a human and opens two-way live interpretation, displacing external translation services that bill by the minute.

A Main Orchestrator Agent recognizes the caller by phone number, greets them by name, detects intent, and routes into the right workflow. If the caller switches topics mid-call, it closes one workflow and opens another without ever dropping the line.

How we built it

The system runs on a hub-and-spoke design: one Orchestrator (the hub) routes into four SubWorkflows (the spokes), with escalation as an always-on override.

The heart of it is a Dual-Agent core. An Active Agent owns the real-time audio stream, using ElevenLabs streaming TTS on a deliberately tiny context window so replies start fast. A Passive Agent listens to the WebSocket token stream in the background, writes to the database, and holds the global conversation state. On barge-in, Twilio halts playback, the Active Agent drops its in-flight generation and pulls the latest state snapshot from the Passive Agent, then resumes on the caller's new direction with prior context intact.

  • Twilio is the telephony backbone: Programmable Voice, SMS, Video, and Media Streams over WebSocket for inbound answering, the outbound cascade, and barge-in playback control.
  • ElevenLabs powers low-latency, natural voice for the Active Agent and the cascade calls.
  • Neo4j stores the graph-relational roster and serves the ranked-candidate and compliant-backup queries via Cypher.

We built the spine first (Twilio answers, ElevenLabs speaks), then the Dual-Agent core and barge-in, then the Orchestrator, then hung the SubWorkflows off it as thin vertical slices.

Challenges we ran into

  • Keeping voice latency believable. We had to stream STT and TTS and keep the Active Agent's context tiny, pushing every database write onto the Passive Agent.
  • Not losing state on barge-in. The interrupt path was the riskiest part of the build, so we snapshotted state in the Passive Agent and tested interrupt handling early.
  • Cascade race conditions. Multiple backups can reply at once, so we used a first-YES-wins lock and tested it under concurrent replies.
  • Graph matching in a time box. We seeded a small synthetic graph and kept the Cypher matching tractable rather than modeling the full domain.
  • Working with health data responsibly. We used only synthetic patient data and kept audit logs append-only.

What we learned

Splitting a live conversation into a fast responder and a background bookkeeper is what makes voice AI feel both quick and reliable. Once state lives outside the thing generating speech, interruptions and mid-call intent switches stop being scary. We also learned how much of an agency's cost is pure coordination, and how directly a single always-available agent maps onto real dollars saved per call.

What's next

  • Finish the Intake and Escalation workflows end to end.
  • Add the Twilio Video validation channel for field hazard checks.
  • Harden for real deployment: HIPAA-aware handling, richer roster data, and a live staff portal for the post-call summaries.

Built With

Share this project:

Updates