Inspiration
Home-health agencies spend roughly 27% of their operating budget on non-clinical administrative labor, most of it manual phone coordination. The worst moment is the 6:00 AM call-out: when a caregiver phones in sick, a human scheduler loses about 45 minutes dialing backups one by one while billable shifts go unstaffed and patients wait. The same office also drowns in patient intake matching, end-of-day medication paperwork, and a language gap that forces them to pay outside medical interpreters by the minute. We wanted to know if a single voice AI could absorb that entire category of work, so a human never has to run the phone tree again.
What it does
The Healthcare Omni-Agent answers every inbound call for a home-health agency and handles the four jobs that eat scheduler time:
- Shift-Backfill Cascade. A call-out triggers a Neo4j query for compliant, nearby backups, then fires an SMS and voice cascade. The first caregiver to say "YES" locks the shift live, turning a 45-minute scramble into about 5 minutes of oversight.
- Intake and Graph Matching. New patient intakes are captured by voice and matched against the caregiver roster using graph-relational intelligence (certifications, languages, proximity), so the agency captures revenue faster and shrinks its waitlist.
- Med-Log and Compliance Sync. An end-of-shift verbal questionnaire is turned into structured JSON and appended to an audit-ready log, removing manual compliance paperwork.
- Interpreter Bridge and Escalation. A distress phrase hot-dials a human and opens two-way live interpretation, displacing external translation services that bill by the minute.
A Main Orchestrator Agent recognizes the caller by phone number, greets them by name, detects intent, and routes into the right workflow. If the caller switches topics mid-call, it closes one workflow and opens another without ever dropping the line.
How we built it
The system runs on a hub-and-spoke design: one Orchestrator (the hub) routes into four SubWorkflows (the spokes), with escalation as an always-on override.
The heart of it is a Dual-Agent core. An Active Agent owns the real-time audio stream, using ElevenLabs streaming TTS on a deliberately tiny context window so replies start fast. A Passive Agent listens to the WebSocket token stream in the background, writes to the database, and holds the global conversation state. On barge-in, Twilio halts playback, the Active Agent drops its in-flight generation and pulls the latest state snapshot from the Passive Agent, then resumes on the caller's new direction with prior context intact.
- Twilio is the telephony backbone: Programmable Voice, SMS, Video, and Media Streams over WebSocket for inbound answering, the outbound cascade, and barge-in playback control.
- ElevenLabs powers low-latency, natural voice for the Active Agent and the cascade calls.
- Neo4j stores the graph-relational roster and serves the ranked-candidate and compliant-backup queries via Cypher.
We built the spine first (Twilio answers, ElevenLabs speaks), then the Dual-Agent core and barge-in, then the Orchestrator, then hung the SubWorkflows off it as thin vertical slices.
Challenges we ran into
- Keeping voice latency believable. We had to stream STT and TTS and keep the Active Agent's context tiny, pushing every database write onto the Passive Agent.
- Not losing state on barge-in. The interrupt path was the riskiest part of the build, so we snapshotted state in the Passive Agent and tested interrupt handling early.
- Cascade race conditions. Multiple backups can reply at once, so we used a first-YES-wins lock and tested it under concurrent replies.
- Graph matching in a time box. We seeded a small synthetic graph and kept the Cypher matching tractable rather than modeling the full domain.
- Working with health data responsibly. We used only synthetic patient data and kept audit logs append-only.
What we learned
Splitting a live conversation into a fast responder and a background bookkeeper is what makes voice AI feel both quick and reliable. Once state lives outside the thing generating speech, interruptions and mid-call intent switches stop being scary. We also learned how much of an agency's cost is pure coordination, and how directly a single always-available agent maps onto real dollars saved per call.
What's next
- Finish the Intake and Escalation workflows end to end.
- Add the Twilio Video validation channel for field hazard checks.
- Harden for real deployment: HIPAA-aware handling, richer roster data, and a live staff portal for the post-call summaries.
Built With
- cypher
- elevenlabs
- neo4j
- speech-to-text
- twilio
- twilio-media-streams
- twilio-video
- voice-ai
- websockets
Log in or sign up for Devpost to join the conversation.