GHC built Aria, an AI agent that answers and makes real phone calls. It runs on about 6,000 lines of Python, and the speech stack is self-hosted: everything from transcription to speech synthesis runs locally on one GPU. The only mandatory paid services are the phone line (Telnyx) and a reasoning model. Building it, rather than renting one, means GHC knows what a voice agent actually costs and where its limits are before recommending one to a client.
What the agent does on a call
Aria answers when the office can't, whether after hours or on a busy line. On the call it qualifies the lead and books the meeting. Seven tools back those actions, connected to a Google Sheets CRM and Google Calendar:
| Tool | What it does |
|---|---|
lookup_lead | Finds the caller in the CRM |
find_open_slots | Reads open times from the calendar |
book_meeting / reschedule_meeting | Books or moves the appointment |
log_call | Writes the call summary back to the CRM |
flag_for_human | Escalates to a person |
mark_do_not_call | Honors an opt-out permanently |
Compliance is built in, with a recorded-call announcement at pickup and permanent do-not-call handling.
How a call flows
A caller dials a Telnyx number. Telnyx posts a webhook and the agent answers. Audio then streams both ways over a websocket as 8 kHz mu-law, the standard telephone codec.
PSTN > Telnyx > webhook (answer, start streaming)
|
mu-law audio <> websocket
|
inbound > VAD (Silero) > STT (faster-whisper) > brain (streams)
> sentence chunker > TTS (Kokoro) > mu-law > callerVoice-activity detection decides when the caller has finished a turn (300 ms of silence by default). The transcript goes to the brain and the reply streams back sentence by sentence into the local text-to-speech engine. The caller hears the first sentence while the rest is still being generated.
The turn-taking is where most of the naturalness work went:
- Barge-in: when the caller talks over the agent, playback flushes immediately.
- Backchannel detection: an "mm-hmm" doesn't interrupt the agent's answer.
- Coalescing: a rapid correction ("Tuesday, no, Wednesday") merges into one turn.
- A latency-triggered micro-acknowledgement covers the occasional slow model response.
Two architectures, one flag
The pipeline above is not the first design; the repository's phase documents record a longer trial-and-error process in which each architecture was measured on real calls and revised or parked. What survived is two pipelines running the same persona and tools, selected by one environment variable. The local cascade above is the default, and its brain is itself switchable between Claude over the network and a local Qwen3-4B running on the same GPU. The second backend replaces the whole cascade with the OpenAI Realtime API, a fused speech-to-speech model. Both emit identical trace events, so every measurement below compares them on equal terms.
Latency, measured from the call audio
Voice agents live or die on response gap, so Aria treats latency as a measured property rather than a claim. An in-app trace records per-stage timing for every turn. Independently of the app, a ground-truth oracle pulls the dual-channel call recording from Telnyx and runs voice-activity detection on each side of the conversation. It then measures the real gap between the caller stopping and the agent's audio starting.
| Backend | Perceived median gap |
|---|---|
| OpenAI Realtime (fused) | ~1.25 s |
| Local cascade (local model) | ~2.0 s |
The fused model wins by about 750 ms. The local model produces its first token fast, around 450 ms, but the cascade's extra stages of chunking and synthesis erase that lead. The honest conclusion from the build: the self-hosted cascade's case is cost and control, not speed.
What this becomes for senior care
GHC's AI receptionist for senior care facilities is this system pointed at a different front desk. The facility problem from our discovery interviews is a phone that rings with no one free to sit at it. The same pipeline answers that call, and the tools swap from sales to intake: the agent books the appointment from the first call and hands anything uncertain to a person. The measurements above are what let GHC scope that honestly, knowing before deployment how fast the agent responds and what it costs to run.
