Text chatbots can get away with a lot. Users read at their own pace, a spinner is fine, and a two-second delay barely registers.
Voice is different. Pause for a second and a half and the candidate starts talking again, assuming you didn't hear them. Now you're both talking, the transcript is a mess and the interview feels broken. In voice AI, latency is the experience itself.
That was the main lesson from building the interview agent behind Rubricate, a platform that vets domain experts through adaptive voice interviews.
The pipeline, in one breath
Every voice agent runs the same loop:
- Listen. Voice activity detection decides when someone is speaking.
- Transcribe. Streaming speech-to-text turns audio into words while they're still being said.
- Decide when the turn is over. This is harder than it sounds.
- Think. The LLM writes the next question based on what was just said.
- Speak. Streaming text-to-speech starts playing before the full sentence exists.
Every stage adds delay, and the delays stack. The work is mostly about overlapping the stages so the user never waits for all of them in a row.
The stack at a glance
| Layer | What I used |
|---|---|
| Real-time transport | LiveKit (WebRTC) |
| Agent framework | LiveKit Agents (Python) |
| Speech-to-text | Deepgram Nova-3, streaming |
| Turn-taking | Silero VAD + a semantic turn detector |
| LLM | OpenAI GPT models |
| Text-to-speech | Cartesia Sonic / ElevenLabs |
| Memory | mem0 for context across sessions |
| Observability | Sentry |
Turn detection is the hard problem
Silence-based endpointing (call the turn over after N ms of quiet) is the obvious approach, and it fails in both directions. If it's too short, you cut off a candidate who is thinking mid-sentence ("so the way I'd approach that is… hmm…"). If it's too long, every exchange feels slow.
What worked was pairing VAD with a semantic turn detector, a small model that looks at the transcript and asks whether the sentence sounds finished. "I'd use a hash map because" is not a finished thought, even after a pause. Expert interviews are full of long, thoughtful pauses, so this mattered more for Rubricate than it would for a customer-support bot.
Adaptive beats scripted
A fixed question list is easy to build and boring to sit through. It also misses the point of an interview: the best follow-up depends on what the candidate just said.
So the agent writes its next question from the last answer. It digs where an answer was vague and moves on where it was strong. That raises a fair worry: if every interview is different, how do you score them fairly? That's the subject of the next post on evals.
Don't skip production hygiene
Voice agents fail in strange, hard-to-reproduce ways: a dropped audio track, a candidate on bad Wi-Fi, a TTS provider having a bad minute. Error tracking from the first day, plus detailed session logs, turned "the interview felt weird" into bugs I could actually fix. Silence detection alone needed several rounds of tuning before it handled real candidates.
What I took away
- Measure latency per stage, not end to end. You can't fix a total.
- Stream everything. Speech recognition, the LLM and TTS should all overlap.
- Turn-taking is a model problem, not a timer problem.
- Managed real-time infrastructure is worth it. WebRTC edge cases are not where a solo engineer should spend their time.