How a voice agent knows you have finished

A voice agent answers in the gap you leave when you stop talking. In conversation that gap is short: across ten languages the typical pause between turns is about 200 milliseconds (Stivers et al., PNAS, 2009). Everything the agent does has to fit around it, and the first job is knowing the gap has started.

That job has a name, turn detection, and it is built from two smaller pieces.

Voice activity detection

Voice activity detection (VAD) looks at audio in short slices, usually 10 to 30 milliseconds, and labels each one speech or not speech. The simplest version compares loudness to a running estimate of the room's noise floor. Better versions use a small neural model, which copes with noise, music and keyboard clicks that fool a loudness threshold.

Either way, the output is a stream of labels. It is not a decision that someone has finished.

Endpointing

Endpointing turns that stream into turns. After speech, wait for some number of milliseconds of not-speech, then declare the turn over. That number is the wait setting, and it is the most consequential knob in a voice agent.

• Too short and the agent cuts people off. People pause mid-sentence to think, often for several hundred milliseconds, and a short wait mistakes that for the end.
• Too long and every reply feels slow, because the wait is added directly to the response time before anything else has even started.

A cut-off is worse than a slow answer. The agent talks over the person, and the speech recogniser gets half a sentence.

Try it

The instrument at the top of the home page runs this logic in your browser. Press Speak and say a sentence. It decides you have finished the way the sections above describe: a frame counts as speech when it is well above a learned noise floor and has a clear pitch, or is much louder than the room, and the turn ends once the silence has lasted 900 ms. The sound stays in your tab.

Then it shows the turn as a stack of spectra, one line per slice of time, with the silences it found drawn in orange and one line marking where an agent that waits 500 ms would have started talking. If you paused for longer than that mid-sentence, "Hear it" plays your own recording with an agent cutting in over the top.

The core of it is small:

// Simplified from the instrument: one update per 10 ms frame. db is the frame's loudness, clarity how clearly it has a pitch.
function update(state: State, t: number, db: number, clarity: number) {
	// The floor follows quiet frames down fast and loud frames up slowly, and stays put during speech,
	// so speech never becomes the floor.
	state.floor += (db - state.floor) * (db < state.floor ? 0.2 : state.speaking ? 0 : 0.002)
 
	const margin = db - state.floor
	const start = margin > 10 && (clarity >= 0.55 || margin > 24) // a voice, or something much louder than the room
	const hold = margin > 6 // a lower bar to stay in speech, so one quiet frame does not end it
	state.speaking = state.speaking ? hold : start
 
	if (state.speaking) {
		state.lastSpeech = t
		return
	}
 
	// Silence after speech: the turn ends once it has lasted long enough.
	if (state.heard && t - state.lastSpeech >= state.waitMs) state.done = true
}

The real version also ignores bursts shorter than 90 ms and spends its first 400 ms learning the room. It is a loudness and pitch gate, not a trained model, so a noisy room can fool it; a neural VAD would hold up better.

Where an energy detector goes wrong

A loudness threshold has no idea what it is hearing. A second voice, a fan that switches on, or a laugh all look like speech. The worst case is the agent's own voice coming back through the speakers: without echo cancellation the agent hears itself, decides someone is speaking and interrupts itself. Real systems put echo cancellation and noise suppression in front of the detector, and often a model-based VAD behind them.

What production systems add

Silence is a weak signal on its own. Two things usually sit on top:

• The words so far. "I would like a table for" deserves a longer wait than "yes". Some systems read the partial transcript, or use a small model trained to predict end of turn, and stretch or shrink the wait accordingly.
• Barge-in. When the person starts talking over the agent, the agent should stop within a few hundred milliseconds. That needs fast detection and clean echo handling, because the detector has to separate the person from the agent's own audio.

The wait is one line in a budget

A reply has to cover several steps after the person stops speaking:

1. the endpointing wait,
2. finalising the speech recognition result,
3. the language model's time to first token,
4. the speech synthesiser's time to first audio, and
5. network and playback.

Streaming overlaps some of these, but the endpointing wait sits at the front of every reply and nothing downstream can hide it. It is also the one part you choose rather than optimise, which is why it is worth getting right, and why the home page lets you hear what a wrong one does.