[ VOICE · SPEECH · AI ]
Building voice-first AI apps
A voice app is a pipeline: speech in, understanding, response, speech out. Learn the four stages, where the latency hides, and how to run the whole loop locally.
Watch the lecture, then read Max's deep-dive below.
- The voice-app pipeline: speech-to-text → LLM → text-to-speech
- Capturing microphone input and transcribing it
- Sending the transcript to an LLM and streaming the reply
- Speaking the response back with TTS
- Wiring it all into a working assistant loop
A voice-first app treats speech as the primary interface — you talk to it, it talks back. Under the hood, every such app is a pipeline of four stages, and the craft is in making that pipeline fast enough that the conversation feels alive rather than like a walkie-talkie with lag.
The four stages
- Speech-to-Text (STT / ASR) — a speech-recognition model turns the audio waveform into words. This is where accents, background noise, and domain jargon are won or lost.
- Understanding — the transcript goes to a language model that interprets intent and decides what to do or say. In an agent, this stage may also call tools.
- Response generation — the model produces the words to speak back.
- Text-to-Speech (TTS) — a synthesis engine (such as Piper for local use, or a cloud voice) turns that text into natural audio.
Speak → transcribe → understand → reply → synthesize → speak. That round trip is the entire game.
Latency is the whole product
In text you tolerate a pause; in conversation you do not. Humans expect a reply within a few hundred milliseconds, and dead air past roughly a second feels broken. Two techniques fight this:
- Streaming, not batching. Do not wait for the user to finish, transcribe everything, think, then speak. Transcribe incrementally, start the LLM as soon as intent is clear, and begin speaking the first sentence while later sentences are still being generated.
- Endpointing / VAD. Voice-activity detection decides when the user has stopped talking so you can respond promptly without cutting them off mid-thought. Get this wrong and the app either interrupts people or leaves them hanging.
You can run the whole loop locally
Every stage has a strong on-device option: Whisper (or faster variants) for STT, a local model via Ollama for understanding, and Piper for TTS. Wire them together and you get a private, offline, no-per-minute-cost voice assistant — the prompt and the audio never leave the machine. For zero-install prototypes, the browser's built-in SpeechRecognition and SpeechSynthesis Web Speech APIs let you stand up a voice loop with no dependencies at all, then swap in the heavier local models as quality demands.
Design for the ear, not the eye
Finally, a voice reply is not a screen transcript read aloud. Spoken answers should be shorter, front-load the key point, avoid lists longer than a few items, and never dump a URL or a table. Write for a listener who cannot scroll back — brevity and confirmation ("I've set that for 3 pm — anything else?") are the grammar of good voice UX.
Keep going
- Whisper (STT) — an open speech-recognition model
- Piper (TTS) — fast, local, natural-sounding neural text-to-speech