Building a real-time voice AI copilot
CuePilot suggests responses mid-call in under 200 ms. The architecture behind that number, the hard parts, and what we would do differently.
CuePilot listens to a live support call, turns it into text, and puts the best response on the agent's screen in under 200 milliseconds.
That number is the whole product. A suggestion that arrives after the agent has already stumbled through an answer is worse than no suggestion, because now it is a distraction.
So the hard part was never the AI. It was removing delay from a chain of steps where every single step wanted to be slow.
What is the chain you are racing against?
Every real-time voice product fights the same four steps.
Each stage adds delay, and the obvious way to build each stage is far too slow for a live conversation.
The target is not "fast on average". It is fast enough, every time, that the agent trusts it mid-sentence. A system that is quick most of the time and occasionally slow gets ignored, because people cannot rely on it.
Why does streaming change everything?
Because the alternative is waiting, and waiting happens four times.
The single most important decision was to never let one step finish before the next begins. Audio moves from the browser to the backend while the person is still talking. Transcription runs on that stream as it arrives, so the reasoning step sees text building up rather than receiving a finished block at the end of a sentence. The suggestion is pushed back over the same open connection the moment it exists.
The moment you catch yourself saying "and then we send the whole thing to the next step", you have added a delay you will spend weeks trying to remove later.
We chose FastAPI for the backend because its async, connection-native design fits this shape rather than fighting it. Transcription runs on a separate worker with its own hardware, so it never becomes the thing everything else waits for.
Where did the milliseconds actually come from?
A series of small, unremarkable wins. There was no single clever trick.
Speed over size. For the reasoning step we chose based on how fast a model responds, not how capable it is on paper. Inside a 200 millisecond budget, that difference is not a nice-to-have. It is the entire margin.
Connections stay open. Opening a new connection for each request quietly destroys real-time performance. Ours are pooled and reused.
Shared per session, not per person. Sharing connections across a session cut overhead and kept things steady when many agents were live at the same time.
Nothing computed twice. Context, setup and warm models are prepared once and reused. Every repeated calculation is delay you are paying for again and again.
None of these are interesting on their own. Together they are the difference between a demo and something people rely on during a real conversation with a real customer.
What was the hardest part?
Staying fast when many people use it at once.
Making one call fast is a weekend project. Keeping it fast while dozens of agents are on live calls is the actual work, and it is where real-time systems quietly fall apart. A pipeline that feels instant for one user can collapse into lag for everyone the moment real load arrives.
Most of the hardening went into two things. First, making the system slow down gently under pressure rather than falling off a cliff. Second, making failures visible, because a real-time system you cannot observe is a real-time system you cannot fix.
When something goes wrong during a live call, nobody has time to reproduce it. The logs have to have already answered the question.
What would we do differently?
Build the test set earlier.
When a system generates suggestions live, it is very tempting to judge quality by watching it work. That is exactly the trap we warn clients about, and we were slower to escape it than we should have been.
A prepared set of real call snippets with known-good responses lets you change models and prompts with confidence instead of caution. We got there eventually. Getting there sooner would have been cheaper, and would have made every model decision faster to make.
What transfers to other products?
Almost all of it, because very little of this was specific to voice.
Stream instead of batching. Reuse instead of reopening. Measure instead of watching. Pick the one number that defines whether the product is useful, and be ruthless about protecting it.
That thinking applies the same way to live translation, a voice assistant, a collaborative editor, or anything where a late answer is the same as a wrong one.
You can read the full CuePilot case study for more on the product itself, and we wrote separately about the guardrails any AI feature needs before it meets real users.
Frequently asked questions
How do you build a real-time voice AI copilot?
Treat it as a systems problem rather than an AI one. Audio streams to an async backend as it is spoken, transcription runs on the stream as it arrives on its own worker, a fast model picks the response, and the suggestion goes back over the same open connection. The rule throughout is that no step waits for the previous one to finish.
How is a response under 200 milliseconds possible with a model involved?
Through many small wins rather than one trick: streaming rather than batching, choosing models for response speed rather than size, keeping connections open instead of reopening them, sharing them per session, and never computing the same thing twice. Each saves a little, and the budget is made of little.
What is the hardest part of a real-time AI system?
Staying fast when many people use it at once. A pipeline that feels instant for one user can collapse under real load. Most of the engineering goes into slowing down gently rather than failing suddenly, and into making problems visible while they are happening.
Does this only apply to call copilots?
No. Stream instead of batch, reuse instead of reopen, and measure instead of watch apply to live translation, voice assistants, collaborative tools, and anything where a late answer is as bad as a wrong one. The specific technology changes, the shape of the problem does not.
How do you test something that runs live?
With a fixed set of real recordings and known-good responses that you run after every change. Watching it work is persuasive and tells you very little, because you cannot compare today's impression to last week's. Build that set earlier than feels necessary.
What should we decide before building something like this?
The one number that decides whether the product is useful, and what happens when you miss it. For CuePilot that number was the delay before a suggestion appears. Once that is agreed, most architecture decisions answer themselves, because you have a clear test for every option.
The short version
Real-time AI is mostly not about AI. It is about refusing to wait, four times over.
Stream every stage, hold connections open, compute nothing twice, and pick models for speed when speed is the product. Then spend most of your effort on staying fast under load, and build the test set before you think you need it.