Building a real-time voice AI copilot: lessons from CuePilot

CuePilot listens to a live customer-support call, transcribes it, and surfaces the best response for the agent to say — in under 200 milliseconds. That number is the whole product. A suggestion that arrives after the agent has already stumbled through an answer is worse than no suggestion at all, because it is a distraction. So the engineering was not really about AI. It was about shaving milliseconds out of a chain of steps that each wanted to be slow. Here is how it came together, and what we would tell anyone attempting something similar.

The chain you are fighting

Every real-time voice AI product is a race against the same pipeline: capture audio, turn it into text, decide what to say, and get that back to the human. Each stage adds latency, and the naive version of each stage is far too slow for a live conversation.

• Capture — pulling audio off the agent's call without adding lag.
• Transcription — speech to text, incrementally, not in one big batch at the end.
• Reasoning — deciding the optimal response from the transcript and context.
• Delivery — getting the suggestion onto the agent's screen instantly.

1. 1Capture audiostreamed from the agent's browser tab
2. 2TranscribeWhisper, incrementally — not one batch at the end
3. 3Reasonthe LLM picks the optimal response
4. 4Deliveron the agent's screen in under 200 ms

Get any one of these wrong and the whole thing feels sluggish. The target is not "fast on average." It is fast enough, consistently, that the agent trusts it mid-sentence.

Why we streamed everything

The single most important decision was to never wait for a step to fully finish before starting the next. Audio streams from the agent's browser over a WebSocket to the backend as it is spoken. Transcription runs incrementally on that stream, so the reasoning layer sees text building up in near real time rather than receiving one block at the end of a sentence. Suggestions are pushed back over the same socket the moment they are ready.

Batch thinking is the enemy of real-time. The moment you catch yourself saying "and then we send the whole thing to the next step," you have added a delay you will spend weeks trying to remove later.

We chose FastAPI for the backend specifically because its async, WebSocket-native design fits this streaming shape rather than fighting it. The transcription runs on a GPU-backed worker so it does not become the bottleneck under load.

Where the 200 milliseconds actually went

Hitting the latency target was a series of unglamorous wins, not one clever trick:

1. Inference speed over model size. For the reasoning layer we prioritised raw inference latency, because at a 200-millisecond budget the difference between a fast and a slow model is not a nicety — it is the entire margin.
2. Connections held open, not reopened. Re-establishing connections per request quietly destroys real-time performance. We pooled and reused them.
3. Multiplexing per session, not per agent. Sharing connections across a session reduced overhead and kept things stable when many agents were live at once.
4. Doing nothing twice. Anything that could be computed once and reused — context, setup, warm models — was.

None of these are exciting individually. Together they are the difference between a demo and a tool people rely on during real calls.

The hard part was stability, not speed

Getting one call fast is a weekend project. Keeping it fast while many agents are on live calls at once is the real work. Concurrency is where real-time systems quietly fall apart: a pipeline that is snappy for one user can collapse into lag for everyone when load arrives. Most of the hardening went into making sure the system degraded gracefully rather than falling off a cliff — and into making the failure modes visible, because a real-time system you cannot observe is a real-time system you cannot fix.

What we would do differently

Honestly, we would invest even earlier in evaluation. When suggestions are generated live, it is tempting to judge quality by watching it work, which is exactly the trap we warn clients about. A curated set of real call snippets with known-good responses would have let us change models and prompts with confidence instead of caution. We got there — but earlier would have been cheaper.

The lesson that transfers

Real-time AI is a systems problem wearing an AI costume. The model matters, but the wins come from streaming instead of batching, reusing instead of reopening, measuring instead of guessing, and being ruthless about the one number that defines the product. That thinking is the same whether you are building a call copilot, a live translation feature, or a voice assistant.

If you are building something where latency is the product, this is exactly the kind of work we take on. See the full CuePilot case study.

Frequently asked questions

How do you build a real-time voice AI copilot?

By treating it as a systems problem, not an AI problem. Audio streams over a WebSocket to an async backend (we use FastAPI), transcription runs incrementally on a GPU-backed worker, an inference-fast LLM picks the response, and the suggestion is pushed back over the same socket — all without ever waiting for one step to fully finish before starting the next.

How is sub-200ms latency possible with an LLM in the loop?

It comes from many unglamorous wins, not one trick: streaming instead of batching, prioritising inference speed over model size, holding connections open instead of reopening them, multiplexing per session rather than per agent, and never computing the same thing twice.

What's the hardest part of a real-time AI system?

Stability under concurrency, not raw speed. A pipeline that's snappy for one user can collapse into lag for everyone when load arrives. Most of the work goes into making the system degrade gracefully and making its failure modes observable.

Does this only apply to call copilots?

No. The same thinking — stream instead of batch, reuse instead of reopen, measure instead of guess — applies to live translation, voice assistants, and any feature where latency is the product.