AICustomer SupportUS
CuePilot
Live call transcription and response suggestions, delivered while the customer is still talking.
- Our role
- Backend · AI pipeline · Frontend · Deployment
- Platforms
- Web
- Python
- FastAPI
- Whisper
- WebSockets
- React
- <200mssuggestion latencyAudio in to suggestion on screen
- Real-timestreaming pipelineWebSockets, not request/response
- Whisperon GPU workersIncremental transcription as the caller speaks
- Multi-agentteam modeParallel sessions across a support floor
At a glance
- Business problem
- Support agents lose time mid-call deciding how to answer a difficult customer. CuePilot had to put a usable suggested response in front of the agent fast enough to be read while the customer is still speaking.
- Technical challenges
- CuePilot chains audio capture, speech-to-text and an LLM inside a budget of roughly 200 milliseconds end to end, while holding open WebSocket connections for every agent on a support floor at once.
- Engineering solution
- CuePilot streams browser audio over WebSockets to a FastAPI service, runs Whisper incrementally on GPU workers so transcription keeps pace with speech, and pushes each LLM suggestion to the agent UI as soon as it is ready.
- Core features
- Live transcription of an in-progress call. AI-suggested responses, updated as the call moves. One-click copy or adapt from the suggestion panel. Session history and per-call analytics. Multi-agent team mode with parallel sessions.
- Architecture
- BrowserWebSocketsFastAPIWhisperLLMReact Dashboard
- Technical highlights
- Connections are multiplexed per session rather than per agent. Whisper runs on GPU workers with pooled connections, so no request pays model load time. Transcription is incremental, so the LLM starts from partial text instead of waiting for silence. The suggestion panel updates in place rather than re-rendering the transcript.
- Business value
- CuePilot lets a support agent concentrate on the customer instead of on composing a reply. The suggestion is already on screen by the time the agent would have started thinking about one.
The problem
A support agent on a hard call is doing two jobs at once. Listening to an upset customer, and composing a reply that does not make it worse. The second job is where the pause comes from, and the pause is what the customer hears.
CuePilot's brief was to remove the second job. Put a good suggested reply in front of the agent while the customer is still talking, so the agent only has to read and adapt.
That turns a support problem into a latency problem.
The constraints
The budget is a conversational pause, not a page load. People leave roughly a beat of silence between turns. If the suggestion lands after the agent has already started speaking, it is worse than nothing — now they are reading and talking at the same time.
Three stages, all serial. Audio has to become text before text can become a suggestion. Nothing in the chain parallelises away.
A support floor, not one agent. Every agent holds an open connection for the length of every call. The system has to stay responsive when that is fifty people, not one.
Everything below follows from one number: about 200 milliseconds, end to end, for all three stages.
How we built it
The pipeline streams rather than requests. Audio leaves the browser over a WebSocket in small chunks, Whisper transcribes incrementally on GPU workers as the chunks arrive, and the LLM starts from partial text instead of waiting for the caller to stop.
Two decisions do most of the work here. Whisper runs on GPU workers with pooled connections, so no individual request ever pays model load time — that cost is paid once, at startup, not per call. And WebSocket connections are multiplexed per session rather than per agent, which is what keeps a support floor from becoming a floor's worth of idle sockets.
The React dashboard renders the suggestion panel in place. The transcript keeps scrolling underneath it, untouched, so a new suggestion never moves the text the agent is reading.
Pipeline in motion
Watch what the agent sees. The transcript builds word by word on the left as the customer speaks. Partway through, before the customer has finished, the suggestion panel on the right fills in.
- Incremental transcriptWords appear as they are transcribed, not after the caller stops. This is what lets the LLM start early.
- Live waveformConfirms audio is still streaming. Silence here means a dropped socket, and the agent needs to know instantly.
- Suggested responseUpdates in place. The transcript underneath never reflows, so the agent never loses their reading position.
- Copy or adaptOne click to send as written, or edit first. Nothing is ever sent without the agent choosing it.
That gap — between the customer still talking and the reply already being on screen — is the entire product. Everything in the architecture exists to make it happen inside one conversational beat.
The agent then does one of three things: send it as written, adapt it, or ignore it. CuePilot never speaks to the customer. It writes into the agent's screen, not into the call, and the agent stays responsible for every word actually said. That boundary was set at the start and it is why the tool is usable in regulated support environments at all.
What shipped
- Live transcription of an in-progress call, updating as the customer speaks
- AI-suggested responses that refresh as the conversation moves
- One-click copy, or edit-before-send, from the suggestion panel
- Session history and per-call analytics
- Multi-agent team mode, with parallel sessions across a floor
- A React dashboard for reviewing past calls
The decisions that mattered
Streaming instead of request/response. The obvious build records a turn, posts it, waits, renders. That is three round trips of dead time before the model even starts. Streaming over WebSockets removes the waiting from the critical path, and it is the only reason the 200ms figure is reachable at all.
Inference speed over benchmark quality. Total latency is the sum of three stages, so a saving anywhere is a saving the agent feels. For the LLM step we picked on raw inference speed rather than on leaderboard scores — a marginally better answer that arrives after the agent has started talking scores zero in practice.
Per-session multiplexing over per-agent connections. Simpler to give each agent a socket. It also means the connection count grows with headcount rather than with concurrent calls, and idle sockets are the thing that falls over first at scale.
GPU workers over a managed transcription API. More to operate. But a hosted API adds a network hop and a queue we do not control, inside a budget where a single extra hop is a meaningful fraction of the total.
If you are weighing something similar, building a real-time voice AI copilot goes deeper on the latency work, and LLM integration, RAG, prompts and guardrails covers the model layer. Our AI automation service page has the shape of a typical engagement.
In the client’s words
Excellent communication, clean code, and timely delivery. It was refreshing to work with developers who genuinely cared about the project's success.
Questions buyers ask
What does sub-200ms actually mean for a support call?
It means the suggestion is on screen inside the natural pause after the customer stops talking. Above roughly half a second the agent has already started answering, and the suggestion arrives too late to be useful.
Does an AI assist tool replace support agents?
No — CuePilot writes nothing to the customer. It puts a suggestion in front of the agent, who reads, adapts or ignores it. The agent stays in control of every word that is actually said.
What drives the running cost of a real-time AI pipeline?
GPU time for transcription and per-token LLM cost, both scaling with call minutes rather than with user count. It is worth modelling against your real call volume before building, because it changes which models are viable.
Can this work with our existing support platform?
Usually yes. The pipeline takes an audio stream and returns suggestions, so the integration question is how your platform exposes call audio. That is the first thing we would look at with you.