LLM integration done right: RAG and guardrails

There is a wide, expensive gap between a prompt that works in a playground and a feature real users can rely on.

The playground is forgiving. You know what to ask, you accept the occasional miss, and nothing depends on the answer. Production is none of those things.

Closing that gap comes down to four layers: retrieval, prompts, testing, and guardrails. Here is how each one works in practice.

Why does a working prompt fail in production?

Because you were the only user, and you already knew what to ask.

Real users phrase things strangely, ask about things you never documented, and occasionally try to break it on purpose. A wrong answer now has a consequence attached: a bad refund, a wrong medical instruction, a lost customer.

"It usually works" is a fine standard for a demo and not a standard you can ship. Everything below exists to raise it.

Why is retrieval the first thing to fix?

Because the model does not know your business, and most failures start here.

The most common AI feature answers questions from your own content: support documents, product data, internal knowledge. The instinct is to paste everything into the prompt. That is slow, expensive, and stops working once your content grows.

The real technique is retrieval. Store your content so it can be searched by meaning rather than exact words, fetch only the few relevant pieces for each question, and give the model just those to answer from.

Here is the part teams miss. If the search returns the wrong passages, the model answers confidently from the wrong context, and you get a fluent, plausible, incorrect answer.

Most complaints about a model "making things up" are really complaints about retrieval fetching the wrong documents. Before you change models or rewrite prompts, look at what the search actually returned.

What makes a production prompt good?

Prompt writing has a mystique it does not deserve. A good one is clear instructions, relevant context, and explicit limits.

Four things matter far more than clever phrasing:

A specific role and task. "Answer only from the provided documents, as a support assistant for this product" beats a vague, chatty personality every time.

What to do when unsure. Tell the model directly to say it does not know rather than guess. This single instruction prevents a large share of confident wrong answers.

The exact output shape. If you need structured data, ask for it precisely and check what comes back before using it.

The boundaries. What it must never do, never claim, and never reveal, regardless of what a user asks.

Keep prompts in version control and treat a change to one as a change to code, because that is exactly what it is. A small edit shifts behaviour for every user at once, and if it is not tracked you cannot explain a problem later.

How do you know a change made it better?

You measure it. There is no shortcut here, and this is the layer most teams skip.

Build a set of real inputs paired with known-good answers. Include the awkward ones: ambiguous questions, hostile users, questions your documents genuinely do not cover. Then run the whole set every time you change a prompt, swap a model, or adjust retrieval.

This is unglamorous and it is the highest-value work in the entire project. It is also what lets you move to a newer, cheaper or faster model later with confidence rather than dread.

A small test set you actually run beats a large one you keep meaning to build.

What guardrails does a live feature need?

The defining question is not how good the answer is when it is right. It is what happens when it is wrong, because sometimes it will be.

Check structured output before anything acts on it. Raw model output should never drive a real action unchecked.

Limit what it can touch. If the feature can take actions, give it the fewest possible, and require confirmation for anything expensive or irreversible.

Filter in both directions. Watch for private data leaking out, for the conversation drifting into territory you are liable for, and for text that tries to give the model new instructions.

Keep a person involved wherever a wrong answer is expensive. A feature that drafts something for a human to approve has a completely different risk profile from one that acts alone.

What about cost and speed?

Both are design decisions, and both surprise teams who leave them until after launch.

Route by difficulty. The easy majority of requests can go to a small fast model, with a larger one kept for genuinely hard cases. Cache repeated questions, because there are far more of them than anyone expects.

For speed, stream the answer so the user sees it forming instead of watching a spinner. Do independent work at the same time rather than one step after another. And decide honestly which parts need an instant answer, because plenty of useful work can happen in the background.

The thread running through all of it

Every layer here is the same idea applied in a different place. A language model is capable and unreliable, so you build a system around it that makes the capability usable and the unreliability safe.

The model is the easy part now. The engineering around it is the product, and it is the bulk of what our AI work actually involves. If you are still deciding what to build, we wrote about choosing between an agent, retrieval and a single call.

Frequently asked questions

Why does an AI feature that worked in testing fail with real users?

Because you were the only tester and you already knew what to ask. Real users phrase things unexpectedly, ask about undocumented things, and sometimes try to break it. Closing the gap needs retrieval that grounds answers in your facts, prompts with clear limits, a test set you run on every change, and guardrails for when the answer is wrong.

What is retrieval and why does it matter so much?

Retrieval stores your content so it can be searched by meaning, fetches the few relevant pieces for each question, and gives the model only those to answer from. It matters because most wrong answers are caused by the search returning the wrong documents, not by the model itself. Fix retrieval before you change anything else.

How do you stop a model from making things up?

Ground it with good retrieval, instruct it explicitly to say it does not know rather than guess, check structured output before acting on it, and keep a person involved wherever a wrong answer is expensive. Then measure all of it against a fixed set of real examples so you know whether it is improving.

How do you test something that does not give identical answers?

You test for correctness and safety, not for identical wording. Build a set of real inputs with known-good answers, including ambiguous and hostile ones, and run the whole set after every prompt, model or retrieval change. Add each production failure to the set as you find it.

How do you keep the cost under control?

Route by difficulty so easy requests go to a small cheap model and only hard ones reach a large one, and cache repeated questions aggressively. Both are decided when you design the feature. Discovering them in the first invoice is the expensive version of the same lesson.

Do we need a human reviewing answers?

It depends entirely on what a wrong answer costs. If it is a suggestion someone can ignore, probably not. If it moves money, changes a record, or gives advice people act on, then yes, at least until your test results give you a reason to trust it alone.

The short version

Fix retrieval first, because that is where most wrong answers come from. Write prompts as instructions with clear limits, and version them like code.

Build a test set and run it on every change, so improvements are measured rather than hoped for. Then add the guardrails that make being wrong survivable, and decide cost and speed on purpose.