Evaluating AI Features Before They Reach Your Customers

Traditional software is deterministic: the same input produces the same output, and a test either passes or fails. AI features are different. The same question can produce different answers, a small prompt change can improve one case and break three others, and a model update from your provider can shift behaviour overnight.

That does not make AI untestable. It means testing needs a different shape. Here is the approach we use before an AI feature reaches real customers.

Define what "good" means first

"The answers should be good" is not a specification. Before measuring anything, write down what success looks like for this feature in concrete terms. For a support assistant, that might be:

• The answer is factually correct according to our documentation.
• It cites the source it used.
• It declines, and offers a human, when the documentation does not cover the question.
• It never makes commitments about refunds, pricing or legal matters.
• It stays under a set cost and response time.

Each of these becomes something you can check. Without this list, every review turns into opinions.

Build an evaluation set from reality

An evaluation set is a collection of realistic inputs with the expected outcome for each. It is the single most valuable asset in an AI project, and it should come from reality rather than imagination.

Good sources include real questions from support tickets, emails and search logs; edge cases your team knows cause trouble; questions the assistant should refuse; and adversarial inputs, such as attempts to make it ignore its instructions or reveal information it should not.

Start small and honest. A few dozen carefully chosen cases that represent real usage beat hundreds of easy ones.

Choose how each case is graded

Different checks need different graders:

• Exact checks for things with one right answer: the correct category, a valid format, the right source document.
• Rule checks for constraints: no prices mentioned, response under a length limit, a citation present.
• Model-assisted grading for open-ended answers, where another model scores the response against a written rubric. This scales well, but the grader itself must be checked against human judgement on a sample.
• Human review for a regular sample, especially for high-stakes outputs.

Mixing these gives you broad coverage without asking people to read every answer.

Measure more than accuracy

An AI feature can be accurate and still unfit to ship. Track at least:

1. Correctness: how often the answer is right.
2. Appropriate refusal: how often it declines when it should, and how often it refuses when it should not.
3. Safety: whether it follows the rules, such as no unauthorised promises and no leaked information.
4. Cost and speed: cost per request and response time at realistic volume.

These often trade off against each other. A larger model may be more accurate but too slow or expensive for the use case. Having the numbers side by side turns these into informed decisions.

Make evaluation part of every change

The evaluation set is not a one-time gate. Re-run it whenever you change a prompt, the documents behind the feature, the model or provider, or the tools the AI can use. Compare against the previous results before shipping. A change that improves the average while breaking an important case should be caught here, not by a customer.

Keep watching after launch

Real users will always find inputs you did not anticipate. Plan for it:

• Log inputs and outputs (respecting privacy requirements) so problems can be investigated.
• Give users a simple way to flag a bad answer.
• Review flagged and low-confidence cases regularly, and add the instructive ones to the evaluation set.

Over time the evaluation set becomes a precise record of what your feature must handle, and each improvement makes the next one safer.

Launch with guardrails

Even a well-evaluated feature should launch with sensible limits: a narrow scope at first, clear wording that the user is talking to an AI, an easy route to a human, and restrictions on any actions the AI can take. Expand the scope as the measurements justify it.

Evaluation and reliability are one of our research tracks. If you are preparing an AI feature for launch, our applied AI team can help you design the evaluation, or you can get in touch.