How to Test an AI Agent Before Production: A Pass-Bar Guide

A typical scene: an insurer’s claims team watches a new intake agent handle ten demo claims without a slip, and the process owner asks the question every owner asks: can we switch it on? Ten clean runs cannot answer that. An agent that is right 90% of the time (an Example figure) still gets all ten right about one time in three, because 0.9 to the tenth power is 35%.

Here is how to test AI agents before production, in the order we do it. Build a test set from real cases, including a few that must never go wrong. Write the pass bar, and name the person who signs it, before the first run. Run every case more than once. Let a single failed safety case stop the release. Keep the files, because for a high-risk system AI Act Articles 9, 10, 12 and 15 read like a description of them.

We read every page and article cited here on 15 November 2026, with the AI Act from the Commission’s AI Act Service Desk, which shows the text as amended. This is an engineer’s reading, not legal advice. Where a point needs your counsel or DPO, we say so.

What does it mean to test an AI agent?

Testing an agent differs from testing a function or a model because the agent acts. It calls tools, changes records, runs for several turns and gives a different answer to the same input from one run to the next. Anthropic’s engineers, in their guide to evals for agents, keep three words apart. A trial is one attempt at a task. A transcript is the complete record of that trial, including tool calls and intermediate results. The outcome is the final state of the environment: a booking agent may say the flight is booked, but the outcome is whether a reservation exists in the database.

That gives three layers to check, each with its own kind of grader.

One word on the process layer, because the two best-known guides pull in opposite directions. Anthropic warns that checking for an exact sequence of tool calls makes tests brittle, since agents find valid routes the test author did not expect, and advises to grade what the agent produced, not the path it took. Mastra’s guide says that an agent reaching the right answer by accessing data it should not have read is a compliance risk even when the outcome looks right. Both are correct. Grade the outcome, and write the process as rules (never do this, always do that), not as a script.

The whole method on one page: every change to the agent starts the loop again, with the same bar and the same signer.

How do you build a test set for an AI agent?

Start small and start now. Anthropic’s advice is that 20 to 50 simple tasks drawn from real failures is a great start, and that evals get harder to build the longer you wait. The first cases come from what you already do by hand: the checks you run before each release, the questions users ask most, and your support queue and bug tracker.

Add the cases your experts already know break the agent. StackAI’s guide suggests support tickets turned into tasks and a “we know this breaks” list from subject-matter experts, and warns that synthetic data tends to be cleaner than real users and can overestimate performance. Label synthetic cases, so you can see how much of the score rests on them.

Three rules make a test set worth trusting:

1. Write each case so two experts would reach the same verdict. Anthropic’s test for a good task is exactly that. Give each case a reference solution that passes every grader, which proves the case can be passed. If an agent scores 0% across many trials, suspect the case before the agent.
2. Balance the set. Include cases where the agent should refuse, ask a question, escalate or do nothing. StackAI puts it plainly: many agents fail not by doing too little, but by doing too much.
3. Cover what the process needs, by category, not just the average case. A starting set for the claims-intake agent might look like this:

For a high-risk system, the AI Act has a view of this set. Article 10(3) asks that testing data sets be “relevant, sufficiently representative, and to the best extent possible, free of errors and complete in view of the intended purpose”. Article 10(4) asks them to reflect the geographical, contextual, behavioral or functional setting of use.

In the amended text, Article 10(6) adds that for systems developed without training AI models, paragraphs 2, 3 and 4 of Article 10 apply only to the testing data sets. If your agent wraps a model you did not train, your test set may be the dataset Article 10 looks at. Whether that reading fits your agent is a question for counsel.

Real cases contain personal data, so GDPR applies to the test set too. Article 5(1)(c) asks that data be “adequate, relevant and limited to what is necessary”, so replace names, numbers and addresses with realistic stand-ins before a case goes in. Our GDPR and AI agents post covers the rest of the data path.

What is a pass bar, and who signs it?

A pass bar is the written threshold a version of the agent must clear before release, together with the name of the person who accepts it. Write it before the first test run. A bar set after you have seen the scores is a description of the scores.

The AI Act says the same for high-risk systems. Article 9(8) requires that testing “shall be carried out against prior defined metrics and probabilistic thresholds that are appropriate to the intended purpose”. Article 15(3) adds that “the levels of accuracy and the relevant accuracy metrics” must be declared in the instructions of use. The metrics you sign before testing are the ones whose measured levels you later declare in the instructions of use.

A pass bar has four parts, and an average alone is not one of them:

• An overall bar. The share of cases that must pass.
• A floor per category. StackAI’s checklist asks for a minimum rubric score by category, not just the overall average, because an average hides a weak category.
• A zero-tolerance list. StackAI calls them P0 events: any one blocks the launch regardless of averages, and in regulated settings these rules often matter more than the average success rate.
• No regressions. A case that passed on the previous version must not fail on this one. Mastra suggests keeping a regression suite that is expected to pass near 100%, separate from the harder capability cases you are still climbing.

Pick the number from the cost of a wrong answer, not from a default. A bar of 80% means up to one case in five may be wrong. Ask whether the process can live with that, and whether a person sees those cases before they matter. Here is a bar for the claims-intake agent, with every figure an Example:

We would not let the team that built the agent be the only signature. The builder knows how to make the numbers look good. The process owner knows what a wrong answer costs.

How many times should you run each test?

More than once, because one run hides failures. Anthropic’s guide describes two measures. Pass@k is the chance of at least one success in k tries. Pass^k is the chance that all k tries succeed, and it is the one that matters for a customer-facing agent: with a 75% success rate per trial, three trials all pass only (0.75)³, about 42% of the time. The τ-bench paper that introduced pass^k reported in 2024 that even state-of-the-art function-calling agents such as GPT-4o succeeded on fewer than 50% of tasks and were “quite inconsistent”, with pass^8 under 25% in its retail domain. Those figures come from 2024 models, but the reason to measure pass^k has not changed.

How many repeats is enough? A rule of thumb from statistics helps. If an event never occurred in n independent trials, the interval from 0 to 3/n is a 95% confidence interval for its rate, according to the rule of three. Our arithmetic from that rule: 100 clean trials cannot rule out a failure rate of 3 in 100, and 300 clean trials cannot rule out 1 in 100. Treat it as a rough ceiling, since repeated runs of the same case are not a sample of live traffic. It still explains why “it passed when we tried it” is weak evidence for a case that must never fail.

Three habits keep repeated runs honest:

• Start every trial clean. Anthropic’s advice is to isolate each trial in a fresh environment, because shared state between runs can cause correlated failures and can also artificially inflate performance.
• Calibrate the judge. For answers a code check cannot grade, a judge model scores a rubric. Anthropic says to calibrate it against human experts, give it a way out such as “Unknown”, and grade each dimension with its own judge. The MT-Bench study found strong LLM judges matched human preferences with over 80% agreement, the same level as humans agree with each other, and also named position, verbosity and self-enhancement biases. So we would not let a model judge its own work.
• Read the transcripts. Anthropic does not take eval scores at face value until someone has read some of them. Its own example: Opus 4.5 first scored 42% on CORE-Bench until grading bugs were fixed, after which the score jumped to 95%. A low score can be the test’s fault.

On our platform, each case runs in its own fresh test world. A failure decided by a judge is run again, up to three trials, and the case passes on two of three. A failure decided by code, such as a forbidden tool running or a write taking effect without approval, is final and never re-run, and a failed safety case is never outvoted by clean ones. A judge is noisy, so it gets a second look. An unsafe action is not noisy.

How do you test an AI agent against prompt injection and misuse?

Write adversarial cases, because the usual defenses do not remove the risk. OWASP’s entry on prompt injection (LLM01:2025) says that retrieval-augmented generation and fine-tuning do not fully mitigate it. It separates two kinds. Direct injection is when a user’s prompt changes the model’s behavior. Indirect injection is when the model accepts input from external sources, such as websites or files, that carry instructions. The indirect kind is the one an agent with tools meets in daily work, and a 2023 paper on it demonstrated attacks against real-world systems such as Bing’s GPT-4 powered chat.

To test for it, you can borrow from public work. The AgentDojo project built a test environment with 97 realistic tasks and 629 security test cases, and describes it as an extensible environment rather than a static suite, because attacks and defenses keep changing.

So your suite needs a standing set of must-never cases, plus new ones as attacks change. The core of the safety pack we run on every agent looks like this:

Add your own list for the process. For claims, that might include never quoting a payout figure before a handler approves it, and never confirming cover for a policy it cannot find. A case that does not apply to an agent, such as a write test for an agent with no tools that write, is recorded as not applicable from the agent’s tools. It is never set by the model or the client.

How do you read the results and decide on release?

Read them by category, and let the zero-tolerance rules override the average. In the example below, the agent scores 95% overall and clears every category floor, and the release is still blocked by one failed safety case. That is what a pass bar is for.

A sample scorecard, with every figure an Example. One failed safety case blocks the release whatever the overall score.

The decision itself belongs to a named person who sees the score, the failed and passed safety cases and what changed since the last version. If you are designing that screen, our post on human oversight in practice covers what the approval step should show so the reviewer is not just clicking.

Decide in advance what a miss means. On our platform, a missed safety case is never waived, and any other missed gate goes live only as a risk that a named person accepts.

What does an auditor want to see? AI Act Articles 9, 10, 12 and 15

An auditor wants a trail from the law’s words to files you can open. Articles 9, 10, 12 and 15 are requirements for high-risk AI systems, which Article 113 applies from 2 December 2027 for Annex III systems (as amended, read 15 November 2026). Whether your agent is high-risk depends on what it does, and our post on the AI Act for companies deploying agents shows how to tell. If it is not, the same files are still useful evidence when a customer or your own audit asks how the agent was tested.

What to keep for each article. The wording is in the table above, and counsel confirms how each article applies to your system.

Two things for counsel: whether your system is high-risk at all, and who is the provider and who the deployer, since Article 17 puts the quality management system on providers. A deployer has its own duties. Article 26(5) asks deployers to monitor the system’s operation on the basis of the instructions for use. Article 26(6) asks them to keep the logs under their control for at least six months, unless other Union or national law provides otherwise, in particular on personal data protection.

Does EN 18286 cover testing an AI agent?

No. EN 18286:2026 is a quality management system standard, and a quality management system is not a test plan. CEN and CENELEC published it in July 2026 as the first standard in support of the AI Act, and Commission Implementing Decision (EU) 2026/2274 has since published its reference in the Official Journal. That gives providers who implement it a presumption of conformity with the Article 17 requirements it covers. That is recent: an Intertek analysis from late September 2026 still called the standard uncited.

The presumption stops at Article 17. An analysis by Intertek, a firm that trains providers on the standard, says it directly: EN 18286 covers Article 17 and does not cover the essential requirements in Chapter III, Section 2. The standards for those requirements, including risk management, data governance and quality, logging, accuracy, robustness and cybersecurity, are still in progress. JTC 21, the joint CEN-CENELEC committee, says EN 18286 is the first deliverable under the standardization request and the others will follow the same path of adoption, Commission assessment and citation. The amended request runs until 28 February 2027. Until a standard for Articles 9, 10, 12 and 15 is cited, you show those through your own evidence, and the Commission notes that applying standards stays voluntary either way.

The link to testing is Article 17(1)(d), in the table above: the quality management system must contain your test procedures and how often they run. EN 18286 is a standard for building that system. Your pass bar and your scorecard are the evidence that Articles 9 and 15 look at. These are the documents you will meet:

How do you keep testing after release?

Treat every change as a new version and run the full set again. The model under your agent will change whether you ask for it or not. Anthropic retires older models, gives at least 60 days’ notice before retiring a publicly released one, and says to test applications on the replacement well before the retirement date. Behavior can also shift inside a model’s life. A study of two 2023 releases found GPT-4 identifying prime numbers with 84% accuracy in March and 51% in June. Its conclusion: the “same” service can change substantially in a short time.

A change that should start the loop:

• a new or updated model, or a provider’s retirement notice;
• an edited prompt, tool or knowledge document;
• a production incident, which first becomes a new test case;
• the calendar, with a scheduled run even when nothing changed.

Live traffic then feeds the next test set. Anthropic suggests sampling transcripts to read every week, and OpenAI’s guidance says to log everything so you can mine the logs for good eval cases. The law points the same way. Article 72 asks providers of high-risk systems to run a post-market monitoring system that collects and analyzes performance data throughout the lifetime of the system. Article 43(4) requires a new conformity assessment after a substantial modification of a high-risk system. Whether a model swap is a substantial modification is a question for counsel.

How does this look on our platform?

We built the test bench around this method, so the pieces line up one to one. Each version of an agent runs your test set and a safety pack, scored against the bar you set, 80% by default and up to 100%. The release gates are the overall bar, the easy cases, regressions against the previous version and safety. Every safety case must pass and cannot be waived, and any other miss can go live only as a risk that a named person accepts. Our own rule is stricter: we do not release below your bar.

A person confirms the release on a review card that shows the score, the safety cases and what changed. Every run is recorded step by step, which is what Article 12 asks the system to allow. The platform page shows each module, and how we work shows where the signed pass bar sits in an engagement.

A checklist you can copy

1. List the process’s must-never events, and turn each into a safety case.
2. Collect 20 to 50 real cases from tickets, manual checks and expert knowledge, with personal data replaced.
3. Add ambiguous cases, tool failures and cases where the right move is to refuse or escalate.
4. Check that two experts would give each case the same verdict, and write a reference solution.
5. Write the pass bar: overall, per category, zero tolerance, no regressions.
6. Have the process owner sign it, with the date, before the first run.
7. Run each case more than once, in a clean environment, with a calibrated judge for open answers.
8. Read the failed transcripts, and fix the test before the agent when the test is wrong.
9. Release only on a named person’s confirmation, looking at the scorecard.
10. Retest on every change, add production failures as cases, and keep the run records.

If you want the documents an auditor reads without building the test bench yourself, that is our job. Bring one process. In 45 minutes you get a first view of its risk class, the controls it needs and whether an agent is the right tool at all: book a scoping call.

Questions

How many test cases does an AI agent need before go-live?

Start with 20 to 50 cases drawn from real failures and the checks you already run by hand, as Anthropic's engineers advise. Grow the set by category as the agent matures, and add a case for every failure you see in production. Coverage of what must never go wrong matters more than the total.

What pass rate should an AI agent reach?

There is no universal number. Set the bar from the cost of one wrong answer in that process, per category, and write it down before the first run. Safety cases get no percentage: every one must pass. On our platform the overall bar is 80% by default and can be raised to 100%.

Is EN 18286 a standard for testing AI agents?

No. EN 18286:2026 is a quality management system standard for providers of high-risk AI systems, tied to AI Act Article 17. It gives a presumption of conformity for the Article 17 requirements it covers, and none for Articles 9, 10, 12 or 15, which cover risk management and testing, data, logging, and accuracy.

How often should you retest an AI agent after release?

Run the full test set on every change to the model, prompt, tools or knowledge, and before any model your provider is retiring is swapped out. Add a scheduled run, and read a sample of live transcripts every week. Each change is a new version with the same bar and the same signer.

Sources

1. www.anthropic.comwww.anthropic.com/engineering/demystifying-evals-for-ai-agents
2. mastra.aimastra.ai/articles/ai-agent-evaluation
3. www.stackai.comwww.stackai.com/insights/how-to-evaluate-and-test-ai-agents-before-deploying-to-production
4. ai-act-service-desk.ec.europa.euai-act-service-desk.ec.europa.eu/en/ai-act/article-9
5. ai-act-service-desk.ec.europa.euai-act-service-desk.ec.europa.eu/en/ai-act/article-10
6. ai-act-service-desk.ec.europa.euai-act-service-desk.ec.europa.eu/en/ai-act/article-12
7. ai-act-service-desk.ec.europa.euai-act-service-desk.ec.europa.eu/en/ai-act/article-15
8. ai-act-service-desk.ec.europa.euai-act-service-desk.ec.europa.eu/en/ai-act/article-17
9. ai-act-service-desk.ec.europa.euai-act-service-desk.ec.europa.eu/en/ai-act/article-26
10. ai-act-service-desk.ec.europa.euai-act-service-desk.ec.europa.eu/en/ai-act/article-43
11. ai-act-service-desk.ec.europa.euai-act-service-desk.ec.europa.eu/en/ai-act/article-72
12. ai-act-service-desk.ec.europa.euai-act-service-desk.ec.europa.eu/en/ai-act/article-113
13. digital-strategy.ec.europa.eudigital-strategy.ec.europa.eu/en/library/first-harmonised-standard-supporting-ai-act-published-official-journal
14. digital-strategy.ec.europa.eudigital-strategy.ec.europa.eu/en/policies/ai-act-standardisation
15. jtc21.eujtc21.eu/en-18286-becomes-the-first-harmonised-standard-under-the-eu-ai-act/
16. www.cencenelec.euwww.cencenelec.eu/news-events/news/2026/en-in-the-spotlight/2026-07-30-ai-quality-management/
17. arxiv.orgarxiv.org/abs/2406.12045
18. arxiv.orgarxiv.org/abs/2406.13352
19. arxiv.orgarxiv.org/abs/2302.12173
20. www.intertek.comwww.intertek.com/blog/2026/10-01-en-18286-first-eu-ai-act-standards-puzzle-piece-is-on-the-table/
21. arxiv.orgarxiv.org/abs/2306.05685
22. arxiv.orgarxiv.org/abs/2307.09009
23. genai.owasp.orggenai.owasp.org/llmrisk/llm01-prompt-injection/
24. en.wikipedia.orgen.wikipedia.org/wiki/Rule_of_three_%28statistics%29
25. gdpr-info.eugdpr-info.eu/art-5-gdpr/
26. docs.claude.comdocs.claude.com/en/docs/about-claude/model-deprecations
27. platform.openai.complatform.openai.com/docs/guides/evaluation-best-practices
28. www.iso.orgwww.iso.org/standard/81230.html
29. www.iso.orgwww.iso.org/standard/82570.html
30. www.nist.govwww.nist.gov/itl/ai-risk-management-framework

Read on or before 15 November 2026. Not legal advice: your counsel and data protection officer decide.

Written by

The Wise Agents team

Engineers who build and run AI agents for regulated companies. We read every article we cite on the day we state, and say what your counsel or DPO must confirm.

About us →