Why does one run tell you so little?
The same scenario can go differently each time: the model samples, the simulated customer phrases things differently, and an early turn changes everything after it. In tau-bench the agent ran at temperature 0 while the simulated user ran at temperature 1.0, and agents still varied across trials.
The paper reports that state-of-the-art function-calling agents such as gpt-4o succeeded on under 50% of tasks and were inconsistent, with pass^8 under 25% in the retail domain. For an agent that moves money, a determined customer gets more than one try.
What is pass^k?
pass^k is the chance that all k independent attempts at a task succeed, averaged across tasks. tau-bench estimates it from n runs with c successes as C(c, k) / C(n, k) per task. pass@k is the opposite: the chance that at least one of k attempts succeeds, which measures whether a solution can be found, not whether it is reliable.
For a refund agent, define success as ‘no clause broken and no dollars beyond policy’. Illustrative: with 8 runs and 6 successes, pass^4 is 15 / 70, about 0.21. The table shows how fast pass^k falls if runs are independent with a fixed per-run success rate.
| Per-run success rate | pass^1 | pass^4 | pass^8 | pass^16 |
|---|---|---|---|---|
| 99% | 0.99 | 0.96 | 0.92 | 0.85 |
| 95% | 0.95 | 0.81 | 0.66 | 0.44 |
| 90% | 0.90 | 0.66 | 0.43 | 0.19 |
How do you put a confidence interval on a failure rate?
Use the Wilson score interval. Instead of adding plus or minus a margin to the observed rate, it asks which true rates could plausibly have produced what you saw. It stays between 0% and 100% and still gives a useful upper bound when you saw zero failures, where the simple ‘plus or minus’ formula gives a width of zero.
The NIST/SEMATECH e-Handbook gives the formula and notes it is recommended for almost all combinations of sample size and rate. With observed rate p, n runs and z = 1.96 for 95%, the center is (p + z²/2n) / (1 + z²/n) and the half-width is z·√(p(1−p)/n + z²/4n²) / (1 + z²/n).
| Failures / runs | Observed rate | 95% Wilson interval |
|---|---|---|
| 0 / 10 | 0% | 0% to 27.8% |
| 0 / 20 | 0% | 0% to 16.1% |
| 1 / 20 | 5% | 0.9% to 23.6% |
| 2 / 50 | 4% | 1.1% to 13.5% |
| 0 / 100 | 0% | 0% to 3.7% |
| 5 / 100 | 5% | 2.2% to 11.2% |
| 0 / 300 | 0% | 0% to 1.3% |
How many runs do you need?
Decide the failure rate you need to rule out, then work backward. A quick guide is the rule of three: with zero failures in n runs, the 95% upper bound on the rate is about 3/n.
- To show a failure rate is likely below 10%, you need about 30 clean runs.
- Below 5%, about 60 clean runs.
- Below 1%, about 300 clean runs.
- Any failures push the number up. Spend runs on high-severity scenarios first and use cheap scripted attackers for broad regression coverage.
How do you tell a real regression from noise?
Compare versions in pairs. Run the old and new agent on the same scenarios with the same seeds, then count only the pairs that disagree: b runs that passed before and fail now, c that failed before and pass now. Pairs that agree carry no information about the change.
McNemar’s test compares b and c. With fewer than about 25 discordant pairs use the exact binomial version. Illustrative: across 200 paired runs, 10 newly failing against 1 newly passing gives a two-sided exact p of about 0.012, a real regression. Six against two gives p of about 0.29, which the data cannot separate from noise.
What should a reliability report show?
One row per scenario, with enough detail to reproduce it.
- Number of runs and number of failures
- Failure rate with its 95% Wilson interval
- pass^k for a k that matches how often a customer can retry
- Mean and worst-case dollars conceded
- Model, prompt, tool, policy and attacker versions, and seeds
- For a release: the paired comparison against the previous version
Questions
Does setting temperature to zero make one run enough?
No. It reduces sampling variation in the agent, but the customer side, tool results and conversation path still vary. In tau-bench the agent ran at temperature 0 and results still differed across trials.
Which k should I report?
Pick k to match exposure: roughly how many attempts a determined customer or a group of customers gets at the same scenario. Report pass^1 as well so readers can see the per-run rate.
Is pass^k the same as pass@k?
No. pass@k rewards finding a solution in any of k tries and rises with k. pass^k requires all k tries to succeed and falls with k, which is what matters when every conversation can cost money.
Early access
Test your agent against this.
We will email you when Countermove opens testing for agents like yours.
Sources
- Yao et al., "tau-bench: A Benchmark for Tool-Agent-User Interaction in Real-World Domains" (arXiv, 2024)
- NIST/SEMATECH e-Handbook of Statistical Methods, 7.2.4.1 Confidence intervals
- Wikipedia: Rule of three (statistics)
- Wikipedia: McNemar's test