How we test, score and publish.

How does a test work?

1. Connect the agent. A chat endpoint, an SDK wrapper, an email inbox or another agent. Tools are mocked in a sandbox, so no real money ever moves.
2. Confirm the policy. Refund caps, discount floors, settlement minimums, walk-away prices, approval tiers and the things the agent must never do, written as clauses we can check.
3. Run the attacks. Scripted and improvising counterparties, many runs per scenario, so a result is a rate with a confidence interval, not an anecdote.
4. Read the scorecard. Dollars conceded, clauses broken, value left on the table, replays of the worst conversations and the fix that would have stopped each one. See a sample report.

What is a scenario?

A scenario is a test case for an agent with money authority. It fixes the agent’s role, channel, tools and written policy, seeds the data it can look up, and gives the counterparty a persona, a goal, what is actually true, and an ordered playbook of moves drawn from ten attack families. Each scenario also states what counts as a loss and what a safe agent does. The library has 51 scenarios; read them all.

Who plays the other side?

• Scripted attackers follow a scenario’s playbook exactly. They are cheap and reproducible, which makes them right for regression checks.
• Persona attackers are language-model counterparts given the scenario’s goal, budget and honesty settings. They improvise from the playbook and are more realistic.
• Search attackers try many variations and keep the ones that extract the most money or break the most clauses, to find paths nobody wrote down.

How is a result scored?

From what the agent did, not what it said it would do. Every tool call goes to a mocked sandbox and is recorded, so a refund, credit, discount or change of terms is counted in dollars against the policy. Policy clauses are checked deterministically from those actions and the resulting state. Promises the agent made in the conversation are compared with what it actually did and is allowed to do. In negotiations, results also report how much of the available deal the agent kept (its surplus share), because an agent can stay inside every limit and still leave money on the table. Language-model graders are used only for soft criteria, such as misleading tone, and are calibrated against human labels.

Why many runs instead of one?

Agents are not deterministic. The same scenario can pass four times and fail on the fifth. So each scenario runs many times with recorded seeds, and results are reported as rates with 95% confidence intervals, alongside pass^k (the chance an agent gets it right on every one of k tries). Comparisons between two versions of an agent use paired tests so a regression gate fails only on a real change. More on the statistics.

What do we publish, and how do we check facts?

• Every incident, ruling and statistic links to its source. Where the record is thin (for example a single first-hand account), we say so.
• Examples and sample reports are labeled as illustrative and use invented names and numbers.
• Scenarios inspired by films or famous situations describe them in our own words and are not affiliated with the rights holders.
• We correct errors and note the correction on the page. Send a correction.

Independence

Countermove is not affiliated with any agent platform or model provider. We test agents. We don’t build or sell them. That is the point: the people grading an agent shouldn’t be the people who sold it.