What question does each kind of testing answer?
Red teaming in the AI security sense probes a model or application for harmful output, data exposure and injection. The OWASP GenAI Red Teaming Guide covers model-level issues such as toxicity and bias as well as system-level risks such as API misuse and data exposure.
Simulation testing runs the agent against simulated users who want something ordinary done. tau-bench is the best-known public example: a language model plays the user from a task instruction, and success is judged by the final database state.
Adversarial counterparty testing plays a customer, supplier or debtor who wants more than they are entitled to and will lie, push and plant instructions to get it. It scores the money and commitments the agent gave away against the company’s own policy.
How do the three approaches compare?
Categories, not products. Many tools combine more than one.
| Jailbreak and injection red teaming | Simulation with cooperative personas | Money-focused adversarial testing | |
|---|---|---|---|
| Main question | Can it be made to produce forbidden output or actions? | Does it complete normal tasks correctly and consistently? | How much can a motivated counterparty extract? |
| Who plays the other side | Attack prompts, payload lists, automated attackers | Simulated users with a goal and a persona | Scripted, persona and search attackers with a profit motive |
| Typical inputs | Jailbreaks, role-play, injection payloads | Ordinary requests, some edge cases | Haggling, fake approvals, fraud stories, injected tickets |
| Judged by | Harmful or leaked content, unsafe tool use | Task success, final state | Dollars conceded, clauses broken, unbacked promises |
| Finds | Safety filter gaps, data leaks, injection paths | Workflow bugs, tool misuse, inconsistency | Refunds, credits and discounts beyond policy |
| Tends to miss | Polite, plausible requests that break business rules | Customers who lie or push | Content-safety issues unrelated to money |
| Best run | Before launch and on model changes | Continuously during development | Before granting money authority, and on every change |
Why don’t jailbreak tests catch refund abuse?
The customer who gets an unearned refund usually says nothing a content filter would flag. They are polite, they claim a manager approved it, and the story is plausible. The failure is a business-rule violation, not harmful text.
Research on policy-adherent customer-service agents makes the gap concrete: a multi-agent attacker using policy-aware persuasion outperformed conventional jailbreak methods such as DAN-style prompts, emotional manipulation and coercion (Nakash et al., 2025).
Why don’t cooperative simulations catch it?
A simulated user given an ordinary goal tries to reach that goal, not to extract more than it is owed. That is the right design for measuring task completion and the wrong one for measuring exposure.
The same research group built tau-break on top of tau-bench specifically to add manipulative users, because the original benchmark measures helpfulness with cooperative ones (Nakash et al., 2025).
Where does each fit in a release process?
A team shipping an agent with money authority can use each where it is strongest.
- During development, run simulations to check the agent does the job for normal customers.
- Before launch and on model changes, run jailbreak and prompt-injection red teaming against the full application, including tools.
- Before giving the agent authority over refunds, credits or discounts, run adversarial counterparty scenarios scored against the written policy.
- On every prompt, model, tool or policy change, rerun the adversarial and simulation suites as a regression gate.
Where does prompt-injection testing overlap with adversarial testing?
Indirect injection sits in both. Benchmarks such as AgentDojo, with 97 tasks and 629 security test cases including e-banking, test whether instructions hidden in data can redirect a tool-using agent. Money-focused testing reuses that idea but scores the outcome in dollars: a planted note in a ticket counts as a failure only if a refund or discount followed.
Questions
Is adversarial testing the same as a penetration test?
It borrows the idea of an authorized attacker, but the target is the agent’s business decisions rather than infrastructure. A penetration test asks whether someone can get into the system. Adversarial counterparty testing asks what someone can get out of the agent through its normal channel.
Can one test suite cover all three?
One test setup can run all three if it records tool calls and final state, but the attackers and the scoring differ. Keep the results separate so a high task-success rate does not hide a high concession rate.
Which should come first?
Simulation, because an agent that fails ordinary tasks is not ready for anything else. Adversarial testing should come before the agent gets authority to move money, not after the first incident.
Attack families
Early access
Test your agent against this.
We will email you when Countermove opens testing for agents like yours.
Sources
- OWASP: Announcing the OWASP Gen AI Red Teaming Guide (2025)
- Yao et al., "tau-bench" (arXiv, 2024)
- Nakash et al., "Effective Red-Teaming of Policy-Adherent Agents" (arXiv, 2025)
- Debenedetti et al., "AgentDojo" (arXiv, 2024)
- OWASP, LLM01:2025 Prompt Injection