How does a Sierra agent issue refunds?
In Agent Studio, teams define journeys in natural language. Steps can reference articles, call external APIs or gather information from the customer, and Sierra cites a sporting retailer that launched with a returns journey. Journeys can vary by channel, for example sending authentication links in chat and confirmation codes over the phone.
The Agent SDK is the code route. Sierra describes it as a way to set goals and guardrails declaratively, compose skills such as triage, respond and confirm, and connect securely to internal or external systems to take actions like updating a subscription. Sierra quotes a customer who uses it to write customer journeys as code and track changes.
Sierra’s public pages don’t describe refund-specific settings such as amount limits or approval steps. Those live in the journey logic and in the API a journey calls.
Which guardrails does Sierra describe?
Sierra puts most runtime control in supervisor models that run alongside the agent.
| Control | What Sierra describes |
|---|---|
| Supervisors | Audit agent actions and responses. On high-risk topics they can move from observing to intercepting a response before it reaches the customer |
| Input supervisor | Screens user input for threats, from multi-turn context poisoning to advanced jailbreaking |
| Graded autonomy | Agents get much less autonomy on sensitive topics and when someone is trying to exploit them |
| Determinism per workflow | The SDK lets you tune how flexible the agent is, with varying levels of creativity and determinism |
| Contact center handoff | Routes the conversation to a team member with a summary |
What do Sierra’s simulations cover?
Teams create synthetic users that differ in language, comfort with technology and tone, and set what the agent knows at the start, such as whether the user is logged in. A separate judge agent grades each conversation: did the user reach their goal, did the agent follow the standard operating procedure, stay on brand, and answer accurately and clearly.
Sierra says agents should pass simulations before changes to any journey are published, and developers can gate releases on specific simulations, run through GitHub Actions or the command line. Test cases can be generated from SOPs, knowledge bases, historical transcripts and conversation flows.
Sierra also developed τ-bench, a public benchmark for customer-facing agents. The τ²-bench repository includes retail and airline domains with a simulated user. It measures models on shared tasks, separate from the simulations you run on your own agent.
What do the simulations leave out?
Sierra’s examples of simulated users vary style and need: language, tone, technical comfort. The simulations post doesn’t describe users acting in bad faith, and it doesn’t say whether APIs are mocked during a run.
Supervisors screen for jailbreaks and context poisoning at runtime. A customer who calmly says a manager approved a refund isn’t attempting a jailbreak, so policy cases like that need simulations of their own before release.
What to test before giving a Sierra agent refund authority
Illustrative cases to add alongside your cooperative simulations:
- A logged-in user asking for a refund on an order that belongs to a different account (identity abuse).
- A claimed prior promise from staff or an earlier chat (fake prior approval).
- A returns journey where the customer haggles for a refund plus store credit (negotiation pressure).
- Concessions requested in small steps over a long conversation (multi-turn escalation).
- A customer who returns in a new session citing what the agent “agreed” last time (memory and multi-session attacks).
- The refund API called twice for one order, checked in the API itself.
Questions
Can Sierra simulations block a release?
Yes. Sierra says agents should pass simulations before journey changes are published, and developers can gate releases on specific simulations through GitHub Actions or the command line.
Do Sierra supervisors replace pre-release testing?
No. Supervisors act on live conversations, intercepting responses on high-risk topics and screening input for jailbreaks. Simulations check behavior before release. Refund policy cases benefit from both, plus limits in the API the journey calls.
Is τ-bench the same as Sierra simulations?
No. τ-bench is a public research benchmark Sierra developed for evaluating models on customer-service tasks. Simulations are the product feature you use to test your own agent and journeys.
Sierra is a trademark of its owner. This guide is independent and not affiliated with or endorsed by Sierra.
Free checklist
Get the guardrail checklist for agents that move money.
A one-page PDF of the controls and the test for each. Download it right away, and we email you the link.
Sources
- Sierra blog: Simulations, the secret behind every great agent
- Sierra blog: Enterprise-grade agents
- Sierra blog: Meet Agent Studio
- Sierra: Agent SDK
- GitHub: sierra-research/tau2-bench