A release gate, end to end.
A fictional expense-reimbursement agent, twelve versioned cases, three recorded output snapshots and one reproducible release decision. Every result can be regenerated from the public fixtures.
Fictional workflow and data. This demonstrates delivery structure and evaluator behaviour. It is not a client outcome, testimonial or production benchmark.
Quality alone did not decide the release.
Candidate v2 removed the baseline's two critical failures, but its tail latency crossed the agreed threshold. It was held for review rather than waved through on a better pass rate.
| Version | Cases passed | Critical failures | P95 latency | Average cost | Decision |
|---|---|---|---|---|---|
| Baseline v1 | 8 / 12 | 2 | 1,100 ms | $0.01325 | BLOCK |
| Candidate v2 | 11 / 12 | 0 | 5,200 ms | $0.01250 | REVIEW |
| Candidate v3 | 12 / 12 | 0 | 2,050 ms | $0.01100 | PASS |
Twelve cases built from observable policy boundaries.
The evaluator does not ask another model whether an answer feels safe. It checks decisions, required tool calls, argument values, policy citations, forbidden actions, latency and cost.
Nothing important is hidden behind the report.
Open the policy, fixtures, evaluator, raw outputs and machine results. The PDF is the summary; these files are the evidence.
What this sample proves, and what it does not.
It demonstrates
- A concrete case and output contract
- Deterministic decision and tool-call checks
- Quality, latency and cost gates together
- A reproducible pass, review or block decision
- A handoff another engineer can extend
It does not claim
- A previous consulting-client result
- Production traffic or customer data
- A universal policy or evaluation framework
- Model quality beyond these fixtures
- Guaranteed outcomes for another workflow