Reproduce
Choose one workflow, reproduce its failures and agree on what must be true before the next release ships.
Production agent reliability
Turn production agent failures into release gates.
In four weeks, I take one production workflow, turn its real failures into repeatable tests and wire pass, review or block checks into the release path.
One workflow Four weeks
01
FailuresTraces, incidents, corrections
02
Regression basketVersioned cases and checks
03
Release decisionPass, review or block
7+ yearsProduction ML and AI
13 model familiesEvaluation at IBM
3 days to 1 dayAzure OpenAI release cycle
One bounded scopeNo open-ended staff augmentation
The engagement
What exists after four weeks
At the end, your team can rerun the same evidence before shipping a new model, prompt, tool or harness change.
A clear list of the failures already appearing in traces, incidents or reviewer corrections
25 to 40 executable regression cases for one production workflow
Hard checks for permissions, tool arguments, citations, schemas and final state
A reproducible baseline showing quality, latency and cost side by side
A CI or pre-release check that says pass, review or block
Working code, documentation and a technical handoff
Primary engagementUSD 10,000
USD 5,000 after signature, required to clear before kickoff. USD 5,000 when the agreed artifacts are delivered.
- Scope
- One workflow
- Meetings
- Kickoff, midpoint, handoff
- Delivery
- Code, tests, gate, documentation
- Environment
- Client-approved; synthetic or redacted inputs are acceptable; no production access or data export required
Need technical judgment without an implementation?
USD 300/ hour
Advisory sessions are booked and prepaid in two-hour blocks.
- Evaluation design, failure review, release decisions or the next experiment
- Preparation and repository review use booked time
- No coding, production changes, on-call help or ongoing messaging
Encode
Turn recurring failures into versioned cases with explicit inputs, expected outcomes and checks.
Gate
Measure current behaviour, then compare model, prompt, tool or harness changes against the same cases.
Handoff
Wire the gate into the release path, document it and hand it to the engineer who will own it.
Method
Evaluate the system the user actually meets
Score the delivered state
A transcript can look correct while a file, tool action or database mutation is wrong. The gate follows the surface the user receives.
Prefer hard assertions
If permissions, amounts, schemas, citations, retries or final state can be checked directly, I encode the rule instead of asking another model to judge it.
Keep the comparison honest
A candidate does not pass because one quality score rose while latency, cost or variance got worse.
Evidence
Built around real release decisions
I have done this work inside production ML teams: finding failures, turning them into checks and deciding whether a model change is safe to release.
REAL Brokerage
Built agent specialists for real-estate workflows with an LLM verification loop and an evaluation harness used to gate releases.
Microsoft Azure OpenAI
Owned configuration, validation and latency benchmarking for OpenAI model releases into Azure, plus model-layer incident escalation.
IBM
Built a code-generation evaluation framework across 13 model families, combining executable correctness with structured model assessment.
Public case 01
PrismAI
The evaluation was looking at the wrong output surface.
A document-producing agent can expose several different states: user-visible streamed text, a corrected segment and stored source used to generate the document. Transcript-only scoring collapsed those surfaces and produced the wrong evaluation boundary.
I changed the runner so required and forbidden literal checks can use the correction segment plus stored source when retrieval succeeds, while earlier unsupported text remains visible as a separate diagnostic. The public record also states what is still missing: rendered-file verification, fail-closed retrieval and deterministic scorer tests.
Good fit
The workflow is already real.
This is for an agent in production or late-stage beta where traces, incidents or reviewer corrections already show recurring failures and one technical owner can make decisions quickly.
Inside the sprint
- One production workflow
- Failure-derived cases
- Replay and comparative baseline
- Explicit release thresholds
- Engineering handoff
Outside the sprint
- Foundation-model training
- General platform rewrites
- 24-hour production support
- Unbounded dashboards or integrations
- Exporting prohibited client data
One workflow. Four weeks.
What failure should your next release never repeat?
Request a 20-minute fit call
Email the workflow and your time zone. I will reply with two or three concrete times.