Verification
Report independently accepted outcomes and total cost, not the number of agent turns or a model's assertion that code works.
Private internal project. Features described here are evidence-limited and are not offered as a public execution service.
Evidence hierarchy
A syntactically valid response is not proof of correctness. Passing visible tests is weaker than passing independent withheld acceptance; a signed receipt documents which checks ran, while additional review evaluates scope, correctness, and potential security regressions.
Ralph retry pilot
On one frozen coding defect, the recorded two-run baseline and two-run retry configuration both produced zero independently accepted completions. Two attempts consumed substantially more tokens and time. All candidates repeated the same patch. The result is negative for this single task, not an estimate of general coding performance.
Guardrails before promotion
Keep an explicit policy for immutable test sets, source revision, model configuration, attempt and spend caps, signature checking, host immutability, and guest teardown. A new retry or routing policy must improve representative held-out outcomes before being promoted.
What's not established
The early-stop experiment has unit-test support but no completed live acceptance as of its recorded review. Whole-system security certification, broad model competence, unattended production operation, and verified total-cost savings have not been demonstrated.
Research standard
Any future results should include task count, repetitions, held-out split, uncertainty, elapsed time, token usage and operating constraints—not just a percentage labeled improvement.