What building an AI dev harness actually taught me
I spent months building a model-agnostic harness for doing real software work with coding agents. In the end, the features mattered a lot less than I expected. What actually mattered, was a handful of ideas that changed how I think about building with agents, instead of just prompting them.
These are the ideas that carry the most weight. The first three genuinely changed my mental model. The rest are practical things you can start using almost immediately.
1. Freeze the acceptance criteria before the agent begins
And put it somewhere it can't edit.
This is probably the biggest shift for me. It sounds obvious in retrospect, but I didn't really appreciate how important it was until I saw it in action.
If acceptance criteria can change after the work exists, they'll slowly shift toward whatever was produced. The agent isn't being dishonest, it's simply judging the work against a target that can still move. If the goalposts are movable, they'll eventually move.
That's not a prompting problem, it's a structural one, so the solution has to be structural too.
Illustrative task / Add an enquiry form
Frozen before work begins
- Email is required.
- Save the enquiry before sending.
- Keep the enquiry if sending fails.
this brief
await validate(form);const enquiry = await save(form);await notify(enquiry);The agent can edit the code. It cannot rewrite the definition of done.
Write the acceptance criteria first. Store them outside the editable workspace, and automatically block any attempt to edit them. Now the agent is working toward a contract it can't renegotiate. The task isn't finished because the agent says it is. It's finished because it satisfies the original contract.
2. Nobody should grade their own work
The model that wrote the code is the worst possible judge to decide whether it's correct. By the time it finishes, it's already convinced itself that the solution makes sense.
So split the responsibilities.
Coding model
Creates the solution.
Deterministic scripts
Check what can be tested.
Separate reviewer
Challenges the result.
The reviewer receives the work and its evidence, without the builder’s conversation.
Let deterministic scripts verify everything mechanical. A script can't invent a successful exit code. Treat the coding model as an engineer, not as the reviewer. Then hand the result to a fresh-context reviewer, one that never watched the code being written, to judge the design, intent, and overall quality.
That simple separation catches far more problems than I expected, and it's surprisingly cheap to do.
3. Enforcement is easy. Calibration is the hard part.
This realization changed how I thought about the whole harness.
Most agent systems, including earlier versions of mine, treat every task roughly the same. A one-line typo fix goes through the same process as a major architectural redesign. That's either a waste of time or too risky.
The key is proportional effort.
First classify the work. Is it a small change that nothing depends on, or core infrastructure that everything depends on? Is it business logic or generic utility code?
Then let that classification decide how much review it gets, which model should handle it, and how much autonomy it's allowed.
Fix a typo.
Light review
Small model · High autonomy
Change a shared utility.
Moderate review
Standard model · Medium autonomy
Redesign a core flow.
Heavy review
Strong model · Low autonomy
Illustrative tasks. The actual scope, dependencies and business impact determine the level of checking.
Getting enforcement in place is easy. Getting the effort to match is where the value really starts to compound.
The Practical Layer
Once those bigger ideas were in place, a few smaller practices paid off almost immediately.
- Put your rules in the harness, not in the prompt. A prompt that says "don't make edits in the main repo" is just advice. The same rule enforced as a blocked tool call becomes impossible to ignore. The important rules shouldn't rely on the model choosing to behave.
- Keep state outside the conversation, and use a fresh worker for every task. Store task records externally, use isolated worktrees, and process changes through a merge queue. Once the state lives outside the chat, context limits stop being your biggest constraint. Tasks become parallel, resumable, and much easier to reason about.
- Cross-provider review beats same-provider-review. Models can be more forgiving of their own work than someone else's. Using a model from a different provider, is a simple, effective way to reduce that bias.
Why the principles work
None of these ideas depend on a particular model. They're about harness engineering. Protocols, verification, review pipelines, and separating responsibilities.
The model at the top of the benchmarks will keep changing.
But the principles probably won't.
The lasting value isn't in the model, and it isn't in the features. It's in the scaffolding you build around them.
“Does this change satisfy the original contract?”
Design · Intent · Quality