AI engineering
The harness is the product, not the model
Two people used the same AI on the same day. One got a working pull request, the other got confident nonsense. Nothing about the model was different.
A colleague asks their assistant to fix a failing test. It edits a file, announces the fix, and moves on. The test is still red. Nobody told it.
You ask yours the same thing. It edits the file, runs the suite, reads the failure, edits again, runs it again, and only then says it’s done.
Same model. Same day. The difference is everything wrapped around it — and that wrapper has a name.
What is a harness?
A language model, by itself, is a function: text goes in, text comes out. It cannot open a file, run a command, or discover whether the thing it just claimed is true. It is closer to a very well-read colleague shouting advice down a phone line than to someone sitting at your keyboard.
The harness is everything that turns that into something which does work:
- Context — what it can see. Which files, how much history, which rules.
- Tools — what it can do. Read, edit, run a command, search.
- Permissions — what it may do without asking.
- Feedback — how it learns whether that worked.
- The loop — how many times it gets to try again knowing the answer.
Swap the model and each individual guess gets a little better. Change the harness and you change how many guesses it gets, what it knows when it guesses, and whether wrong ones survive contact with reality.
Before you read on
An assistant edits a file and breaks the build. Which change most reliably stops it happening again?
-
It helps at the margins, but a bigger model with no way to run your tests still cannot know it broke the build. You are paying more for a better guess.
-
This is the fix. The failure wasn't a reasoning failure, it was a feedback failure — nothing ever told it the outcome, so it had nothing to correct against.
-
Better instructions prevent some mistakes but cannot catch one after it happens. A prompt is guidance before the fact; a test result is evidence after it.
Why does it matter?
Because the instinct — pay for the better model — is usually the expensive answer to the wrong question.
One shot, no evidence
It writes something plausible. Nothing checks it. Any error reaches you fully formed and stated with total confidence.
you: fix the failing test in cart.js
model: [writes a change]
"Fixed — the total now includes tax."
# Did the test pass? Nobody knows.
# Not even the model.Same model, allowed to find out
It can run the suite, read the failure and try again before you ever see the result.
you: fix the failing test in cart.js
model: [reads cart.js and its test]
[edits] [runs: npm test cart]
-> 1 failing: expected 12.50, got 10.00
[reads failure, edits again]
[runs: npm test cart] -> pass
"Tax was applied before the discount."Nothing about the model changed between those two. The second works because it was allowed to discover it was wrong.
How does the loop work?
The loop is the engine room of a harness: the cycle it repeats until the work is actually done rather than merely attempted.
Pull in what's needed to decide — the relevant files, the error, the project rules, what happened last turn. This is the step people skip, and it sets the ceiling on everything after it.
A model reasoning over the wrong three files cannot be rescued by a better prompt.
The model picks the next action. Not the whole plan — the next action. This is the only step that is genuinely the model's job, and it is the smallest part of the cycle.
The harness carries it out. Note that the model does not do this: it states an intention and the harness executes it, which is exactly where permissions live.
# the model asks
edit("src/cart.js", ...)
run("npm test cart")
# the harness decides if it's allowed,
# does it, captures the resultThe result goes back in — exit code, output, the diff that actually landed. This is what separates an agent from autocomplete. Without it, the next decision is made in the dark.
If your agent never sees the outcome, you don't have a loop. You have a sequence.
Is this done? Not 'does it look done' — is there evidence. A passing test, a clean type-check, a 200 response. If yes, stop. If no, back to Gather with what was just learned.
The most common production failure is a loop that judges on vibes and exits early.
Run that once and you have a chatbot with tools. Run it until the evidence says stop and you have something that finishes work.
What are the limitations?
Harnesses fail too, and only one of the three common ways is the model’s fault.
It stops too early. It decides it’s done because the code looks right. No test ran, or one ran and the failure was dismissed as unrelated. The fix is to make the exit condition a check rather than an opinion.
It never stops. It re-reads the same file, tries the same fix, hits the same error, forever. Usually the feedback isn’t actually informative — the same output is being fed back with nothing new in it. Cap the iterations, and change the input when a turn produces no new information.
It stops for the wrong reason. It runs out of context and quietly drops the first half of the task. That is a context-management problem, and it belongs to the harness, not the model.
Before you read on
An agent has looped fifteen times on the same failing test without progress. What's most likely wrong?
-
Sometimes true, but fifteen near-identical attempts is a specific signature. A model out of its depth usually produces varied bad attempts, not the same one repeatedly.
-
Repetition without progress almost always means the feedback isn't changing. It's being asked the same question with the same inputs, so it gives the same answer. Something must change per turn or it cannot converge.
-
That would produce more varied attempts, not progress. Randomness isn't a substitute for information — you'd get fifteen different wrong answers instead of one repeated.
Worth saying plainly: a good harness does not make a weak model clever. It stops a capable model from working blind. Those are different claims, and only the second one is reliably true.
What this means for you
- When an assistant disappoints, check the wrapper before the model. Can it see the failing test? Can it run the thing that proves it worked? Does the result come back to it?
- Make 'done' a check, not an opinion. A passing test or a clean type-check is evidence; 'this looks right' is not, and an agent that exits on confidence will ship confident mistakes.
- If it loops without progress, something in the loop isn't changing. More attempts won't help — different information will.
Take the last task your assistant got wrong and ask one question: was it ever told the outcome? If not, that's your fix, and it's cheaper than a bigger model.