The Trap: When the Harness Gets Good Enough to Ignore

I go through phases.

Some weeks I’m in a really good flow with this. I’m reading the improvements, working through the event log, making decisions about the conventions, actively steering the harness. It works well and I can feel it working.

Other weeks I’m just not, for want of a better word, interested.

And over time I’ve noticed something uncomfortable about the correlation between those phases and the quality of what comes out the other end.

Involvement Isn’t About Writing the Code

I want to be precise about what I mean by involved, because it isn’t what people usually assume.

I don’t mean sitting there writing the code myself. I’m perfectly happy for the agents to do that. I mean being involved in working with the harness: making the decisions, directing it, choosing what goes in and what doesn’t.

When I’m doing that, I get results I can use.

When I hand the harness a broad prompt and say do this, it goes off on genuinely enormous tangents and I end up somewhere I don’t understand.

Which brings me to the best example I’ve got, and it isn’t a flattering one.

Three Hundred Improvements

In the last article I described the improvement loop: the agents record where they got something wrong, and I approve those improvements before anything enters the agents’ context.

The key word there is I.

I let the harness run for forty-odd tasks without going through the improvements. By the time I looked, I had about three hundred of them sitting there waiting for me.

So I did the obvious thing. I built a system to deal with them.

I called it triage. The idea was that it would read through the improvements and the patterns, group the ones that were similar — the same mistake made repeatedly, or close variations of it — and then work them through into the conventions model.

And it did. It really did.

It wrote a set of Python files to work through the improvements and figure out what needed doing. It produced JSON files holding all of that information. It got quite involved, quite quickly.

Out of the three hundred, it identified around thirty or forty clusters worth acting on. So thirty or forty pieces of work. I built commands so I could process as many as I wanted in a run, with a default of four.

Then I ran it.

Four Improvements, Half a Week’s Tokens

It processed four improvements.

Four. And it used roughly half my weekly token limit to do it.

It was writing directly into my agents, which was what I’d asked for. But the processing was enormous, and it had that same feel as the loop I wrote about earlier: something grinding away well past the point of being sensible.

Then I looked at what it had actually written.

My orchestrator file went from somewhere around thirteen hundred lines to about six thousand.

Six thousand lines of instruction, injected into the agent whose entire job description was supposed to be short, clear and non-negotiable. Bloat, and a lot of it.

I Didn’t Understand My Own System Anymore

This is the part that actually bothered me, and it wasn’t the cost.

I tried hard to work with what it had produced. But at a fundamental level, I didn’t understand it. I couldn’t tell you what it had decided or why. I’d lost the thread.

I built a system to give myself visibility, and then stopped looking at it.

And because triage had been built directly into the Bishop harness rather than sitting alongside it, there was no clean way to unpick it.

So I stopped. Properly stopped, and started again from the beginning.

I brought back my five agents. I brought back the standard memory system. I started working with it the way I had before.

That’s also the reason I ended up going the SQL route for memory and improvements. Not because SQL is exciting. Because I needed something simpler that I could actually process day to day without it costing a fortune in tokens to reason about.

The Trap

Here’s the pattern I think is worth warning people about, because I walked straight into it.

You put in the work. You do the improvements. You maintain the conventions. And because you’re doing that work, the system genuinely gets better. The answers get better. Your confidence in it goes up, and it deserves to go up.

Then one day you think: right, it’s good now. Let it go.

And that’s exactly when it goes off the rails.

The trust wasn’t misplaced. It was earned by your involvement, and then you removed the thing that earned it. The better it gets, the more tempting it is to stop doing the thing that made it good.

And It Wasn’t the AI’s Fault

I want to be clear about this, because it would be easy to write that story up as an AI failure.

It wasn’t.

I gave it a very broad instruction and pointed it at three hundred improvements. It did what I asked. The result was wrong because I was too lazy to do the improvements myself, so I asked a machine to absorb a backlog I’d created by not doing my own job.

Expecting it to manage three hundred improvements, decide what mattered, and rewrite my agents accordingly is not a reasonable thing to ask. That’s not an AI limitation. That’s me expecting too much.

The Fix Is a Routine, Not a Feature

The answer wasn’t a better triage system. The answer was to stop letting the backlog build up in the first place.

So it’s part of my routine now. If I’ve let something run overnight, I go through yesterday’s improvements the next day. Five, ten, fifteen improvements is a job I can do properly. Three hundred is not a job at all, it’s a wall.

And once you frame it that way, the shape of the commitment becomes obvious.

I don’t need to be present for every task. Tasks can run overnight. That’s fine, that’s what the harness is for.

I need to be present for the improvements, and for the changes that go back into the agents and skills. That’s the part that isn’t delegable, because that’s the part where the system becomes more like me or less like me.

Which takes me right back to where the first article ended.

I built this as an extension of me. Not a replacement for me. Every time I’ve forgotten that distinction, the system has reminded me, usually expensively.

Next time I’ll get back to the improvement loop and Bishop’s memory properly: what the SQL-based version looks like, why it’s easier to work through daily, and what I keep versus what I let go.