From Typist to Director

I used to do test-driven development with Claude. I’d write the test, it would write the code, and we’d go back and forth until it passed. Somewhere along the way I stopped.

I never decided to stop. I started using a task tracker I’d built for my agents, and the tests just fell off. Before that I had two scripts, dev.sh and prod.sh, that told an agent what state the app was in so it didn’t have to guess. Those fell off too.

This morning I looked at the marketing site for Matchstick, the trading replay app I’m building. It’s a marketing site. It has a waitlist form. It has 18 test files and about 3,300 lines of tests. The unit test for the waitlist form alone is 409 lines.

The repo root had nine Markdown files with names like REFACTORING_SUMMARY.md and SMOKETEST_RESULTS.md. I didn’t write any of them. An agent wrote them to show its work, and then it left them there.

The app itself has about 85 test files and roughly 20,000 lines of tests. I can’t tell you how many of those lines protect anything.

That’s the problem.

Why agents do this

An agent that writes code and then tests that code is grading its own homework. The easiest way to pass is to write tests that can’t fail. So that’s what it does.

Once you look, you see the patterns. A test that computes the expected answer by calling the same function it’s testing. A test that checks a string is a string, which TypeScript already checks. A test that breaks every time you rename something, because it checks how the code works instead of what it does.

The deeper issue is that one model wrote both halves. If it misread the requirement, the code is wrong and the test agrees with it. The green checkmark only means the agent agrees with itself.

I’ve started calling it test slop. The volume isn’t the worst part. The slop is fragile and misleading, and it buries the few tests that matter.

Flip who writes the tests

Wes Bos talked about this on the Syntax podcast. For complex logic, he writes the unit tests first, by hand. He was building a parser for social media URLs, so he wrote an expect() for every URL format he wanted to support. Then he asked the AI for a function that passes them all.

That’s the whole trick. The judgment lives in the assertions, and the human writes them. The agent does what it’s good at: grinding until a fixed target turns green.

It’s also what I used to do, before I stopped.

Build the gauntlet instead of reading every line

Writing every assertion by hand is its own grind. Reading every line an agent writes is worse. It throws away the speed that made the agent worth using.

Robert C. Martin, Uncle Bob, has said he no longer reads the code his agents write. He surrounds them with strict constraints instead. He built SwarmForge, a pipeline that hands work between specialized agents: a specifier, a coder, a cleaner. He backs it with automated gates like complexity limits, module size limits, and mutation testing.

Mutation testing is the one I’d steal first. It breaks your code on purpose. It flips a comparison or deletes a return, then runs your tests. If the tests still pass, they weren’t testing anything. I don’t know a more honest answer to “are these tests real?”

Don’t overbuild it

There’s a limit. Recent research found that automatically generated multi-agent setups often do worse than a single model reasoning carefully, at up to ten times the cost. More agents means more handoffs, and every handoff is a place for things to go wrong. I’ve built some of that scaffolding myself, and I’ve taken some of it back down.

Linus Torvalds keeps it simple. Teams that brag that 99% of their code is written by AI could just as well say 100% of it is written by compilers, he’s pointed out, and nobody says that. Compilers made programmers something like a thousand times more productive. AI adds maybe ten.

The Linux kernel now has written guidance for AI coding assistants, proposed by maintainer Sasha Levin. A patch made with AI help carries an “Assisted-by:” tag, and an AI agent can’t add a Signed-off-by. Only a human can certify the patch. So the person who submits it owns all of it. The policy doesn’t move that responsibility anywhere. It pins it on the person.

What I’m doing now

Here’s the rule I gave my agents today:

• Only commit a test if leaving it out could let a real security, money, or data-loss bug ship unseen.
• Don’t write unit tests after the code to prove it works.
• For a complex feature, write one end-to-end test for the happy path and one for the sad path.
• Write unit tests only for real edge cases.
• Keep scratch scripts and notes out of the repo.

For Matchstick, the tests that matter are around licensing and payment, saving a replay session, and validating imported market data. Those are the places where someone loses money or loses their data. Everything else has to earn its place.

I didn’t delete anything today. I had agents sort every test file into keep or cut, with a reason for each. I’ll make the cuts myself.

The job used to be measured in lines of code. That measure is gone. What’s left is taste: which edge case deserves an assertion, which test is worth maintaining for years, and whose name is on the commit.

Mine is.