The content defect no article-by-article review could see
We built a library of 56 articles with AI assistance. Every one passed review. Most of them were, individually, good.
We only found the problem because we went looking for something else. We wanted to pick a handful of the articles as style exemplars, samples an automated system could learn from so that it would write in the range we write in. When we went to select for range, the range wasn’t there.
Forty-six of the fifty-six ended on the same closing section. The same forty-six sat on the same skeleton: an opening that framed the problem, a bulleted named artifact, a numbered process, a closing operating rule. In one batch of twelve articles, we could find only two distinct structures.
Nobody had been careless. Every article had gone through the same review, and the review had done what it was built to do.
The review was answering a different question
Each article had been checked against a rubric: is the claim supported, is the argument sound, is the central artifact named and usable, would a reader make a better decision after reading it. Every answer was yes, and those answers were correct. Each review was an accurate report about the article it was looking at.
The uncomfortable part is that the rubric already contained a rule against exactly this failure. “Same outline repeated mechanically” was listed as an automatic failure. A separate rule prohibited the same multi-step framework appearing in consecutive posts. Both rules existed before the first article was written, both were enforced, and both failed.
They failed because “repeated” is not a property of the article in front of you. A reviewer holding one piece cannot see an outline being repeated, however carefully they read, because repetition only exists between items. The rule itself was right. We had installed it at the level of the single article, where it could never trigger.
Generating more variants does not fix it
Our process already had what looks like a defence. For each concept we generated three variants and scored them against each other, then published the winner. That feels like it should produce variety, and in a narrow sense it does. The three variants did differ, in their openings, examples, emphasis and quality, so choosing between them was a real choice.
But every one of those comparisons was local. None of them asked whether the winner resembled the last twenty winners.
Three variants can differ from each other in every visible respect and still be three members of the same structural family. If that family is the one the generator reaches for by default, then scoring the variants just picks the most polished expression of the drift. We were running a contest to find the best version of the same shape, twenty times in a row, and recording each result as evidence of quality.
Variation within an item and variation across a corpus are different measurements, and no amount of the first approximates the second. This surprised me. I had to see it in the data before I believed it, and it is what the whole exercise taught us.
The variety was real, which is why it hid the problem
The output was not low-effort either. The named assets, meaning the central device each article hands the reader, really did vary: fourteen distinct nouns across the library, including Contract, Card, Map, Envelope, Register, Chain, Loop, Stack and Gate. The topics, the arguments and the examples varied as well.
That surface variety is what made the defect invisible. Put any two articles side by side and they read as different pieces of work. Read twenty in sequence and you feel the shape before you can name it.
This is where I think the commercial risk sits, and this part is my inference; we did not measure it. A first-time reader almost certainly will not spot the template. The risk is that repeated exposure makes the body of work feel manufactured even when no individual article does, and the reader who keeps coming back is usually the one deciding whether to hire you. I have no data on that reaction and I am not going to invent it.
Still, that reading is what makes this worth fixing instead of filing under untidiness, and nothing in a per-article review process looks at that reader.
This is not a prompt problem
The obvious reading is that we needed better prompts, more explicit instructions, a demand for structural variety in the brief. We tried versions of that, and it treats the symptom.
The failure lives above the level of any individual instruction. You can write an excellent brief, get an excellent article, review it honestly, approve it correctly, and repeat that forty-six times into a body of work that reads like a template. In our case every local step was competent and the output of the system was still wrong. If your mental model is “quality is what happens when each piece is good,” you cannot see this failure mode from anywhere you are standing.
What we measure differently now
We have changed where we look for this failure. We have not yet shown that we have fixed it, and I want to keep those two claims apart.
What changed is the level. Our review was already answering is this good correctly, so we now also ask of every finished piece what structural family does this belong to, and how many of the recent pieces belong to the same one. That is a comparison between items, and nobody can make it while holding one.
We have also stopped arguing with the drift at the end of the pipeline. If the generator reaches for one structure by default, the scoring step is the wrong place to correct it, so the shape has to be decided where the piece is planned, before anyone judges it. We do not yet know whether that widens the distribution.
The next batch is the test. Until it exists and has been measured the way the first one was, by counting distinct structures across the set instead of quality within each piece, all we have done is move where we are looking, and a changed process is not evidence of a changed outcome. That exact gap has already caught us once in this same body of work: the anti-repetition rule was real, it was enforced, and the library still came out the shape it came out.
The general shape of this mistake
The principle underneath is short. A local quality check cannot detect a property that only exists at the level of the set. Each article passed a real test, and the set failed a test nobody ran. There was no dishonesty or shortcut anywhere in the process, and no rule was missing. The check simply ran at the wrong level.
I will leave the wider applications to the reader, because content is the case I have evidence for. If you produce anything with AI at volume (articles, product descriptions, outreach, landing pages), the item-level record can be excellent while the aggregate turns into something nobody would have chosen. It is hard to catch because every individual answer defends itself, and every individual answer is right.
Asking is this good? will not find it. The question that finds it is what does this become, forty-six times?
