On August 19th, I read a document my own software had generated — a handoff packet, the kind a family gives to an elder law attorney or a geriatric care manager when they need a stranger to understand a situation quickly.
It was branded Helper Compass. Seven times: the header, the footer, the “Prepared by” row, and the introduction written for each of the four recipients — attorney, care manager, physician, facility.
Helper Compass is not a product. It was never proposed, never discussed, never approved. An agent invented the name during an earlier build, and it went into the documents families hand to professionals who are deciding whether to take a case. A lawyer receiving that packet has no way to know which parts of it are real. The credibility of every other field on the page — medications, dates, decision history — rests on the reader assuming the software knows what it is talking about, and the letterhead was fiction.
Nothing caught it. There was no gate. I caught it because I happened to read a packet.
Seven weeks earlier, in a different workspace — a line of small-business AI guides with no relationship to eldercare, no shared code, no shared customer — I had found internal build metadata sitting inside 18 of 23 shipped guides. Audit scores. References to a payment processor we had discontinued. An instruction written to an AI assistant, published as customer copy.
That one had also been caught by accident: a branding check that happened to widen its scope, not by anything designed to look for it.
The fix I built that day was a function called lint_modules(). It scans the literal text about to ship for internal-only vocabulary and refuses to emit output on a match. Not a warning — a refusal.
When I found Helper Compass, that July fix surfaced, and the response changed shape. Not fix the packet, which I would have done anyway, but this lint belongs anywhere generated text ships, which is a roadmap item rather than a repair.
I want to be careful here, because this is the example that proves the least.
Both defects are the same on their face: wrong internal text reaching a customer. Any competent similarity search over my own notes would surface the July incident from the August one, because the two descriptions share most of their vocabulary. Nothing in that connection requires a system that had generalized anything. It requires a system that can match “text that shouldn’t have shipped” to “text that shouldn’t have shipped.”
So the vivid case is the weak case. The one that matters is the one I nearly skipped.
The same packets had a second flaw.
Where the care recipient’s name should have appeared, the document printed the words Care recipient. The database column that holds that name is never populated, so the template fell through to its label.
The connection that arrived for that one came from a rule I wrote in July about a marketing homepage displaying hardcoded course counts — a number stubbed in during a build and never wired to the database, sitting there looking authoritative and being wrong.
Consider what those two things have in common at the surface. One is an integer on a web page in a course catalog. The other is a string in a PDF in an eldercare product. Different data type, different domain, different product line, different failure symptom. Search the text of one for the vocabulary of the other and you get nothing. There is no shared phrase, no shared component, no shared table, no shared customer.
What they share is only this: a specific, plausible-looking value standing where unknown information should be, in a place where an honest blank would have been better. That is not a similarity between the two defects. It is a category that both of them are members of, and it exists nowhere in either defect’s own description.
That is the whole finding, and it is narrower than the one I wanted.
Matching on surface gets you the lint. Matching on abstraction gets you the placeholder — and the placeholder connection is the one I would never have made on my own. I found the defect; anyone reading carefully would have. What careful reading of the packet does not supply is a marketing page in a different product line. Nothing in that document points there.
Which means the write-up discipline is not administrative overhead sitting on top of the practice. It is the part that determines reach.
A defect written up in the language of its own project — the course count on the homepage is wrong — gives a future search very little to work with once the surface vocabulary stops overlapping. The same defect written up as a class — a plausible-looking specific standing in for unknown information — is available to every future project, including ones that do not exist yet, whose vocabulary need not overlap with the original incident at all.
I want to be exact about the size of that claim, because it is easy to inflate. I am not saying a project-specific record is unretrievable. A sufficiently capable search might infer the latent similarity between two detailed incidents without either one having been generalized first. What I can say is what this architecture did: precomputing the abstraction at write-up time produced a match that I have no evidence would have arrived otherwise, in a case where the surface cues were gone entirely. That is a finding about how this system behaves, not a law about memory.
I did not do it on purpose in July. I wrote the general version because the specific version felt too small to be worth recording — an instinct that happened to produce the useful behavior for a reason unrelated to why it was useful.
This is also where I should say what the system did not do, because the flattering version is wrong and it is wrong in a specific way.
It did not find either defect. A person reading a document found both. What arrived afterward was the category — this is a member of a class you have already solved — and the category is what changed the size of the response.
Detection and classification are separable, and they failed and succeeded independently here. That is not a rhetorical distinction; it is the difference between two operations of the same system, one of which was absent entirely.
Then there is the other direction, where the system participates in detection — and its limits there are just as sharp.
On August 12th, I published a case study naming Instrument Lag: the period in which a corrected measurement exists but downstream reports, dashboards, or evaluators keep consuming the superseded one.
One week later I was recalibrating the gate I use to evaluate my own published pieces. The recalibration required recomputing each recent post’s trailing-six median from scratch, and that recomputation is what exposed the defect: one gate close had reused a threshold carried forward from the previous post instead of recomputing it. The bar was wrong by roughly 12 views. It had gone unnoticed for three weeks.
That is Instrument Lag, in the instrument, seven days after publishing the piece that named it.
The same session turned up something larger. The gate had returned FAIL on all five closes since it went live in July. Five runs, no passes. Five failures do not by themselves prove a broken test — a genuinely hard bar can be missed five times — but they do establish that the gate had never once demonstrated it could discriminate, and I had spent a month reading its verdicts as though it had.
Neither of those was mine. I did not read a table and notice. An agent working the gate-close procedure found both, in the course of correcting a figure it had given me an hour earlier.
And the causal chain matters more than the anecdote. The audit did not run because the vocabulary existed. It ran because a recalibration was underway that forced every number to be recomputed.
What followed the finding was a mechanical guard: the trailing-six values and the computed median now have to be written out at every gate close. When the working is shown, a reused number is visible. When only the verdict is recorded, it isn’t.
That guard follows from finding the reuse. It does not require the concept. So I should be precise about what the seven-day-old vocabulary actually contributed here, which is less than the placeholder case: it let me file the defect as a member of a class I had just published rather than as one wrong number in one row. That changed how the finding was understood, not what I did about it. The naming did not cause the detection, and this time it did not change the response either.
The Honest Part
The system is blind outside the places I have already built instruments, and that blind region has a name — Detection Debt, the liability that accrues from defects for which a system produces no signal at all: no failed check, no flag, nothing that looks wrong. Silence that reads as correctness until someone goes looking without a specific reason to.
Classification does not touch Detection Debt. It operates on findings, and a finding has to exist first. The audit could examine the gate because the gate was instrumented. Nothing was instrumented around the packets, so nothing could have caught Helper Compass, and nothing did.
The boundary is worth drawing exactly, because it is not that customer-facing output is inherently blind — the guide pipeline has a lint precisely there. It is that this product’s shipped output had no instrument, and the reason is ordinary: the guides pipeline got one after being burned, and the eldercare packets had not been burned yet. Detection Debt sits wherever an instrument has not been built, and instruments get built where something already went wrong. The debt is therefore concentrated in exactly the places with no incident history — which look, from the inside, like the places that are fine.
And the more fluently a system explains what you found, the easier it becomes to read explanation as coverage.
The classification layer is also confident, and the lint case shows what that costs. A surface match — two incidents sharing most of their vocabulary — produced a roadmap item: this check belongs anywhere generated text ships. That may be right. But the scope of that decision was set by a category I never tested the second case against.
I have a guard for the promotion half of that problem. The Second Build Test says a pattern stays provisional until it survives a second, independent application — domain shift, intent independence, input variance — and the placeholder case cleared all three. But the test only governs whether a second instance licenses generalizing. It says nothing about whether the match was correct in the first place, and I do not have an instrument for that. I have not found a way to distinguish, in the moment, between a category that genuinely contains the new case and one that merely accommodates it. Both feel like recognition.
And the timeline is longer than the pitch suggests. The early connections were pleasant and mostly decorative. The ones worth having required that the intervening write-ups be done as classes rather than as incidents — which is more work at exactly the moment the incident is closed and I want to be done with it. I skip it when I am tired. Every skipped one is a future match that silently does not happen, and there is no signal for that either.
So: what does compounding feel like?
It does not feel like speed. It feels like being handed the category.
You find the thing. Then something supplies the sentence that says this is a member of a class you have already solved, and here is what you built last time — and the repair you were about to make turns into infrastructure you should have generalized weeks ago.
The system is not smarter than me. It keeps a record across a longer interval than my attention covers, in a form that survives my forgetting what the connection was for.
Whether the category it hands me is the right one is the part I still cannot check.
How this was made: drafted in working sessions with Claude, revised across multiple rounds I read and scored myself. The judgment — what’s true, what’s cut, what ships — is mine throughout, including this line.
Robert Ford builds products, writes stories and essays, and publishes The Intelligence Engine — a practitioner research publication about AI systems that compound. His other writing lives at Brittle Views.