I built a skill, /compound, to answer one question about my own AI practice: is this actually getting better, or is it just getting bigger? Eighteen times over five months I ran it. Seventeen of those checks told some version of a story I could live with — hardening, flowing, occasionally cautious. The eighteenth told me a fix I’d shipped six days earlier had failed. The exact number the fix was built to bring down had gone up instead: 18.3 hours a day to 20.65.
I almost logged it as the finding.
The Friction
Three consecutive checks, spanning eleven days, had named the same problem: my logged hours per day were climbing fast — 10.8 in early July, 17.8 by the 29th, 18.3 three days after that. Each check pointed at the same likely cause: I was working multiple projects at once, and the old time-tracking method was crediting each one the full session length, not a fair share of it. Spend one hour rotating across three open projects, and the old method logged three hours of work, not one.
I didn’t build the fix the first time I was told to. Of those three checks, two told me directly to do something about it: the first of the two recommended it, and I let it sit. The second named it, unbuilt, as the single highest-leverage thing on the board — overdue against the window I’d given myself to build it. It took that same instruction landing twice before I actually did anything: a new way of tracking time that counts actual hours spent, not hours claimed. If I worked three projects in parallel for an hour, that’s one hour total, split across the three — not three hours stacked on top of each other.
It shipped August 3rd. Six days later, the eighteenth check ran its usual 30-day average and concluded: “the fix built to address it didn’t reverse the trend.”
It read like a real finding. It wasn’t one, and it took about ten minutes to find out why.
The Build
The eighteenth check’s average covered the 30 days ending August 9th. But the new fix only started recording data the day it shipped — it has no history before August 3rd. So of the 30 days in that average, 24 of them happened before the fix even existed. Averaging a six-day-old fix against a month that’s 80% unaffected by it, then calling the result proof the fix failed, isn’t really measuring the fix. It’s measuring the calendar.
The second problem was worse, because it wasn’t new. Back in July — before this fix was even built — a separate piece of research inside the same practice had already run these exact numbers. Logged hours had climbed from 10.4 a day in February to 18.8 a day in July, while the actual number of hours in a day, obviously, never changed — it stayed flat at 11 to 15 hours the entire time. The conclusion, written down in July: the old counting method inflates in proportion to how many projects you’re juggling at once. It measures the tool, not the person. That conclusion was already sitting in a file, correct, three weeks before the eighteenth check cited the exact number that file had already debunked — and used it to call a real fix a failure.
The check itself was the problem. Nothing in how it worked ever told it to look at the new, corrected data. It had access to the file. It just never occurred to it to check.
I fixed it the same day: the pace calculation now uses the accurate, new data for any date it covers, and when a comparison spans the switch-over point, it reports the before and after separately instead of blending them into one misleading average. Then I re-ran it to see what the real number actually was. Not 20.65 hours a day. Not even close: 7.46 to 8.17, depending on how you draw the boundary of a still-in-progress day.
The gap between those two numbers has a mechanism behind it: the corrected data also reports how many projects I’m typically touching within a single hour of work, averaged across the week — 6.77. That’s a week-long average, not a precise multiplier for this specific gap; some hours were more fragmented than others. But the direction holds. The old method was crediting one real hour to every open project at once instead of splitting it, and 6.77 is the shape of why.
The pace itself turned out to be ordinary. Whether touching seven things in a given hour is a good way to work is a different question — the corrected number doesn’t settle that, it just stops answering the wrong one. What it settles is narrower: whatever the real cost of that pace is, it isn’t 20.65.
The Insight
None of this would have been catchable at all under a different design choice /compound made on day one and never revised: it refuses to collapse seven separate readings into a single score. That decision hasn’t changed once across eighteen checks and five months — it’s why a bad number in one dimension could be isolated and fixed instead of disappearing into an average that still would have looked fine. One of those seven dimensions exists for exactly this purpose: checking whether past decisions get remembered and used at the right moment, instead of sitting, correct and ignored, in a file somewhere. That’s the part that should have caught this. It didn’t, because nobody had pointed it at itself.
Call the gap Instrument Lag: the period in which a corrected measurement exists, but downstream reports, dashboards, or evaluators keep consuming the superseded one. For as long as that gap holds, the old number gets cited as if it were still true, and every conclusion drawn from it inherits the staleness without announcing it. The fix isn’t finished when it ships. It’s finished when the things measuring you know it exists.
This is a different failure than one this practice has already named. A defect nobody has any way to notice — indistinguishable from a clean run until someone happens to check — is Detection Debt. This wasn’t that. The file with the right answer already existed, was already correct, and had existed for three weeks. It just wasn’t consulted. Detection Debt is what happens when no check exists at all. Instrument Lag is what happens when one does, and it’s still reading last month’s version of the truth.
This is also a narrower claim than “the system lied.” The system didn’t lie. Every number in the eighteenth check was computed correctly from the file it was told to read. The failure sat one layer up, in the decision about which source the check should trust — and nothing in the system forced that dependency to be revisited when the source changed.
The Honest Part
The tool didn’t catch this. I did. Nothing in how /compound runs — including the part built specifically to catch exactly this kind of gap — flagged that it was reading bad data, until I said something. That’s a real limitation, not a technicality: a self-grading system that only corrects itself when a human happens to remember a three-week-old document hasn’t actually closed the loop it claims to close.
I added a rule to prevent this going forward: before finalizing any finding, identify any prior diagnosis, metric redesign, or source change that would make the finding invalid, and reconcile the conflict before reporting it. But that rule now sits in the same file that just proved rules like it don’t enforce themselves automatically. Whether it holds is not yet demonstrated. It’s stated.
The same tool has another part built to catch exactly this kind of self-neglect: a direct question, every run, about whether I can still explain my own system from memory. I don’t think that’s a good question, and I said so the same day. I don’t carry my own skill list in my head, and I don’t need to — I keep an actual reference panel for that. Checking that panel against what’s really installed, instead of quizzing my memory of it, is what surfaced something real: four skills missing from the panel, one command listed under the wrong name entirely. The fix for a broken self-check wasn’t answering it more diligently. It was replacing the question with the thing I actually use to answer it — which means the mechanism built to catch me neglecting the system was, itself, testing the wrong thing. Same day, same practice, one level up from the first bug.
I still don’t know how many other numbers or checks in this practice are quietly testing the wrong thing right now. Both of today’s — the hours, the skill quiz — surfaced because I happened to look closely at something specific. Nothing about this system guarantees a third one gets found the same way.
That’s the broader exposure. Any dashboard, eval harness, or self-monitoring system can correct a measurement in one place while continuing to consume the invalidated version somewhere else. The fix and the audit of the fix are separate dependencies — the underlying system can be correct while its own reports keep describing the old one. Unless the system reconciles them explicitly, they drift.
Case Study Insight: A measurement fix isn’t complete when it ships. It’s complete when the gap this practice now calls Instrument Lag — where a corrected measurement exists but nothing downstream has started reading it — closes. Until then, the discredited number keeps grading your work, correctly computed and quietly wrong.
Robert Ford builds products, writes stories and essays, and publishes The Intelligence Engine, a practitioner research publication about AI systems that compound. His other writing lives at Brittle Views.
How this was made: drafted in working sessions with Claude, revised across multiple rounds I read and scored myself. The judgment — what’s true, what’s cut, what ships — is mine throughout, including this line.