---
title: "Introducing Comprehension Coverage"
url: https://tiarebalbi.com/en/blog/introducing-comprehension-coverage
markdown: https://tiarebalbi.com/en/blog/introducing-comprehension-coverage.md
description: "Authorship stopped implying understanding. Comprehension coverage: a per-module, evidence-based, decaying map of what your humans still understand — and why."
author: "Tiarê Balbi Bonamini"
locale: en
published: 2026-09-21
updated: 2026-09-21
category: "AI"
tags: ["ai-development", "comprehension-coverage", "engineering-practices", "architecture", "agents", "concept"]
translation: https://tiarebalbi.com/pt-br/blog/introducing-comprehension-coverage
---
# Introducing Comprehension Coverage

Every codebase carries two maps. The first shows what the test suite verifies — measured to two decimal places, tracked in CI, argued about in every engineering organization on earth. The second shows what the humans still understand — and nobody measures it at all.

For twenty years, nobody needed to. The second map came free with version control: whoever wrote the code understood it, so `git blame` doubled as a map of understanding. That assumption quietly carried every knowledge metric the field ever built. And in the agent era it is simply false. The person who "authored" three thousand lines last month may understand all of them, some of them, or none of them.

So here is the concept this post introduces, the one the last six posts in this series have been walking toward: **comprehension coverage** — a per-module, per-person, evidence-based measure of which humans still understand which parts of a system, decaying over time unless re-evidenced. Test coverage for the only component that cannot be regenerated: the understanding in your team's heads.

## A theory, not a text

This started as background reading for a tool I am building, and the reading kept pulling the thread further back than I expected — all the way to 1985. Peter Naur's essay [_Programming as Theory Building_](https://gwern.net/doc/cs/algorithm/1985-naur.pdf) argues that a program is not primarily its text; it is the theory held by the people who built it — the ability to explain why it is shaped this way, to map the world onto the code, to see which changes fit and which fight the design. The text is a byproduct. And Naur is brutal about what happens when the people go: a program dies, he writes, "when the programmer team possessing its theory is dissolved." Rebuilding the theory from documentation alone, he argues, is strictly impossible — his advice for a revived program with no surviving theory-holders is to discard the text and solve the problem fresh.

Reading Naur sent me looking for the empirical side: how fast does the theory actually rot? The best study I found is Krüger et al.'s "Do You Remember This Source Code?" (ICSE 2018), which measured how quickly open-source developers forget their own work: half of their own files, within 30 days, median. And the detail that stopped me: what predicted retention was _doing_ — the number of own commits correlated strongly with remembering (ρ≈0.67) — while merely _tracking_ other people's changes to a file predicted essentially nothing (ρ≈0.04, not significant). Watching code change does not build the theory. If that finding sounds familiar, it is the [spectator trap](https://www.tiarebalbi.com/en/blog/spectator-trap-staying-in-control-ai-development), measured in a lab seven years before agents made it a lifestyle.

Why does the theory matter economically? Because comprehension is most of the job. [Xia et al. (IEEE TSE, 2018)](https://ieeexplore.ieee.org/document/7997917) instrumented professional developers at work and found roughly 58% of development time goes to program comprehension. Reading is the work. The theory is what makes reading fast.

## The proxy that just broke

Since the theory itself is invisible, the field spent two decades measuring its shadow: authorship. Fritz and Murphy's degree-of-knowledge models scored developer–file familiarity from who created and changed what. The truck-factor algorithms were built on top of it — [Avelino et al.](https://homepages.dcc.ufmg.br/~mtov/pub/2016-icpc.pdf) found 65% of the GitHub systems they studied rested on two or fewer key developers. CodeScene's knowledge maps industrialized the same signal. Even GitHub's CODEOWNERS, which many teams treat as a knowledge registry, turns out to be something else entirely: a [recent study](https://arxiv.org/abs/2512.05551) of the most-starred GitHub repositories found that 79% of the individual owners listed were not among their repo's top hundred committers — it is a review-routing layer, a claim of responsibility, not evidence of understanding.

All of it — every metric in that family — rests on one axiom: writing code is evidence of understanding it. That axiom is what agents deleted. A June 2026 preprint by independent researcher Brett Wheeler, [_The Substrate Collapse_](https://arxiv.org/abs/2606.20882), makes the argument with a precision I have not seen elsewhere: once code arrives through an agent, the authorship footprint becomes compatible with full understanding, partial understanding, or none — so no reweighting of authorship data can recover the inference, because the failure is structural, not a calibration problem. And this is not hypothetical drift: a [2025 study](https://arxiv.org/abs/2507.08160) simulating 50% generative-AI involvement found 73% of truck-factor values changed across their projects. The shadow-measurement era ended; the tools just have not noticed yet.

I want to be careful about what Wheeler's paper is and is not: it is a single-author preprint, uncited so far as I can find, and it deliberately proposes nothing — it diagnoses, stakes a falsifiable prediction, and stops. But its closing gap statement is the sentence that turned my reading into a project. What is missing, he writes, is the comprehension instrument "at the scale of a whole system and a whole team, continuous and unobtrusive." That is a specification. Someone should build to it.

## The concept: comprehension coverage

Comprehension coverage is my attempt at that specification, and it stands on five design commitments.

**Evidence, not authorship.** A line in the map is earned only by acts that plausibly build or demonstrate theory: authoring by hand, reviewing deeply (not rubber-stamping — the distinction the [review-bundle post](https://www.tiarebalbi.com/en/blog/review-bottleneck-unbundling-code-review) was about), diagnosing an incident in that module, writing the ADR, explicitly re-walking the code. Merging an agent's diff unread earns nothing. Watching tokens stream earns nothing — Krüger's ρ≈0.04 says so empirically.

**Decay by churn, not by calendar.** Krüger measured forgetting against time, and time matters — half the familiarity gone in weeks, not years. But the sharper clock for a codebase is change: comprehension of a module that has been rewritten twice since your last real contact is stale regardless of the calendar, and comprehension of a frozen module survives long quiet periods. So the score decays as a function of churn since the evidence, with wall-clock decay as the floor. This is the design decision I most want challenged, because it goes beyond what the forgetting literature directly measured.

**Per module, mapped to the architecture.** The unit is the same module boundary your architecture tests already know — the slices from the [fitness-function constitution](https://www.tiarebalbi.com/en/blog/fitness-functions-control-plane-agentic-coding). The output is a map: for each module, which humans hold current, evidenced comprehension — and which modules are _dark_: running in production, changing weekly, understood by no one currently on the team. Dark modules are Naur's dead programs that nobody has noticed dying.

**Continuous, not one-shot.** Comprehension has been measured before — researchers do it with quizzes (Anthropic reports a randomized trial where AI-assisted authors scored 50% on comprehension quizzes about their own code versus 67% for manual authors). But a quiz is a snapshot. Coverage is a stream: every merge, review, and incident updates the map, the way every commit updates test coverage.

**Module aggregates are public; individual scores are not.** This is an instrument for locating risk in a system, not for ranking people. The moment individual comprehension scores feed a performance review, Goodhart arrives and the evidence dries up — people will farm attestations the way agents farm test suites. The output that travels is "module X has no current comprehender," never "developer Y scores 0.3."

## What already exists — and what does not

Introducing a concept obliges the search for prior art, and the search changed this essay. Comprehension debt — the liability side of this ledger — was named in 2025 and popularized this year by Addy Osmani. One-off comprehension measurement is an established research method. And one practitioner got closer than anyone: Leonid Sokolovskiy's [_Feature Comprehension Score_](https://mironyx.dev/blog/feature-comprehension-score), published months before Wheeler's paper, generates assessment questions from a feature's own artifacts at ship time and scores the team's aggregate answers — a real, built, comprehension-evidence instrument, with a tool behind it and a stance I share: "This is not a test of individual developer performance."

But the FCS, as I read it, deliberately stops where coverage begins: it fires once per feature rather than continuously over the codebase, aggregates to a single team number, tracks no decay, and maps nothing onto the code itself. So the claim I will defend is the narrow one, hedged exactly as far as my search reaches: as of September 2026, I can find no continuous, per-module, per-person map of comprehension evidence — nothing that does for understanding what `git blame` did for authorship, now that the two have come apart. If someone has built it, I want to know about it more than I want to be first.

## What the map makes possible

The moment the map exists, it composes with everything this series has built. A dark module becomes a routing rule for the control plane: agent changes to it demand a designated comprehender on the review, or are refused entirely. A coverage threshold becomes a fitness function — the human side of the system wired into the same gates as the architectural constitution, so the build can say what no dashboard currently can: _this change is fine, but nobody who understands the thing it touches has looked at it._ Onboarding stops being vibes — assign the new engineer to re-evidence two dark modules and the map shows the lights coming on.

And the concept inherits a validation protocol it did not have to invent. Wheeler's preprint stakes its falsifiable prediction on exactly this instrument existing: systems with healthy authorship-based truck factors but low measured comprehension should suffer disproportionately when a novel incident demands real theory of the system — and if they don't, his argument and this concept both lose. That is a test any team with incident data can eventually run, and I would rather publish the concept with its refutation conditions attached than pretend it arrives proven.

Two honest limits, before the close. Evidence of comprehension is not comprehension — a deep review can still miss the point, and any evidence type I weight becomes a thing that can be performed; the instrument needs the same skepticism about its own gates that the last post demanded of fitness functions. And the ethics are load-bearing, not decorative: this map only stays honest in a culture that treats a dark module as a system risk to fix, never as an accusation against whoever let it go dark.

The next post in this series will be the instrument itself — a small open-source library that builds this map from a repository's own history: evidence extraction, churn-indexed decay, dark-module detection, and a gate mode for CI. It is in progress now — [spec v0.1 and a Python reference prototype are already public](https://github.com/tiarebalbi/comprehension-coverage) — and writing this essay first was deliberate: the concept should stand or fall on its own argument before a single line of the tool biases the judgment. The Substrate Collapse ended with a specification and a dare. This is my answer to the specification. The tool is my answer to the dare — and the map of your own system, when you first render it, will answer a question you may prefer unasked: how much of what you are running does anyone still understand?