When Documentation Lies

The Gap Nobody Admits To

Every engineering team I've worked with or evaluated has had the same unspoken problem: their documentation is behind their system. Not by a little - by enough that reading the docs and understanding the system are different activities.

Nobody talks about this openly. It's not that teams want their documentation to be wrong. It's that documentation doesn't have a user who complains when it's stale. Code has users - other engineers, QA, end users - who notice when it's broken. Documentation only gets noticed when someone needs it and finds it wrong.

This is the fundamental problem with documentation as a software artifact: it has no runtime feedback loop. The system evolves. The documentation doesn't. The gap grows.

---

What "Documentation Drift" Actually Looks Like

Documentation drift isn't dramatic. Nobody sits down and decides to write misleading documentation. It happens through a series of small, individually reasonable decisions.

The "we'll update it after launch" decision. A feature ships. The docs don't. Someone says "we'll update the docs later" and "later" means "when someone asks" or "never."

The architectural decision that doesn't propagate. A service that was synchronous becomes asynchronous. The API contract changes. The implementation is updated. The documentation describing the expected behavior is not.

The inheritance problem. A system is built by team A. Team A rotates out. Team B inherits the system. Team B updates the code to match new requirements. The documentation stays with what Team A wrote.

The "it's obvious from the code" assumption. The engineer who wrote the code knows why it works that way. They skip the documentation because the code is self-explanatory - to them, today, while the context is fresh. Six months later, the code is still there. The context is gone.

The pattern I see most often in audits: documentation that describes the system as it was, not as it is. Not wrong in an obvious way - wrong in a subtle way that misleads you precisely when you're trying to understand something unfamiliar.

Why This Is a Production Hazard

Documentation drift becomes a production hazard when you need to understand a system under pressure.

An incident happens. The on-call engineer reaches for the runbook. The runbook describes a system that doesn't match what they're seeing. They make decisions based on a mental model that's wrong. The incident gets worse.

A new engineer joins the team. They read the architecture document. They form expectations about how services interact. They make changes that break assumptions the documentation encoded but the code no longer reflects.

A migration is planned. The migration guide describes the current state. The actual current state is different. The migration fails in ways that wouldn't have happened if the documentation had matched reality.

The common thread: in high-pressure or high-stakes situations, people reach for documentation. When documentation is wrong, it misdirects precisely when it matters most.

---

What I'd Do Differently

Make documentation part of the definition of done

The PR doesn't merge if the documentation isn't updated. This sounds bureaucratic. It's also the only thing I've found that works.

When you make documentation part of the merge criteria, something interesting happens: engineers start writing less documentation because they have to maintain it. They become more selective about what they document. The documentation that exists is more likely to be accurate because it was updated at the moment of change, when the context was fresh.

Treat documentation like code review

Documentation should go through review the same way code does. Not for style - for accuracy. Someone should read the documentation change and ask: does this match what the code actually does?

This doesn't mean every documentation change needs a two-hour review. It means the PR that changes behavior should include a documentation update, and that update should be reviewed by someone who can verify accuracy.

Add runtime verification to critical docs

For critical systems, I've found it useful to add automated checks that verify documentation against reality. If your documentation says the timeout is 30 seconds, your integration tests should assert that the timeout is 30 seconds. If the code changes the timeout to 60 seconds, the test fails and someone has to decide: update the code or update the docs.

This is a form of executable documentation - documentation that enforces itself through tests. It doesn't scale to all documentation, but for API contracts, SLA definitions, and architectural constraints, it's more reliable than hoping someone remembers to update the docs.

Audit documentation like you audit code

Most teams have processes for code quality - linting, testing, review. Documentation doesn't get the same treatment.

I've found it useful to do a documentation audit as part of incident post-mortems — the approach I document in auditing a production AI platform. Not "did we update the docs?" as a blame question, but "where did our documentation mislead us during this incident?" as a learning question. This surfaces drift in a context where the cost is concrete and visible.

---

What I've Learned About Writing Documentation That Stays Accurate

Write less, but what you write, write precisely. Comprehensive documentation that is wrong is worse than minimal documentation that is accurate. Every sentence you write is a sentence you commit to maintaining.

Document decisions, not implementations. The implementation changes; the decision rationale is more stable. "We chose Postgres over MongoDB because we needed join support and transactions" is more durable than "we use Postgres."

Make documentation ownership explicit. Every significant system should have a documentation owner - not someone who's responsible for writing all the documentation, but someone who's responsible for ensuring it stays accurate. When no one owns the docs, no one updates them.

Use documentation as a forcing function for uncertainty. When you can't document a system's behavior clearly, that's often a signal that the behavior itself is poorly defined. Ambiguous documentation often points to ambiguous requirements.

---

Final Thoughts

I've worked in systems where the documentation was so accurate that I could trace every behavior to a written description. I've also worked in systems where the documentation was so wrong that the only reliable source of truth was the code itself, read carefully.

The gap between those two situations isn't primarily about effort. It's about incentives. Code has tests that fail when it breaks. Documentation has no equivalent feedback mechanism. The teams that maintain accurate documentation are the ones who built that feedback mechanism deliberately - through review processes, through ownership assignments, through executable documentation that enforces itself.

Documentation is not a first-class engineering artifact. It doesn't get tested, it doesn't fail CI, it doesn't have incidents when it's wrong. The only way to keep it accurate is to treat it as if it does.