Maybe you started with a modest CLAUDE.md or AGENTS.md, which you meaningfully crafted over time. Then you adding some core docs (which was best practice anyway), like ARCHITECTURE.md. Before you knew it, you were writing massive specs and generating full-blown features. A lot of those specs shared a lot of context. Never one to miss an optimization, you yanked out the boilerplate and made some Markdown files.

Thus, your /docs directory was born. Not from some strategic plan, but as a natural response to your need for more context. As /docs filled up, you may have even created sub-directories, or a table of context. If you have taken things this far, you are in good company - this is the exact approach OpenAI uses internally.

How we think about and manage context has evolved quite a bit in a few short months. Every step forward has been reasonable and natural, but recognize that your /docs directory is a temporary solution. As you add more docs while changing code, subtle inconsistencies are beginning to pile up. Over time, the context your LLM need to generate code is degrading and there is no easy way to add observability.

Some looming issues to consider if you have a growing /docs directory of Markdown:

• Discoverability: How does an LLM know it should read a specific document? Will it be referenced at the right time, during planning and generation?
• Ownership and domain expertise: If context docs are part of the codebase, who writes and maintains them? If you have a mid-sized company, you probably don't want engineers maintaining docs about DevOps or design patterns. But what if there is a code change that adjusts timeouts or revises a frontend theme. How do you manage this?
• Doc rot: Even well-maintained documentation devolves over time as small mistakes, repetitions and inconsistencies occur between docs. LLMs can help, but if LLMs manage consistency blindly our docs start to lose the plot. "Vibe documenting" might not crash the server, but we have no observability when a doc is wrong. If our context is wrong, code generation degrades with no smoking gun and no clear path to resolve it.
• Velocity mismatch: Context changes at different speeds. Codebase context can change with every commit, but a high-level architecture doc might not change for months. If we have 200 docs, how should we manage docs at different rates?
• Lack of hierarchy: There is no inherent structure connecting or orienting context documents. Documents may be linked conceptually or sorted into directories, but LLMs lack a clear way to navigate documents.
• Bespoke doc structure: Everyone writes context docs in their own way. Your docs may be well structured, but it is unclear if your structure is useful for how an LLM navigates and consumes information.

These are thorny issues. Larger companies may have internal solutions for some of these, but they are non-trivial.

LLMs Won't Save Us

It is easy to turn passive and think the next model might solve context management for us. They are certainly trying to do just that. But context is only adjacent to code generation. LLMs can get better at inferring context from code or using it effectively, but producing and managing context is an entirely different problem.

Context is a blueprint that defines how our product exists (or should exist) in the world. It isn't something a model can generate without us, because the context we give an LLM describe intent or usage, and sets real world boundaries. "Context" is an amalgam of concepts that leads to the code, not something that can be extracted from it. This begs the question:

If context is so important, why are we managing it with a solution straight out of 1985?