The system worked. That was the sentence that ended every conversation about replacing it. The system works. And it did — it processed data, it fed downstream operations, it kept the business running. It also consumed a six-figure annual consulting budget to keep working, depended on a small number of people who carried critical knowledge in their heads rather than in documentation, and produced outputs that routinely required manual validation because nobody fully trusted the results without checking them.
I was brought in as part of a systems integration role at a healthcare benefits administrator — a company managing health plan eligibility and enrollment data on behalf of employer clients. My mandate included understanding the existing data infrastructure and identifying opportunities to modernise it. What I found was a pipeline that had been running for years, doing essential work, and accumulating a kind of invisible debt that only became visible when you tried to change something.
The System We Inherited
The pipeline's primary job was processing enrollment and eligibility data — moving information between employer systems, the benefits administrator, and downstream carriers. This is regulated, sensitive work. Errors don't just produce bad data; they affect people's healthcare coverage. The stakes of the existing system being "good enough" were therefore high, and the bar for replacing any part of it was correspondingly conservative.
The system had been in place long enough that its original architects were no longer on the team. What remained was the running code, a small number of people who had learned to operate it through years of direct experience, and a consulting relationship that covered the gap whenever something broke or needed to change. That consulting relationship was costing the organisation roughly six figures annually — not for new development, but purely for maintenance and incident response on a system that was, in principle, fully owned.
When I mapped out what the consulting spend was actually covering, a pattern emerged. The same categories of issue kept recurring: data format inconsistencies from upstream sources that the pipeline had no formal mechanism to handle, processing logic that had been patched over time without corresponding documentation updates, and validation failures that required manual interpretation to diagnose because the error messages described symptoms rather than causes. The system wasn't failing. It was requiring constant expert attention to not fail.
What Discovery Actually Revealed
The standard framing for a project like this was straightforward: audit the existing system, identify the gaps, build the replacement. What made it harder was that the existing system's behaviour wasn't fully represented in any single document. The code captured what the system did in normal cases. The consultants captured what to do when it didn't. The institutional knowledge held by the remaining team members captured a third layer — the undocumented rules, the field-level exceptions, the cases where the system's output was technically within spec but required interpretation before it could be used.
Discovery involved reconstructing that third layer through a combination of stakeholder interviews, analysis of historical processing logs, and — most usefully — sitting with the people who had been running the system daily and asking them to walk me through specific recent incidents. That process surfaced a significant number of behaviours that weren't documented anywhere. Some were intentional design decisions that had never been written down. Some were bugs that downstream processes had quietly adapted to accommodate, meaning they had effectively become part of the specification. Identifying which was which required more time than the project plan had allocated for it.
The lesson was a simple one with large implications: in a system that has been running long enough and been touched by enough people, the behaviour is the specification. Understanding what you're actually replacing requires understanding what it actually does — not just what the documentation says it does.
The Rebuild Approach
The replacement was built with three explicit goals that the original system hadn't been designed around. First, every processing decision had to be documentable in plain language — not because auditors would read it, but because the next engineer to touch it needed to understand why it worked the way it did without calling anyone. Second, error handling had to be specific enough that a non-specialist could diagnose a failure from the error output alone, without needing institutional context. Third, upstream data inconsistencies — which had been the single largest driver of consulting spend — had to be handled within the system rather than escalated to manual intervention each time.
That third goal required the most significant rethinking. The original pipeline had been built with the assumption that upstream data would conform to spec. In practice, it never reliably did. Each inconsistency that fell outside the expected format triggered a manual process. The rebuild treated format variation as a known condition to be handled gracefully, not an exception to be escalated. That single change — designing for the data that actually arrives rather than the data that's supposed to arrive — accounted for the majority of the reduction in recurring support costs.
What Changed, and What It Cost
Within the first full operating cycle after cutover, the consulting call volume dropped substantially. The categories of issues that had driven most of the recurring spend stopped recurring — not because the underlying data quality had improved, but because the system now handled those conditions rather than failing on them. The annual consulting budget was reduced significantly, with the remaining spend covering genuine edge cases and planned enhancements rather than maintenance of known weaknesses.
The less quantifiable change was in operational confidence. The team that had previously relied on specialist knowledge to operate the system could now run it independently. Error messages pointed to specific records and specific causes. Processing logs were readable. The documentation reflected what the system actually did. None of that was technically sophisticated. All of it was consequential.
What It Actually Taught Me
The technical work on this project was not particularly complex. What was complex was the reconstruction of intent — understanding why a system that had been built by capable people had become this expensive to operate, and what decisions, made differently from the start, would have changed that trajectory.
The honest answer is that most of the decisions that led to the eventual cost were individually reasonable. Handling an upstream format exception manually is faster than building a rule to handle it automatically, when you're doing it once. Patching a processing issue without updating the documentation is faster than updating the documentation, when the incident is active and stakeholders are waiting. Each shortcut made sense in isolation. Accumulated over years, they produced a system whose operating cost had quietly grown to absorb most of the value it was generating.
Building for legibility — structuring systems so that future maintainers can understand them without the original architects — is not an instinct that emerges naturally from the pressures of delivery. It has to be a deliberate choice, made early and defended consistently. This project made that case more concretely than anything I'd read about it.
