A note before we start: the people and organizations in this story are real. The company’s name is not. I’ve changed it to protect their privacy, not to soften what happened.
Setup
My engagement manager called with six words: they want you to build it. Meridian wanted Order Manager, the one-page Visio diagram, with a few paragraphs of context on each component, that I’d attached to a process study nobody had asked me to solve.
That one-page diagram became a 22-month, 20-person engagement. The question it answered was the same one the SVP had never been able to answer accurately in the past: where do we actually stand with our sales, from the moment something gets sold, through order entry, through provisioning, to the moment Meridian starts collecting money on it?
The team
We started small. My engagement manager, a senior manager by then, two technical managers, and me, on the functional side, staring at that one page, figuring out how to make it real. By the time we were done, the team had grown to twenty people: seventeen technical, three functional.
The plan, at least the plan we walked in with, was supposed to be fast.
The plan
The original design was light on purpose. A lightweight web interface would capture the basic information needed to submit a sales order, feeding a new order-entry database built to bypass the old transactional system that had silently overwritten start dates and broken every SLA report Meridian had. Rules in that new database would validate each order and pull in anything else required before sending it through to Meridian’s DB2 mainframe database for provisioning.
We also built a reporting layer. Order status was visible directly in our own database from the moment a salesperson submitted an order to the moment it was complete. Provisioning was different: we had no direct visibility once an order left our hands, so the plan was to pull updates straight from DB2 into a MicroStrategy data warehouse, giving leadership a full picture from sale to fully provisioned and revenue collecting. Fast and simple. There was no reason it shouldn’t have been.
We initially estimated nine to twelve months.
We were building the front end in Internet Explorer 3, using an early Java IDE called Marimba Bongo. My part of it, from the very beginning, was weekly trips to Richmond, down Monday, back Thursday, to gather the actual rules the system needed to run on. That’s where we found out something I’d already half suspected from the first process study: more than half of the order-entry functionality and logic had never been documented anywhere. Most of what kept the place running lived in the heads of one or two senior people who knew how to troubleshoot whatever came up. None of it was written down.
That alone would have made nine to twelve months tight. It wasn’t what turned that into twenty-two.
The wall
Once we had our plan, we needed buy-in from the different groups across Meridian, the mainframe group being one of several. Sales, order entry, and provisioning were all on board, they understood the real problems and liked the solution. The mainframe group, which owned the two legacy order-entry systems and the 400-plus screens built on top of them, was the one holdout. Our design bypassed those screens entirely, writing straight to the database with the validation logic living in our own layer. That’s when they started blocking us.
They said no. Flat no. You will not bypass the screens, they told us. The screens protect everything.
I didn’t believe that then, and I don’t believe it now. Bypassing those screens was completely possible. What I knew, from every interaction with that group before and after, was a team that had quietly become a black hole inside Meridian, the kind of place where a request went in and the answer was always some version of it’ll be done when it’s done. We weren’t just proposing a faster system. We were proposing transparency, and fast delivery of new functionality, in a place that had built its entire position on neither existing.
What made it worse was that everyone who actually stood to benefit already agreed with us. Sales leadership wanted this. They knew that real tracking would let them manage their own people better and figure out exactly where information was falling through the cracks. Order entry wanted this. Provisioning wanted this. The SVP of Commercial Services, the client actually paying for the work, wanted this. Sales and order entry, the same two groups whose misaligned incentives caused the mess in the first place, were finally aligned on something, and provisioning was right there with them. None of that mattered to the mainframe group. They reported up an entirely separate chain, one with no accountability to any of them, and they had the one thing that made all of that agreement irrelevant: veto power over the database.
I don’t think that group was acting out of malice, exactly. I think they were acting exactly the way their incentives told them to act. Their job, as they understood it, was to protect the accuracy and stability of the data on those mainframes, and anything that put that at risk, or put the job security of the people who understood those systems at risk, was a threat to be stopped, not a proposal to be evaluated. Nobody on that side ever took the time to actually understand what we were trying to do, because their incentive system gave them no reason to. Understanding us carried risk. Refusing us carried none.
Meridian wasn’t operating like one company here. It was operating like several companies that happened to share a building, each one doing exactly what its own incentives told it to do, with nobody positioned to ask what was actually best for the business as a whole.
The workaround
So we scrambled. We still needed a way to collect validated order data over the web, no faxes, no handwritten orders, everything captured cleanly through Internet Explorer. That part didn’t change. What changed was everything downstream of it.
Instead of writing directly to the database, we now had to write through the screens themselves, using a screen-scraping tool called EDIFY, real, industrial-grade technology, backed and well-funded, that worked by simulating a human being: filling in a field, clicking enter, moving to the next screen, exactly the way a person at a terminal would have. It was far less efficient than a direct database call would have been. It was also the only path we had left.
Somebody had to learn all 400-plus screens well enough to document, order type by order type, exactly how a person would move through them, so the EDIFY developers could program it to do the same thing. That somebody was me, with two analysts helping me get through it. What had been a lightweight, thin-client design became a heavy, thick-client one, with Marimba Bongo carrying far more logic than it was ever meant to, and performance problems that came with it.
It was a nightmare. I mean that plainly, not as a figure of speech. I would wake up at two in the morning having just solved another screen-flow problem in my sleep, and write it down on a pad next to my bed. In the morning I’d find a solution I’d sketched out for myself in the middle of the night.
The nightly extract
The mainframe group wasn’t done. We still needed at least a daily extract of everything happening on the provisioning side, since we had no direct visibility into that once an order left our hands. We asked to query DB2 directly. Told no again.
What we got instead was a batch file, dumped once a night, around one in the morning, covering the prior 24 hours. Not incremental. Not just what had changed. Everything currently sitting in any provisioning state, every single night, in full. The pattern was clear: no to direct database access, no to incremental extracts, yes to the one option that would be hardest for us to execute.
The first time we ran our extract, transform, and load (ETL) process against that file, it took twenty-five hours to process twenty-four hours of data. You don’t need a technical background to understand that math doesn’t work.
I wasn’t on the technical team. But I’d spent the last three years working on mainframe-based systems at American Management Systems (AMS), background none of my Deloitte teammates had. When I heard about this problem, I asked the guys if anyone had heard of SyncSort. Nobody had. I’d used it at AMS. I downloaded a trial client-server version myself. We ran it against the same file that had been taking twenty-five hours.
It finished in one. The twenty-five-hour problem was never really a technical one. It only existed because of the choices the mainframe group had already forced on us, full dumps instead of incremental extracts, screens instead of direct calls. SyncSort solved a problem that didn’t need to be a problem in the first place.
The launch
That single decision by the mainframe group cost Meridian real money. The final build came in at more than three times what the original design, direct database calls instead of EDIFY and screen scraping, would have cost. It also left the system more fragile: every time the mainframe screens changed, EDIFY had to be updated to match, an ongoing tax nobody had signed up to keep paying. Meridian paid it. Its shareholders, employees, and customers paid it, in fragility and delay that never had to exist. The mainframe group never did, not in any way its own incentive system was built to notice.
What shipped, in spite of all of it, was something Meridian had never had: complete, end-to-end visibility into the status of every order, sales through order entry through provisioning, updated daily. Not real time. We were still living with that nightly extract. But for the first time, leadership could see exactly how much revenue was sitting at each stage of the pipeline, on any given day, instead of guessing or waiting on a hand-built spreadsheet. The data that had always been assumed to exist somewhere, the data the SVP from last week’s piece thought his manuals already gave him, finally actually existed, end to end, in one place.
It worked well enough that Meridian green-lit a Phase 2.
The point, so far
“The Bricks Who Had 80% of the Answer” was about a leadership failure: dismissing the people closest to the work, and trusting an outdated manual instead of verifying anything against reality. This one is about something different. This is about what happens when you actually go build the fix, and the organization you’re building it for isn’t organized to let you succeed.
Sales and order entry, back in that piece, were measured on metrics that had nothing to do with each other. By the time we got here, they’d aligned, and so had the SVP paying for the work. The misalignment just moved, onto a group with no stake in the outcome at all.
People operate within the incentive systems they’re given. That’s not a character flaw in any one person on that team. It’s what Deming meant, more precisely than the way it usually gets quoted: in Out of the Crisis, page 315, “94% belongs to the system (responsibility of management), 6% special.” The mainframe group wasn’t sabotaging us out of spite. They were behaving exactly as their system told them to behave. The failure was that no one above both groups had built a system where doing right by Meridian and doing right by your own group were the same thing.
A business can have brilliant people on every side of a problem, a design that works, a team willing to build it twice if that’s what it takes, and still lose years and millions of dollars to a gap nobody assigned anyone to close. Not a technology gap. An incentive gap: a divide between what one part of the organization was rewarded for protecting and what the whole organization actually needed. I’d spend most of the next twenty years running into some version of that same gap, in different companies, wearing different clothes, before I understood what it would actually take to close it for good.
If you had to guess, right now, which group in your own organization is quietly optimized to protect itself instead of the outcome everyone else is chasing, how long would it take you to name them?
