A note before we start: the people and organizations in this story are real. The company’s name is not. I’ve changed it to protect their privacy, not to soften what happened.
Setup
Early June 1996. My first assignment at Deloitte Consulting was this two-month process study for a major telecom carrier. We’ll call them Meridian Communications.
I walked into the office of the SVP of Commercial Services. In the room with me were the Deloitte manager and partner I reported to for this assignment.
The problem
The SVP laid it out. Meridian was signing large contracts for thousands of phone lines running over trunk lines, think call centers and office complexes. Depending on the product, they were promising delivery in eleven or nineteen days. The internal system said they were hitting that. Recently, though, the wall calendar said otherwise: thirty to forty-five days, sometimes more. Nobody could explain the gap, or why it had just started happening.
The dismissal
I asked the SVP the obvious question for me: who’s closest to this, who actually understands the process, start to finish?
He told me it was the order entry team in Richmond, Virginia.
“Okay, great,” I said. “I need to speak with them. I need to go down there and spend a day with them.”
“You don’t need to go down there,” he told me. “They’re a bunch of bricks.”
I will never forget those words. I was in complete shock. I’d never heard anyone be so dismissive toward an entire group of people within a company before.
He pointed at the manuals. “I’ve got all the manuals right here, and everything you need is in them.”
“Well, if they’re closest to the problem,” I said, “I’d like to spend at least a day with them to see what I can learn.”
He begrudgingly allowed me to go down to Richmond for the day. I was living in a condo near the Braddock Road Metro station at the time, the north end of Old Town Alexandria. Richmond was about a hundred miles south, an hour and forty minutes down I-95.
Richmond
I did some advance planning to pack in as many interviews as possible and still get down and back to Richmond in a single long day.
The order entry people greeted me warmly, and it quickly became apparent that they were dedicated to their jobs and to making sure the clients got connected. They understood what was going on end to end. And they knew what most people didn’t: the manuals were hopelessly outdated, and the process wasn’t properly documented anywhere.
Those so-called bricks were some of the most dedicated, hardworking, intelligent people I’d ever met, working under difficult circumstances. They understood roughly 80 percent of the actual problem, and they supplied roughly 80 percent of the eventual solution through their knowledge of the process and systems as actually built, not as designed.
These orders were configured and provisioned through two mainframe systems, old green screens, eighty characters wide by twenty-four lines tall. Every time new functionality was needed, instead of adding new screens, the developers figured out where on the existing screens they could cram it in. They even resorted to pop-up-style functionality to get even more onto a screen. No one had gone back and updated the manuals in years. Nothing about them reflected how the two systems actually worked anymore, or how the work actually happened. And that was before we even got to the process of getting an order from the salesperson, all the way through provisioning, to service turned up for the customer.
Root cause
The root cause of the reporting disconnect, why the systems were saying orders were being delivered in eleven to nineteen days, while the calendar showed thirty to forty-five or longer, was that the two systems were transactional, not historical systems. Every time an order was kicked back to sales for a correction, it was re-entered under the same order ID, and the system silently overwrote the original start date. An order that came in Monday and wasn’t corrected until Thursday would report as having started on Thursday. This could happen multiple times per order: each time it was kicked back to sales and came back in, the date and time stamp was updated for the start of the order, under that same order ID, silently erasing a week or more from the record while the real clock kept running underneath, because the sales organization was tracking orders based on the day they were originally submitted to order entry, not the day it was actually accepted by order entry as a proper, complete order.
Why did this suddenly become a problem, when in the past it hadn’t been? That traced to a siloed incentive system, instead of shared incentives across everyone responsible for taking care of the customer. Sales and order entry were measured against different metrics that had nothing to do with each other. Salespeople were incentivized to sell, get the order in, and once it was out of their hands, complete or not, move on to the next one. Order entry was measured by how quickly they entered orders once received from sales, but they were looking at the problem as a whole and trying to serve the customer. They were the hidden solution holding everything together to make the eleven-to-nineteen day timeframe work. They were doing the hard work of tracking down missing information, calling salespeople to remind them to get things in, and contacting customers directly when needed. They were focused on real customer service, solving the problem end to end.
The tipping point
Order entry’s leadership finally told the team to stop helping sales: if there was a problem with the order, send it back, that was sales’ problem to get right. Order entry’s job became simply entering the order into the system to get it to provisioning, so any delay sat with sales instead of with them. Salespeople now had to chase down information they were poorly equipped to find, work order entry used to handle for them, because salespeople sell. That single policy change turned a collaborative, same-day-fix team into a reject-and-wait silo, and it cost real money: poor-quality service to the customer, reduced revenue growth, and more than likely, contractual penalties for missing deadlines.
Underneath that, it was actually worse. Because these were transactional systems, they weren’t designed for the type of reporting needed to run a business. The SVP’s own executive reports, the ones driving these large commitments, were actually assembled by hand: a group pulled data out of the mainframes and rebuilt it into Excel spreadsheets for leadership, because the mainframe systems themselves weren’t configured to produce the full picture systematically.
When we gave the SVP an update, to say he was shocked was an understatement. He’d assumed the system manuals were complete and accurate, both from a process perspective and a screen perspective. He had no idea these were transactional systems, not historical ones.
The solution
I was sitting at home finishing up the write-up of the process study. At that time, that’s what I was told we did: we write up the results and hand them to the client. Solving the identified problems was someone else’s domain. I couldn’t leave it as diagnosis only, so I started sketching anyway. It turned into a one-page Visio diagram, a web front end, feeding a validation layer, feeding a proper historical database. That database would connect directly into Meridian’s DB2 database, with a data warehouse behind it for reporting. I sketched two versions: Order Manager Light, a web front end still requiring manual re-entry into the roughly 400 legacy mainframe screens, and the full Order Manager, which bypassed those screens entirely and wrote straight to the database. My boss’s call: drop Light, keep only the full version.
I attached the concept to the final report, and went on to my next two-month process problem for Meridian. A few weeks in, my manager on the engagement called. “They want you to build it,” he said.
“Who? Build what?”
“Meridian. They want you to build Order Manager.” That’s the story for the next article.
The point, so far
The SVP wasn’t wrong to want proof instead of a guess. He was wrong about where to find it, and he had no real way to know that, because nobody had ever given him a way to tell the difference between a manual and the truth. He had no functioning way to verify anything coming out of his own area of the company. So when he needed to point to the truth fast, he pointed at the inaccurate, outdated documents in front of him instead of the people who actually did the work, because the paperwork was the only thing that looked like an answer.
Deming is often paraphrased as saying most workplace failures trace back to the system, not the worker. The actual line is more specific, and more useful: in Out of the Crisis, page 315, “94% belongs to the system (responsibility of management), 6% special.”
The order-entry team was doing the best job possible under extremely difficult conditions, inside a pair of systems continuously jury-rigged to add new functionality. Those systems were built to get orders provisioned and services stood up, not to answer the questions the SVP and everyone else actually needed answered: the executive reporting required to run that part of the business. They kept customers provisioned in the timeframes Meridian had promised, right up until they were told to stop doing the job that needed doing and just do the job they were hired for: stay in their own lane. The SVP wasn’t defending bad judgment either. He was operating the only feedback loop he had.
Next week: how we built the system that finally let leadership see what was actually happening.
Continued in Veto Power, No Stake in the Outcome.
