American View: What Do Home Renovations and Technology Upgrades Have in Common?

Management

One of the easiest ways to identify a seasoned veteran in your IT department is to watch how they react to a project going completely off the rails. While your new and mid-career techs will be shocked or angry, your campaigners will be irritated … but not at all surprised. Years of serving in the IT trenches creates techs that have seen some [censored], man … They may not know precisely what’s going to fail in a project, but they always expect something to. It’s inevitable.

I was reminded of this aphorism over the weekend thanks to a borked up home improvement project. No, really. Let me explain …

My wife and I bought our house in 1999. The place was a little over twenty years old at the time and was riddled with problems stemming from age, wear, and slapdash construction. The sellers, for example, failed disclose a major slab leak that they hadn’t fully repaired. All the plumbing leaked. The electrics were temperamental, and half the outlets appeared to have been wired by hobbyists. The water cut-off valve to the house and the main sewer drain line had both been shattered and never repaired. Small wonder, then, that we were able to get the house at a ridiculously cheap price without needing to haggle. That was an omen.

Like many Americans, the pandemic lockdown forced us to confront our enormous backlog of overdue home improvements. In 2020 we replaced the kitchen counters, the stovetop, the garbage disposal, the vent-a-hood, the water heater, the garage door opener, a dozen light fixtures, the front door, the storm door, the house surge protector, and even the doormat. We re-painted half the walls just for kicks. We’ve been keeping a running prioritised list of “stuff what needed doing” for twenty years, and finally had the time – and boredom – required to have at it.

We couldn’t attack everything on our list, though. We consulted some contractors about tearing out and replacing our (badly) rebuilt shower. Their estimates seemed to have been priced using the MSRP of a new Honda Civic. Frustrated, we put a little aside every month all through 2021 in the hopes that we could finally tackle the project by 2025 … assuming our (badly) rebuilt shower didn’t completely fail (again) and flood the living room (AGAIN!) before then.

Our living room has flooded so many times I think we’re legally allowed to advertise it as an “indoor pool.”

Our luck seemed to change last October: a family friend who flips houses had some time free up in his winter schedule and offered to sort our rebuild project for a quarter of what other contractors had quoted us. We seized the opportunity. We already knew the quality of the man’s work and were delighted we could finally get a problem sorted before it became an acute crisis.

Our contractor friend started work last week. He’d predicted a week’s work all-told. Fortuitously, the demolition phase went much faster than anticipated. Everything was looking up … so, of course, something was required to go wrong. It’s karma … or a grumpy Zeus. Or maybe sunspots; I don’t know. These things are inevitable, like death, taxes, and gritty Batman reboots.

So, yeah. The demo phase went fine. The pouring of the new slab went like clockwork. The project went off the rails on day three: while trying to replace the 1974-vintage light/vent unit, our contractor discovered that our bathroom ceiling was a complete mess. The original contractors that had installed the bracing and the electrics had kludged the job so badly that it was a wonder we still had a ceiling. Everything was in danger of collapsing, catching fire, or (per our usual luck), collapsing while on fire.

“This is pretty typical for the era,” our contractor explained while showing us the insane jury rigging just above the drywall. “The developer was probably racing to get this subdivision up as fast as possible and so were all his competitors. Specialists like plumbers, electricians, and whatnot were in high demand. They were rushing from job to job trying to keep up. It’s clear looking at this that someone who wasn’t very skilled screwed up the installation and left it as soon as it met minimum standards for completion.”

He was right … the entire ceiling was uneven and sagging, about to give way, and braced with metals rods, shims, and glue. I can’t imagine how it had stayed up. Not that it mattered; it wasn’t like we could sue the original builders for their slapdash work. All we could do was wait while our contractor tore everything out and fixed the mess … which meant our renovation project got longer and more expensive.

“If you’d have told me up front that your house was built over a hell-mouth, I’d have quoted you for the fire-resistant drywall.”

I was irritated … but not at all surprised. I’d spent enough time working as a consultant that this “surprise” wasn’t. I don’t think I ever had a job that involved replacing, expanding, moving, or upgrading production that didn’t go horribly wrong because of some inexplicable mistake or bad decision that the original site or system owner had made. Reacting to “surprises” was standard operating procedure.

I’m convinced that this isn’t a matter of simple incompetence or lack of professionalism. I’ve been part of far too many jobs where time pressure, poorly conceived plans, inadequate tools, and accidents conspired to demand a jury-rigged solution. Sure, it would always be better to slow down, re-baseline the project, and do the job completely by-the-book to ensure that future operators wouldn’t have to fix our mess. Unfortunately, real world projects often can’t afford to slow down and do things right. Installers, integrators, and consultants are under enormous pressure to get the job done now to prevent an overly aggressive critical path from immolating.

At one site, I found that all the network cabling in the data centre had been run with 50- and 100-ft patch cables, creating a “spaghetti monster” blob of excess cable on the floor large enough to nap on. Seems that the integrator had run out of one 3-ft cables and used whatever he had leftover.

While trying to run new KVM cabling under a raised data centre floor, I discovered that some previous cable techs hadn’t properly deinstalled the old cabling that had run from the server racks to the operators’ stations. They’d simply cut the cables where they exited the floor and left the unseen tangle of old wires under the tiles to serve as rat condominiums … and someone had then woven new KVM runs through that mess.

In a particularly vexing data centre buildout, the team installing the new racks insisted on only using two-post telecom models as they’d done in their telecom days. The builders fought me tooth and nail over the four-post kit I needed … right up until I delivered over my 350-pound, 7U servers and dared them to “hang” the beasts only from the front. They tried … and physics laughed at their hubris.

Gravity does not give a fig about “how we’ve always done things,” mate.

These things shouldn’t happen, but they do. When techs are pressured to deliver a minimally acceptable “solution” and are given too little time to do their work properly, they’ll hack something together that satisfies the bare minimum performance specs and call it a day. That might satisfy the project manager, but it’s a horror movie curse inflicted on the client: today’s just-in-time hack inevitably becomes a project stopping mess tomorrow.

The obvious solution to this well-understood problem is to plan intelligently and give your specialists the time they need to do their jobs correctly. Then employ conscientious supervision and rigorous Quality Assurance checks to ensure the work gets performed by-the-book. When a task is botched, use your buffer time to re-do it correctly. Or, to quote my contractor, “Duh.”

The world being what it is, that rarely ever happens. Complex technology projects are always under pressure to speed up and cut costs. Contractors only profit when their job pays them more than their labour costs (see previous, re: “duh”). Clients demand the lowest possible price and the least interruption to production possible, making operating margins impossibly thin. This means specialists rarely have the time needed to slow down and do things right the first … or to fix their mistakes. The result is a minimally acceptable job that includes preventable flaws. Everyone gets part of what they want and goes away unhappy.

Doesn’t matter if it’s installing a server or standing up a new building; poor planning and pressure to deliver miracles always results in poor performance. Poor performance on installation makes for unnecessarily awful conditions for later upgrades, repairs, and replacements. In our case, had the original builders just built our house well, we wouldn’t need to be replacing a shower.

Fortunately, I’ve put in my years serving in the IT trenches and have seen some [censored], man … I may not have known precisely what was going to fail on this renovation job, but I expected something to. Before greenlighting our shower project, I made sure to saved up a 20% overage to cover unexpected overruns and I planned for the project to run twice as long as it was scheduled for. While I was appalled by the discovery, I wasn’t surprised … and I wasn’t upset about it. I looked at the mess, agreed with our contractor that it needed to be sorted, and greenlit the project expansion.

The only other option we had was to kludge something now and hope to be out of the house before it failed. Again. To blazes with that.

Digital Transformation