Change Advisory Board: A 1980 Solution to a 2021 Problem
"Patch Tuesday is a disaster" was the opening line of a conversation I recently had with an IT executive.
"We sit through hours of Change Advisory Board meetings, review all the changes, then we roll them out and wait for the phone to ring. It's almost inevitable."
I had flashbacks to my time running IT operations for a $10B business in the early 2000's – a Friday afternoon meeting in advance of "downtime Sunday" which consisted of a maximum four-hour window where there were no SLAs for availability. It came around just once a month and was pre-negotiated with the business for the entire year ahead. Inevitably, in the days preceding the change window, I would get a call from at least one business segment leader asking me to cancel the planned downtime because of some critical project their team was working on.
Predictably, I would say no because if I said yes just once, the jig was up forever. Fast forward, a decade or so into the 2010's and Patch Tuesday (or your equivalent) had become a weekly occurrence. Mostly because the rate of change was accelerating and DevOps was starting to take hold – smaller changes, more often – feed the beast of customer appetite for new features and capabilities. Things couldn't wait for a month to get into Production. By 2021, this process is a farce, and anyone involved in a Change Advisory Board knows it – it's not just that the Emperor has no clothes – it's that he's nearly dead from extended exposure.
The Change Advisory Board (CAB) concept was born decades ago as a check in the process of Change Management to ensure that planned updates to Production IT operations would not cause an outage, degrade performance, or bankrupt the company. Changes were expensive in hard dollars (programmers wrote assembler to optimize resource usage) while business dependence on IT was relatively low.
The CAB was the designated final arbiter as to whether a change was moved into Production or not. In the ensuing decades, those realities inverted, and the hard dollar cost of change is negligible compared to the dependence on IT systems to support every aspect of the business from revenue acquisition to retention. Yet, the CAB mostly remained in place as a vestige of Change Management processes, which have long since become close to worthless.
DevOps demands that changes move seamlessly from development to production – and in a cloud-first, browser-based delivery paradigm, that works. But what about all the changes to the vast legacy estate of Windows native applications and the complex infrastructure stack of chip firmware, operating system, hypervisor, middleware, security, and desktop presentation layers (to mention just a few). The demands for updates/changes/patches etc., to these systems, are just as real to respond to the business' appetite for customer engagement.
Large-scale Enterprise companies are already trying to get to an exception-based CAB, where the only changes reviewed are the ones that fail a series of prescriptive, rigorous tests. "We are trying to automate decision making around change" is how another IT executive described this process to me, "if it passes all the tests, it's automatically promoted to Production. There should be no people in the change process, from originator to the consumer."
More than 90% of the issues that arise after a change process have nothing to do with the actual change. Rather, they have everything to do with the Law of Unintended Consequences – where a condition existed in the environment triggered by the change. There is no earthly way the CAB could predict the possible ramifications of all the proposed changes. The hours reviewing, discussing, pondering, and head nodding (mostly by senior people with the least likelihood of appreciating a nuanced implication of a planned change) is merely paying homage to a bankrupt concept.
As we moved through the past twenty years of increasing demand, many obvious "step factor" innovations created dramatic shifts in the way we work. In 2000, asking IT to provision a new server was a process of filling in forms, allocating budget, and then waiting two or three months. By 2010, that had turned into a month or so for a Virtual Machine at some outrageous internal bill back price. AWS turned it into 60 seconds, the swipe of a credit card and pennies on the dollar by comparison. Expectations were forever shattered. They turned "IT into a business", which was a line every CIO had used to describe their goals for the previous, and the subsequent decade, and abysmally failed at.
Developers took note and, at that moment, no excuse for how hard it is to procure and provision servers was ever again good enough. Central IT was no longer the only game in town. Subsequently, the tools made available for developers to manage the lifecycle of changes, build code faster, automate testing, etc., have revolutionized the pace at which changes can be produced. They churn out candidates for promotion to Production faster than it could have been imagined just years earlier, only to run smack into 1980 and the CAB. The only remaining gate that IT universally controls is when changes roll into Production. On the surface, it would seem that little to no innovation has occurred to facilitate this "last inch" of the process.
The reality is, it's all a factor of confidence, or lack thereof, that IT can predict and control what will happen. For the same reasons, a business leader would call me asking to stop Downtime Sunday back in 2000, and we continue to hold onto the CAB today. No one trusts that these changes will be flawlessly rolled into Production resulting in zero issues, nor that they can just as easily be reversed if the worst happens. With good reason, IT has demonstrated time and again that this is a process fraught with peril.
Despite all the CAB discussions, we inevitably end up in a Root Cause Analysis (RCA) meeting the following morning. While the average developer likely has run a myriad of tests in a controlled lab environment, there is generally minimal testing done where the rubber meets the road – a Production equivalent environment under the types of stress conditions and interoperability scenarios that will occur immediately after the change.
Given the appetite for change and the goal to enable systems to promote changes directly into Production, we must blow past the vestiges of historic Change Management and finally put a stake thru the CAB by adopting a massive scale, an automated testing framework at the point where the applications and infrastructure meet. You cannot get there in a single motion. Test coverage needs to be built up over time, and a single test can be a force multiplier when used across multiple test scenarios.
Only with such wide-scale automated testing will we ever get to that "automated decision making" mentioned above. To arrive at that type of change velocity, automated testing at a massive scale is the only feasible path, and it will roll right over the CAB and any other people-based process that stands in its way. Those who take that journey will fulfill the promise of DevOps and deliver a truly differentiating advantage to their business. Isn't it time to give the Emperor a blanket and a well-earned retirement?
Do you agree? What are you doing in this area? I'd love to speak with you if you have the same or a differing opinion to discuss how to move the industry forward.
Please reach out to me at email@example.com or check out how Login VSI can help you build a broad and deep testing framework by downloading a trial of our flagship product, Login Enterprise, at www.loginvsi.com/downloads.