Embarking on a DevOps transformation puts many leaders in a frozen state of quiet panic, but the pressure to innovate quickly is constant and pervasive. Technology is playing an ever-increasing role in the lives of businesses and consumers and it's obvious that those companies who do not innovate quickly will perish. But how else do you shorten the amount of time required to put innovative technology to work for your customers and your business? "DevOps" is the industry standard answer. However, you could read countless blog posts on what it is, how to start, what to do, etc., without gaining any insight into how to do DevOps at your company.
But there is hope. DevOps is more achievable than you might think, and you can find the path to success in the most unlikely (and uncomfortable) of places. One of the senior leaders I work with once quoted, "Never let a good crisis go to waste." She was referring to the incredible amount of energy, innovation, teamwork, and problem solving she saw in her organization whenever a production outage occurred. And this is exactly where any leader can look to understand the true path to DevOps specific to their organization.
The Hidden Wisdom of a War Room
The drill for my client was standard. A production outage would be identified by the network operations control (NOC) center, and a war room would quickly convene. If the issue dragged on, more and higher profile people would be alerted to the issue and join the war room. As inefficient as this sounds, it had an accelerating effect due to a few reasons:
All the right people required to implement a change, approve it, and get it into production were in the room, engaged in the process.
Only the most essential testing and validation steps were required to implement the change.
The people making the changes were given elevated permissions in production for a limited time to investigate problems and make changes to production, as long as those changes were documented and approved by the others in the war room.
Most importantly, the changes they were making were very small, which is why they could get by with a lower level of testing. Sometimes the change was just to gather additional logs to help diagnose the problem. Other times the change would disable a new feature in the hopes it would resolve the issue.
As a result, changes were pushed relatively quickly and safely into production. Unfortunately, this process was very inefficient and costly for the organization. There's no reason why this couldn’t be accomplished on a smaller (and less panic-driven) scale by asking some key questions and crafting your development teams accordingly.
5 Questions to Ask About Your Company’s Crisis Response Procedures
Start by assessing what happens at your organization during a crisis. When the organization is experiencing a production outage or a revenue-impacting event:
Who needs to be involved? Why?
What approvals are required to get a fix in place?
What is the minimum amount of testing required to ensure key business processes won't be broken? How do we perform that testing?
What is the process for getting the fix in place?
How is the problem diagnosed and broken down into small fixes that can be applied quickly to resolve the issue with minimal risk?
Use Your Answers to Guide Your Path Forward
Based on the answers to these questions, you can take three actions that will start and guide your DevOps journey:
Break work down into smaller and smaller deliverables and deploy each one to production.
Create development teams that have sufficient representation to grant all required approvals.
Take this example: If the CISO or an appointed delegate must grant approval of all changes going into production, then you need an appointed delegate on your development team (not necessarily 100% of the time, but it needs to be a single person).
Begin automating the processes for getting the team’s changes into production:
Deploying the changes into production.
Security scanning and testing procedures.
Testing of critical business processes.
Continue Your DevOps Journey
By using your company's crisis response procedures to identify the processes that are truly required to put changes into production, you can begin to lay the groundwork for a successful DevOps initiative. From here you can launch a DevOps pilot initiative to implement and adjust these practices and figure out what works for your organization. You can leverage the success of your pilot to roll out DevOps practices across your organization. Ready to take the next step in your DevOps journey? Contact us at firstname.lastname@example.org.