For the past 12-18 months, Melbourne IT has been on a journey pushing toward the edge of the known DevOps Universe.
As we progress along the DevOps maturity continuum, I’d like to reflect on one aspect of Corporate thinking that was a barrier to truly realising progress, until we recognized that we were (incorrectly) applying thinking from one paradigm to another.
I’ll use the term DevOps here as a noun for a culture, the act of true collaboration between Developers, Operations and the rest of a team working to deliver quality software with agility.
A short digression
In the Copernican revolution, Nicolaus’ predecessors struggled to reconcile observable evidence in the skies against a theory placing the Earth at the center of the Universe. Retrograde movements of certain stars & anomalies in the apparent size of Venus were just a couple of these that were easily explained with the new theory that placed the Sun instead at the center. Discarding geocentrism with heliocentrism required more than simple rejection of an old theory, however, it required rejection of a philosophy of life – Earth was no longer the unique & special center of the Universe. Humans, were just an insignificant compilation of atoms on a pale blue dot, in an incomprehensibly large Universe.
Copernicus was not the first to propose a heliocentric universe, but his political connections, background and timing enabled him to shift the thinking of the social Zeitgeist and ultimately realise this radical new vision which opened up a world of new possibilities.
But what the hell does this have to do with DevOps?
DevOps is not really a new idea, smaller companies & teams have been managing systems on their own for years. Typically, however, as the size of a company grows we see specialisation of roles and corresponding Corporate structures. In part, this is due to business pressures, outdated Corporate Governance procedures (such as Auditors requiring ‘segregation of duties’) and was considered to be ‘best practice’. In reality, however, this results in placing certain functions (and therefore, people) at the center of an ever-changing Universe, one that is never truly reconciled against observable evidence. Testers become the bottleneck when code needs to be tested, Ops become the center when a new environment is needed or code is required to be shipped and so on for each silo assigned an R in the RACI. This is the pre-Copernican, geocentric view that is core to the current Corporate’s world view, where RACIs are a commonly used tool to define such separation and are key in the KPI & role assignment process.
Here is a typical (cut-down) RACI pre-DevOps:
Note how it quite clearly articulates who’s job it is do perform a certain function, which in the old philosphy made sense. But observable evidence in the field suggests this way of thinking is flawed:
- It sends a signal to the group, that it is not my responsibility for other aspects of delivery – quality, shipping, reliability etc.
- It creates friction in cooperation in the group – I can’t trust somebody else to do my job, to meet my KPI expectations. Can I trust that Dev to push his code to Production or am I going to be woken up at 3am again?
- Possibly most importantly, as a side-effect of the above, it reduces the surface area of knowledge spread across people. This means I now have less people who understand the entire solution, meaning
- Staff rotations/turnover is painful
- Developers – the primary application maintainers – lose the ability to debug and resolve issues quickly when there is a critical incident
- Testing is reduced to functional, business related testing and hardcore systems testing is made difficult
- Worse still, it means there is less motivation, more friction in improving the overall solution – Engineers who are ‘under the hood’ can’t tinker and improve the entire engine, just a small part of it, as they need to raise jobs, time their release with others etc. “Put that in the too hard basket – I have stuff to do!”
- …and the list goes on
It often starts with a position of mistrust – I do not trust you with responsibility for aspect X, therefore I will not provide you with authority/access to contribute to aspect X, leading to a lack of understanding of aspect X by all who are not contributing to its existance.
Today’s Enterprises are ready to reject this world view.
The more we think on it, the more it doesn’t make sense to consider DevOps in the traditional management mindset of a RACI. What we propose instead, is Shared Ownership & Roles based on Expertise (SORE?). In practice, this means:
- All members of the team are Responsible, Accountable, Informed and Contribute to the entire solution. Everyone understands how the solution is built and deployed
- It is not one persons’ responsibility for quality, for shipping, for creating a solution to meet the needs of the business.
- That is not to say that a team shouldn’t contain people with expertise in managing servers and network devices, or engineers who get a kick out of breaking a system or any of the functions on the aforementioned RACI – we draw on that expertise, but we don’t structure ourselves around it.
- Ops resources are collocated with the rest of the team. In an ideal world, they share reporting lines with a cross-functional product team but this is not necessary.
- Everyone shares on-call duties and can support the system when it explodes (and it will)
- The team is assigned a shared goal / KPIs are aligned
- Everyone is trusted and encouraged to contribute to any aspect of the delivery as a default position
DevOps is to shipping as Agile is to project methodology; it’s about getting everyone in the room together – including those traditionally in the ‘Ops’ group – to deliver an outcome.
Shifting from Silos to a DevOps culture is like shifting from Geocentrism to Heliocentrism – it requires a cultural shift to realise the benefits, and todays’ Enterprises need to rethink tools such as the RACI in order to succeed in this new world.