September 4th, 2014

DevOps for the Enterprise: RACI destroys DevOps

Today’s Enterprises need to rethink tools such as the destructive RACI in order to create a collaborative DevOps culture.

— Matthew Fellows —

For the past 12-18 months, Melbourne IT has been on a journey pushing toward the edge of the known DevOps Universe.

As we progress along the DevOps maturity continuum, I’d like to reflect on one aspect of Corporate thinking that was a barrier to truly realising progress, until we recognized that we were (incorrectly) applying thinking from one paradigm to another.

I’ll use the term DevOps here as a noun for a culture, the act of true collaboration between Developers, Operations and the rest of a team working to deliver quality software with agility.

A short digression

In the Copernican revolution, Nicolaus’ predecessors struggled to reconcile observable evidence in the skies against a theory placing the Earth at the center of the Universe. Retrograde movements of certain stars & anomalies in the apparent size of Venus were just a couple of these that were easily explained with the new theory that placed the Sun instead at the center. Discarding geocentrism with heliocentrism required more than simple rejection of an old theory, however, it required rejection of a philosophy of life – Earth was no longer the unique & special center of the Universe. Humans, were just an insignificant compilation of atoms on a pale blue dot, in an incomprehensibly large Universe.

Copernicus was not the first to propose a heliocentric universe, but his political connections, background and timing enabled him to shift the thinking of the social Zeitgeist and ultimately realise this radical new vision which opened up a world of new possibilities.

But what the hell does this have to do with DevOps?

A lot.

DevOps is not really a new idea, smaller companies & teams have been managing systems on their own for years. Typically, however, as the size of a company grows we see specialisation of roles and corresponding Corporate structures. In part, this is due to business pressures, outdated Corporate Governance procedures (such as Auditors requiring ‘segregation of duties’) and was considered to be ‘best practice’.  In reality, however, this results in placing certain functions (and therefore, people) at the center of an ever-changing Universe, one that is never truly reconciled against observable evidence. Testers become the bottleneck when code needs to be tested, Ops become the center when a new environment is needed or code is required to be shipped and so on for each silo assigned an R in the RACI. This is the pre-Copernican, geocentric view that is core to the current Corporate’s world view, where RACIs are a commonly used tool to define such separation and are key in the KPI & role assignment process.

Here is a typical (cut-down) RACI pre-DevOps:

DevOps RACI

Pre-DevOps RACI

Note how it quite clearly articulates who’s job it is do perform a certain function, which in the old philosphy made sense. But observable evidence in the field suggests this way of thinking is flawed:

  • It sends a signal to the group, that it is not my responsibility for other aspects of delivery – quality, shipping, reliability etc.
  • It creates friction in cooperation in the group – I can’t trust somebody else to do my job, to meet my KPI expectations. Can I trust that Dev to push his code to Production or am I going to be woken up at 3am again?
  • Possibly most importantly, as a side-effect of the above, it reduces the surface area of knowledge spread across people. This means I now have less people who understand the entire solution, meaning
    • Staff rotations/turnover is painful
    • Developers – the primary application maintainers – lose the ability to debug and resolve issues quickly when there is a critical incident
    • Testing is reduced to functional, business related testing and hardcore systems testing is made difficult
    • Worse still, it means there is less motivation, more friction in improving the overall solution – Engineers who are ‘under the hood’ can’t tinker and improve the entire engine, just a small part of it, as they need to raise jobs, time their release with others etc. “Put that in the too hard basket – I have stuff to do!”
  • …and the list goes on

It often starts with a position of mistrust – I do not trust you with responsibility for aspect X, therefore I will not provide you with authority/access to contribute to aspect X, leading to a lack of understanding of aspect X by all who are not contributing to its existance.

Today’s Enterprises are ready to reject this world view.

The more we think on it, the more it doesn’t make sense to consider DevOps in the traditional management mindset of a RACI. What we propose instead, is Shared Ownership & Roles based on Expertise (SORE?). In practice, this means:

  • All members of the team are Responsible, Accountable, Informed and Contribute to the entire solution. Everyone understands how the solution is built and deployed
    • It is not one persons’ responsibility for quality, for shipping, for creating a solution to meet the needs of the business.
    • That is not to say that a team shouldn’t contain people with expertise in managing servers and network devices, or engineers who get a kick out of breaking a system or any of the functions on the aforementioned RACI – we draw on that expertise, but we don’t structure ourselves around it.
  • Ops resources are collocated with the rest of the team. In an ideal world, they share reporting lines with a cross-functional product team but this is not necessary.
  • Everyone shares on-call duties and can support the system when it explodes (and it will)
  • The team is assigned a shared goal / KPIs are aligned
  • Everyone is trusted and encouraged to contribute to any aspect of the delivery as a default position

DevOps is to shipping as Agile is to project methodology; it’s about getting everyone in the room together –  including those traditionally in the ‘Ops’ group – to deliver an outcome.

TL;DR

Shifting from Silos to a DevOps culture is like shifting from Geocentrism to Heliocentrism – it requires a cultural shift to realise the benefits, and todays’ Enterprises need to rethink tools such as the RACI in order to succeed in this new world.

  • Jon Stribling

    Check out http://holacracy.org.

    Also, Bain’s RAPID decision model is successfully being used in agile organisations with devops practices.

    • http://www.onegeek.com.au Matt Fellows

      Thanks Jono I’ll definitely check that out. I know of a few very technical organisations embracing some of these ideals (GitHub & Valve to name some of the more (in)famous examples) but it’s great to see interest in driving those values into more ‘traditional’ companies.

      There is no doubt that lean organisations – not just in the software delivery space – have an edge in many cases.

  • Alain Sacquet

    Thanks for the post
    I just find it. A bit late for a vibrant reply.
    My point is RACI perspective and the siloes. There are two ways to understand the DevOps shift: No more siloes or other siloes. No more planets you are circling around, or an other one, a new star.
    SORE as described is not a heliocentrical theory, but a polycentrical one. All members of the team are stars. The analogy doesn’t work that much if you want the end of the siloes.
    In my opinion, DevOps is indeed a Copernicus revolution. The new stars are the”features”, the main subsystems of the Information system. No more the fonctionnal step teams of the waterfall journey. It means that feature teams are the new siloes we have to put under a global architectural control.
    It simply appears that this new organization seems to be more effective.
    Feature teams are quite responsible of the product they are in charge of.
    DevOps, among other caracteristics, is a new organization, not the end of the gravitational law.

    Thank you for reading

    Alain

    • http://www.onegeek.com.au Matt Fellows

      I agree, we don’t want to create a new thing we gravitate around, so perhaps the analogy ends there. But the point was a different perspective, a different way of thinking was (and still is) required to properly understand and benefit from the new approach. DevOps only takes this so far, as discussed other ways of working extend this thinking outside of the IT structure across the entire business, connected what were disconnected processes previously: I think that is the natural, logical evolution of this sort of thinking, fully cross-functional teams across a business.