June 14th, 2013

Devops Dojo for Distributed Teams

My notes from the “Open Dojo” session #1 – DevOps and Distributed Teams Hangout session.

— Matthew Fellows —

Here are my notes from the “Open Dojo” session #1 – DevOps and Distributed Teams Hangout with Atlassian today. You can watch the session and see the notes here:  https://plus.google.com/events/co7ase10gnctdgenjis2860jkjg

What is the biggest challenge with distributed teams?

  • Distance/TZ
  • Go out of your way to work “face-to-face” using Video Conferencing (VC) – Skype, Hangouts
  • Periodically actually visit face-to-face in the distributed offices, don’t underestimate the value of this
  • Multiple TZs can also be beneficial e.g. 24/7 support means an issue can be worked on around the clock
  • Daily stand-ups over VC can help keep things on track


  • Having a group of people that actually want to improve is a key starting point (Peter L). It starts with Culture
  • It took 2 devs about 1 month to build a set of tools to automate E2E release for an internal project at Atlassian
  • Automated Testing
    • Incremental changes from manual tests to automated tests.
    • Revolution / ‘down tools’ not necessary
    • Measuring Success
    • Metrics: count nr bugs, production regressions

Development Environments

  • Use of Feature branches in Git
  • Made the process of checking out to having running software super fast (< 30 mins w\ Production Data)
  • Automate Development environment setup
  • Use of Puppet to provision instances (configuration of servers, apps etc.)
  • Use of Vagrant w\ Puppet

Deployment Pipeline

  • Use of Puppet to deploy feature branches to target environment
  • Target environment can be provisioned using Puppet
  • Maven runs builds, build plan executed in Bamboo to build branch
  • Feedback via HipChat to developers
  • Master is always built automatically as soon as code is committed into master
  • Deployments from master
  • Pull requests via BitBucket
  • Code reviews
  • “It’s all about incremental changes”
  • Also, Bamboo will be looking to build server/environment provisioning type features into the deployment process
  • Team Building?
  • “Ship It” days are really good team building. Business gets out of the way, pizza and beer. Merge/Deploy
  • Beer cart (or Beer tricycle in San Fran)
  • Weekly online games

Private Cloud / Public Cloud?

  • KVM(?) virtualisation for build agents
  • VMWare for IT Systems
  • a little bit of EC2 in there

“How do you compartmentalise change? You obviously make use of “Dark Features”, what about when a change is required to be coupled to a downstream API or DB change?”

  • Dark Features are pretty much hard coded in Jira, no real infrastructure in place to automate this process
  • BitBucket uses Django w\ infrastructure for dark features
  • Generally changes released using Git Feature Workflow
  • Changes requiring downstream changes are subject to a modified deployment workflow (risk-based?)
  • Database changes are also modified incrementally (new columns added, migration performed, change released, columns removed)
  • Make changes backwards compatible if possible
  • Key is to reduce dependencies if possible. , even if it means a second release on the dependent system down the track to remove any temporary functionality/kludge. This makes deployment safer and rollback easier.

“Thanks! So, would it be fair to say that you have a risk-based deployment model depending on the type of feature you’re working on? If so, who makes the call?”

  • Yes, and it’s a fairly ad-hoc, w\ no formal process for it
  • Developers make the call, depending on complexity and gut feel
  • Test coverage linked to complexity
  • There is a build engineering team, responsible for build infrastructure. Product teams responsible for releasing their own code
  • Rate of change can be controlled
  • Pairing Developers during change periods can reduce risk significantly
  • As mentioned earlier – reducing the size of the change is KEY