January 25th, 2016

Waiting for dependencies in Docker Compose

Using Docker Compose to automate your test process is awesome, but waiting for dependent services can result in flakey tests. In this post, I demonstrate the use of Dockerize to simplify this process.

— Matthew Fellows —

It’s no secret I’m a big fan of Docker, and one of its biggest strengths is the ability to compose a fully functional local development environment that simulates a real system in minutes, helping us achieve Dev/Prod parity without sacrificing simplicity.

For example, you may want to spin up your environment in order load/resilience test it in isolation, say as part of a CI process. I’ve blogged about this before, but I didn’t mention one small annoyance – Docker Compose will build a dependency tree, and properly wait for linked services to become available, but it cannot – and will not – understand if the service within the container is up and running. This is left to the application to handle, and probably rightly so.

Unfortunately, this means a little extra work, implementing hacks like this:

Fortunately, there is a better way…

The Solution: Dockerizing your test harness

Bruno Rocha has a nice little bash script that you can use to solve the problem, and whilst it gives you some flexibility in how you determine that a service is “up”, you will need to install any prerequisites (like telnet, curl etc.) and script this yourself for each project.

But there is a simple way: as of today, v0.1.0 of Jason Wilder’s Dockerize has the capability to wait for dependent services (on tcp, http or https), and installing it is a cinch. Simply add the following to your Dockerfile:

Once you’ve installed dockerize in the dependent container you can simply prepend your command with “dockerize”, supplying any hosts you need to wait on:

Your Docker Compose file will look something like this:

9 times out of 10 you are waiting on a service to start listening on a port, so this should be all you need most of the time.

Dockerize is a neat little project that solves a bunch of Docker pain points around things like run time configuration and log streaming, and now it can make your test process even simpler!

Example Docker Compose Project

To demonstrate how all of this stuff works, I’ve created a simple Docker Compose setup. It is composed of front-end API that deliberately takes 5 seconds to start up, and a test harness that hits the API 100 times, failing if it receives any non-20x response codes. You can see the before and after in branches “fail” and “dockerize” respectively.

Head on over to the GitHub repo to see this black magic in action.

References/Further Reading

  • http://www.startupnextdoor.com/ John Washam

    Thanks for the article, Matthew. Waiting for the port to become active is not always reliable. The MySQL image, for example, will make port 3306 available but still take about 6 seconds to initialize the database engines the first time the container runs. So any attempt to connect in those seconds after the port is made available will fail.
    I had to get around this in entrypoint.sh by making the dependent service wait 10 seconds the first time it runs.

    The second time the container runs, this is skipped, and connects immediately as you’d expect in a docker composition.

    Here was my implementation in Ghost (Node-based blog):

    • http://www.onegeek.com.au Matt Fellows

      Thanks John Washam, it’s certainly not a fool-proof system however this approach (and its shell script predecessor) has worked well for me over the past year or so. That being said, there will unfortunately always be requirements for these sorts of hacks. On the surface, it seems a bit disappointing that MySQL is listening on the port when it’s not in fact ready to receive connections.

      Looking at your implementation, would it be possible to remove the arbitrary sleep (https://github.com/jwasham/docker-ghost-template/blob/master/ghost/copy/entrypoint.sh#L7-L10) and instead loop until the “$GHOST_CONTENT/themes” directory is created?

      • http://talktothemanager.com/ John

        Thanks, but you’re reading it backwards. :)
        That condition is only true when the container is first run, along with the database container’s first run. So it’s waiting for the database to initialize, not the directory to be created. The directory it’s using in the test will exist the next time the container runs, and won’t need to sleep. These are first-run problems only when the containers are “fresh-out of the oven” and have to cool a bit before you can eat them.

        • http://www.onegeek.com.au Matt Fellows

          That’s what you get when reading code on a mobile ;). I thought perhaps that directory was created when it was “cooled-down”. Never mind!