Docker Image Automated Tests
We are going to explore automated testing for docker images. I found this to be a bit more difficult than application testing because images are generally built with shell scripts and I couldn’t find a comprehensive framework in the likes of JUnit or RSpec that didn’t require a lot of dependencies. We will look at a few continuous delivery (CD) solutions so we can see the differences in implementation.
I assume a basic understanding of Docker images, what they are and how to build them using a Dockerfile.
Facilities for Automated Testing
We will show examples of automated testing for the following services:
Fortunately the same test code will be shared by all. Each service has it’s own configuration file for invoking the tests which we’ll look at to invoke the tests.
We’ll be using https://github.com/double16/forked-daapd.git for our example. forked-daapd is a music server using the DAAP protocol recognized by iTunes and other software, and also using the Roku music protocol. Our test is simple, make sure the server is listening to the port. More tests can and should be written, but this will suffice for our purposes.
Docker Hub
Docker Hub provides facilities for building and publishing docker images. It is the default registry in docker installations and public image hosting is free.
Docker Hub provides automated testing facilities through Docker Compose files. Any files in the same folder as the Dockerfile that end with .test.yml will be considered a Compose file to be used for testing. Docker Hub will perform a one-off execution of the sut service. This is where your test scripts are executed. The return code from the sut service is used to determine if the tests passed or failed. A return value of zero is considering passing, non-zero is failing.
File Structure
Here is an overview of the files we’ll be creating. (The files needed to produce the application container are omitted.)
.travis.yml # Config for TravisCI circle.yml # Config for CircleCI docker-compose.test.yml # Config for shared test environment Dockerfile # Builds the images test - Dockerfile # Builds the image running the tests - results # Mapped into the test container to hold the xunit results - sh2ju.sh # Records test results in `xunit` (aka `JUnit` format) - test.sh # Runs the tests - wait-for-it.sh # Waits for the container under test to start up
docker-compose.test.yml
docker-compose.test.yml defines the container environment for our tests and what to invoke. The sut service is what Docker Hub will invoke as a one-off command, equivalent to docker-compose -f docker-compose.test.yml run sut. We link the image under test, in this case daapd, and the container will be available to us with the host name daapd. The /results volume will be used to store test results using xunit/JUnit format so that the CI environment can report on our individual tests. Any number of necessary services, volumes, networks, etc. can be used to support the tests, they will not affect the published image.
version: '2'
services:
daapd:
build: .
sut:
build: test
links:
- daapd
volumes:
- ./test/results:/results
The alpine image being used for the tests is a very small image based on busybox. If you need more functionality, other images will work as well.
FROM alpine:3.4 COPY . . CMD ["./test.sh"]
test.sh
The tests can run any command as long as the exit status indicates pass/fail. The script will need to handle returning a failing exit code if any of the tests fail, otherwise the exit code of the last command is returned from test.sh.
The wait-for-it.sh script is used to wait for the container to be started. It has options for setting timeouts and requires very few commands to do it’s work.
The sh2ju.sh script reports command status using the xunit (aka JUnit) XML format. Most CI environments can consume this and it makes for nice reports.
#!/bin/sh /wait-for-it.sh daapd:3689 # Provides juLog and juLogClean source /sh2ju.sh # Remove any previous test results juLogClean # Run a test named 'opentcp' that ensures the server is listening juLog -name="opentcp" nc -zv -w 10 daapd 3689
Test It!
With this repo configured in Docker Hub as an automated build, pushing will trigger the image to be built and tested. Only when the tests pass will the image be published.
CircleCI
Integration with CircleCI is straightforward. CircleCI uses the circle.yml file in the root of your repo to configure the pipeline.
Since docker-compose.test.yml is a standard Compose file, we can invoke Compose in our circle.yml file to run the tests. CircleCI wants test results in the directory referenced by CIRCLE_TEST_REPORTS, so we’ll copy them over.
You’ll want to configure your CircleCI project to run on the latest Ubuntu available (trusty at this time), and report on JUnit results. At the time of this writing the docker install is older, so we’ll install version 1.10.0.
machine:
pre:
- curl -sSL https://s3.amazonaws.com/circle-downloads/install-circleci-docker.sh | bash -s -- 1.10.0
services:
- docker
dependencies:
pre:
- sudo pip install docker-compose
test:
override:
- mkdir -p ./test/results
- docker-compose -f docker-compose.test.yml build sut
- docker-compose -f docker-compose.test.yml run sut
- cp ./test/results/* $CIRCLE_TEST_REPORTS
TravisCI
Integration with TravisCI is simple as well. The pipeline configuration is stored in .travis.yml (don’t miss the leading dot). The docker version provided is most recent than CircleCI, so the configuration file is smaller.
sudo: required services: - docker before_install: - mkdir -p ./test/results - docker-compose -f docker-compose.test.yml build sut script: - docker-compose -f docker-compose.test.yml run out
Jenkins
Jenkins 2.0 (and earlier versions with the workflow plugin installed) support configuring a pipeline using a Jenkinsfile in the root of the repo. You must have docker and docker-compose installed on the Jenkins slave (or master) running the pipeline.
Create a new Multibranch Pipeline item and add your repo in the source code section. Run the pipeline and Jenkins will build and test your container.
node {
checkout scm
sh """
mkdir -p ./test/results
docker-compose -f docker-compose.test.yml build sut
docker-compose -f docker-compose.test.yml run sut
"""
step([$class: 'JUnitResultArchiver', testResults: '**/test/results/TEST*.xml'])
}
Summary
Docker containers are a popular packaging and deployment architecture. Images can and should be tested as with other software and there are straight-forward methods available. The approach described here uses simple shell commands to perform tests. The docker-compose.test.yml design allows a separate test image to be constructed, so more complex tests using other languages and frameworks can be used without increasing the size of the image being tested. CI tools that include Docker as part of the test infrastructure should be able to incorporate automated tests as part of the delivery pipeline.
About the Author
