Sep 20, 2017

Docker Parameterized Builds Using Git Tags, Part 1 of 2

When building a Docker image, sometimes it is desirable to have an image for different package versions or even different base images. One way to do this is with a branch for each version. This is manageable for a few versions with little changes. It quickly becomes hard to manage. If there are multiple packages the permutations become a nightmare.

We are going to look at using git tags to specify what the Dockerfile should build without branching. In fact, the same commit can be used to build many images. This approach assumes the build will take place on Docker Hub. Other tools can be used if the shell scripts described here can be called in the same way Docker Hub calls them.

This article heavily relies on shell script, specifically bash.

We’ll start with an image providing the Dropbox client. It is a good candidate because the client has few dependencies and a stable install process. It works nicely with using a git tag to publish a new version.

Docker Hub Hooks

Docker Hub will look for shell scripts in the hooks directory of your repo and use them to build and test your image instead of building from the Dockerfile directly. We will use the hooks/build script to build the image and hooks/test to test it. The full documentation for hooks is available at https://docs.docker.com/docker-cloud/builds/advanced/. We also want to run these scripts from our local machine, so we can’t assume the environment added by the Hub.

File tree

- Dockerfile
\ hooks
  - build  (1)
  - test  (2)
  - env  (3)
  1. build script
  2. test script
  3. shared environment script, explained later

Dockerfile with Build Args

The first case we’ll look at is a single package version where the image is built the same for all versions. We’ll pass the version into the Dockerfile via a build argument. Build arguments allow values to be passed from the docker build command into the Dockerfile. Defaults can be set in the Dockerfile.

Dockerfile

FROM alpine:3.5
ARG VERSION=33.4.23  (1)
ARG BUILD_DATE  (2)
...
  1. Using an = allows a default value to be set
  2. This needs to be specified in the docker build command

The command might look like:

docker build --build-arg BUILD_DATE=2017-09-01T10:00:00Z -t dropbox:33.4.23 .                 # (1)
docker build --build-arg BUILD_DATE=2017-09-01T10:00:00Z VERSION=34.4.20 -t dropbox:34.4.20 . # (2)
  1. The VERSION is not specified, so the default of 33.4.23 will be used
  2. The VERSION is specified, 34.4.20 will be built

Sharing the Environment

The build and test process provides information about the image via environment variables. We’re going to take those variables and do some processing to produce other variables, such as extracting the package version. The build and test scripts will both need access to similar information so we’ll put that into a hooks/env script and source it.

hooks/env

[ -n "$SOURCE_BRANCH" ]  || SOURCE_BRANCH=$(git symbolic-ref -q --short HEAD)  # (1)
[ -n "$GIT_SHA1" ]       || GIT_SHA1=$(git rev-parse -q HEAD)
 
if [[ "${SOURCE_BRANCH/-*/}" =~ ^[0-9][0-9.]*$ ]]; then  # (2)
	VERSION=${SOURCE_BRANCH/-*/}
fi
 
[ -n "$SOURCE_TYPE" ]        || SOURCE_TYPE=git  # (3)
[ -n "$DOCKERFILE_PATH" ]    || DOCKERFILE_PATH=.
[ -n "$IMAGE_NAME" ]         || IMAGE_NAME=dropbox-container:${SOURCE_BRANCH}
  1. These values are passed by the Hub, but if we are running locally we can get them from git.
  2. Parse the version out of the branch name ignoring any revision such as ‘-r1’ at the end. This will allow versions such as ‘1.2.3’, ‘1.2’, ‘1’, ‘1.2-r5’, but not ‘1.2beta’. If no version is found, we rely on the Dockerfile to provide a default.
  3. Set defaults for build arguments if we’re not running on the Hub

Building

The hooks/build script allows us to specify how the image is built. It gives us more freedom than simply building the Dockerfile. By the end of the script we’ll need to have built an image tagged with the value of the environment variable IMAGE_NAME, this is what the Hub will publish.

hooks/build

#!/bin/bash -xe
. ./hooks/env  # (1)
docker build \  # (2)
	--build-arg BUILD_DATE=`date -u +"%Y-%m-%dT%H:%M:%SZ"` \
	--build-arg "SOURCE_COMMIT=$GIT_SHA1" \
	--build-arg "DOCKERFILE_PATH=$DOCKERFILE_PATH" \
	--build-arg "SOURCE_TYPE=$SOURCE_TYPE" \
	${VERSION:+--build-arg "VERSION=$VERSION"} \  # (3)
	-t $IMAGE_NAME .  # (4)
  1. Use the hooks/env script to parse the VERSION.
  2. The Hub won’t build an image when it finds the build hook script, so we need to do it.
  3. The VERSION is passed into the Dockerfile here and sets the VERSION variable, without modifying the Dockerfile. This variable expansion construct will only include --build-arg ... if the VERSION variable is defined.
  4. We must tag the image with IMAGE_NAME for the Hub to publish it.

The script isn’t complicated, the main work is still done in the Dockerfile. The build hook allows us to parameterize the Dockerfile based on the tag (or branch) being built.

Testing

The test hook allows us to ensure the image is correct before publishing. The better the test, the less likely a broken image will be published to the Hub. The script can do most anything, the only requirement is that the exit code is 0 if the tests past.

hooks/test

#!/bin/bash -xe
. ./hooks/env # (1)
CID=$(docker run -d $IMAGE_NAME)  # (2)
sleep 10s
# Message is: Please visit https://www.dropbox.com/cli_link_nonce?nonce=_____ to link this device.
LOGS="$(docker logs $CID)"
docker rm -f $CID
echo $LOGS | grep -iq 'please visit.*to link'  # (3)
  1. Use the hooks/env script to give us the same environment as when we built the image.
  2. Run the new image in the background and keep the container ID so we can remove it later.
  3. When Dropbox starts on a fresh container, it needs to link to an account. There is a message output with instructions on how to connect an account. We grab the logs and use grep to look for the message. If the message is found, exit code 0 is returned.

Build and Test Locally

We should now have our Dockerfile building using a VERSION build argument, a hooks/env to determine the VERSION, a hooks/build script to build the image and a hooks/test script to test it. Go time!

$ ./hooks/build
...
$ ./hooks/test
...
$ echo $?
0  # (1)
  1. This is the success code from the test. 0 is success, anything else is a failure.

You should be able to run docker images and see your new image.

Official Build

Now it’s time to get your image on Docker Hub. Commit and push the repo. The automated build will see your scripts in the hooks directory and use them. The build log on the Hub includes both the build and test runs so you’ll need to read through it to find where the build ends and the tests begin.

Updating a Tag

This is the point we wanted to reach: releasing a new version of our image is as easy as tagging the repo. In our example, when Dropbox released version 34.4.22, we can release the image with:

Releasing our Dropbox image for version 34.4.22

$ git tag 34.4.22
$ git push --tags
Total 0 (delta 0), reused 0 (delta 0)
To bitbucket.org:double16/dropbox-container.git
 * [new tag]         34.4.22 -> 34.4.22

Done! You could also tag the repo from your hosting providers site, even using your phone.

If the build fails, we’ll need to do some extra work to get the tagged docker image to build. If something is wrong with the repo and needs to be fixed, the easiest is to add a revision to the tag, such as 34.4.22-r1. The hooks/env script we wrote removes the -r1 when building the VERSION variable. We could also move the tag, which generally is discouraged.

If this is a transient error, you can use remote build triggers documented at https://docs.docker.com/docker-hub/builds/#remote-build-triggers. When enabled, you can use curl or another similar utility to trigger the build to run again.

Conclusion

We walked through automating the build of a Docker image using the git tag as a version parameter. We looked at how to customize the build and test phases of the Docker Hub automated build. Now maintaining your image is easier and if something about the package changes, the test will fail and you’ll know to fix it.

This was a simple example. We can get more complicated by creating the Dockerfile dynamically in the hooks/build script. So we might want to compose several packages together so we have pre-built images to suit many needs. In part 2 of this post we’ll create an automated build that combines Gradle, Docker and Docker Compose. We can specify the combinations we want in the git tag. Taking it a step further, we’ll pull a list of versions of Gradle from the official Docker image and create permutations of versions and tag the repo. It’ll be fun 😉

About the Author

Patrick Double profile.

Patrick Double

Principal Technologist

I have been coding since 6th grade, circa 1986, professionally (i.e. college graduate) since 1998 when I graduated from the University of Nebraska-Lincoln. Most of my career has been in web applications using JEE. I work the entire stack from user interface to database.   I especially like solving application security and high availability problems.

One thought on “Docker Parameterized Builds Using Git Tags, Part 1 of 2

  1. Pascal says:

    This is solid! Would you share a repo using this method?

    I actually build my docker images via bash scripts but they are manually trigger. I still don’t see where to magik happen for docker hub to build a Dockerfile automatically.

    Thanks!

    1. This code was taken from https://bitbucket.org/double16/dropbox-container . Shame on me for not including in the post!

      You can create an automated build on Docker Hub by clicking on the Create menu in the top right and then choose Create Automated Build. Part of the process is to give your repository URL.

Leave a Reply

Your email address will not be published.

Related Blog Posts
Natively Compiled Java on Google App Engine
Google App Engine is a platform-as-a-service product that is marketed as a way to get your applications into the cloud without necessarily knowing all of the infrastructure bits and pieces to do so. Google App […]
Building Better Data Visualization Experiences: Part 2 of 2
If you don't have a Ph.D. in data science, the raw data might be difficult to comprehend. This is where data visualization comes in.
Unleashing Feature Flags onto Kafka Consumers
Feature flags are a tool to strategically enable or disable functionality at runtime. They are often used to drive different user experiences but can also be useful in real-time data systems. In this post, we’ll […]
A security model for developers
Software security is more important than ever, but developing secure applications is more confusing than ever. TLS, mTLS, RBAC, SAML, OAUTH, OWASP, GDPR, SASL, RSA, JWT, cookie, attack vector, DDoS, firewall, VPN, security groups, exploit, […]