Automating Terraform Projects with Jenkins
Terraform is tool for managing infrastructure as code. It has been a great success in extending the infrastructure knowledge to more team members (which is the whole point of DevOps!). One of my struggles though has been how to effectively manage Terraform projects in a Continuous Integration type workflow. There are a number of considerations when doing this:
- Only 1 change should be applied at a time
- Need to shared the current infrastructure state w/ all users
- Want to to be able to evaluate infrastructure Pull Requests just like with application code
Terraform solves a few of these issues by providing a mechanism for remote state storage. Hashicorp also has a hosted subscript service, Atlas, that can be used for managing Terraform projects. But the price point is often steep for many consumers.
Luckily, the Jenkins CI project has been working on a new mechanism for defining and executing work pipelines and it is now available in the v2.x release of Jenkins.
This post, will show you how to set up a Jenkins Pipeline for planning and applying your Terraform projects. This post was written against the following versions:
- Jenkins v2.2
- Terraform v0.6.16
Scripting Remote State Setup
Terraform doesn’t have a way of specifying the remote state configuration in the project files (as of v0.6.16). This causes some pain when trying to share a project amongst a team or with the CI server. I’ve found the following pattern to work fairly well.
First, every Terraform project is expected to have an executable init script in the root of the project.
This script sets up the remote state configuration for the project.
This is an example script I use that will configure the S3 backend for remote state.
#!/bin/bash -e PROJECT="$(basename `pwd`)" BUCKET="myorgs-infrastructure-state" init() { if [ -d .terraform ]; then if [ -e .terraform/terraform.tfstate ]; then echo "Remote state already exist!" if [ -z $IGNORE_INIT ]; then exit 1 fi fi fi terraform remote config \ -backend=s3 \ -backend-config="bucket=${BUCKET}" \ -backend-config="key=${PROJECT}/terraform.tfstate" \ -backend-config="region=us-east-1" } while getopts "i" opt; do case "$opt" in i) IGNORE_INIT="true" ;; esac done shift $((OPTIND-1)) init
Second, we establish the expectation with our team that upon checkout from the repository, this script is executed first to configure the remote state and any subsequent work is preceded by updating the remote state:
$ terraform remote pull
Adding Terraform to Jenkins
Adding Terraform to a Jenkins server is a simple as adding a Custom Tool.
First, go to Manage Jenkins | Global Tool Configuration screen on your Jenkins server (for v2.x). Under Custom Tool, select the Custom Tool installations… button, then Add Custom tool.
Configure the settings like so,
Creating a Jenkins Pipeline for Terraform
Now, that we have the tools in place, we can create a new Jenkins Pipeline job to execute our workflow.
The flow of our pipeline should execute a Terraform plan and then prompt a user to either apply or discard the plan.
To create the job, select New Item from the sidebar, enter the project name, and select Pipeline as the job type.
In the Pipeline box, add the following as the Script definition:
node { // Mark the code checkout 'Checkout'.... stage 'Checkout' // // Get some code from a GitHub repository git url: 'git@github.com:myorg/infrastructure.git' // Get the Terraform tool. def tfHome = tool name: 'Terraform', type: 'com.cloudbees.jenkins.plugins.customtools.CustomTool' env.PATH = "${tfHome}:${env.PATH}" wrap([$class: 'AnsiColorBuildWrapper', colorMapName: 'xterm']) { // Mark the code build 'plan'.... stage name: 'Plan', concurrency: 1 // Output Terraform version sh "terraform --version" //Remove the terraform state file so we always start from a clean state if (fileExists(".terraform/terraform.tfstate")) { sh "rm -rf .terraform/terraform.tfstate" } if (fileExists("status")) { sh "rm status" } sh "./init" sh "terraform get" sh "set +e; terraform plan -out=plan.out -detailed-exitcode; echo \$? > status" def exitCode = readFile('status').trim() def apply = false echo "Terraform Plan Exit Code: ${exitCode}" if (exitCode == "0") { currentBuild.result = 'SUCCESS' } if (exitCode == "1") { slackSend channel: '#ci', color: '#0080ff', message: "Plan Failed: ${env.JOB_NAME} - ${env.BUILD_NUMBER} ()" currentBuild.result = 'FAILURE' } if (exitCode == "2") { stash name: "plan", includes: "plan.out" slackSend channel: '#ci', color: 'good', message: "Plan Awaiting Approval: ${env.JOB_NAME} - ${env.BUILD_NUMBER} ()" try { input message: 'Apply Plan?', ok: 'Apply' apply = true } catch (err) { slackSend channel: '#ci', color: 'warning', message: "Plan Discarded: ${env.JOB_NAME} - ${env.BUILD_NUMBER} ()" apply = false currentBuild.result = 'UNSTABLE' } } if (apply) { stage name: 'Apply', concurrency: 1 unstash 'plan' if (fileExists("status.apply")) { sh "rm status.apply" } sh 'set +e; terraform apply plan.out; echo \$? > status.apply' def applyExitCode = readFile('status.apply').trim() if (applyExitCode == "0") { slackSend channel: '#ci', color: 'good', message: "Changes Applied ${env.JOB_NAME} - ${env.BUILD_NUMBER} ()" } else { slackSend channel: '#ci', color: 'danger', message: "Apply Failed: ${env.JOB_NAME} - ${env.BUILD_NUMBER} ()" currentBuild.result = 'FAILURE' } } } }
Once defined, run the job once to get Jenkins to setup the SCM configuration. The job will display like so:
Wrapping Up
The above steps shows you how to setup up a Jenkins Pipeline job for planning and applying changes to the master branch of a Terraform project repository. With a few minor tweaks, you can also create a pipeline job that performs only a plan on Pull Requests that are submitted against the repository. Again, this is useful for team collaboration as infrastructure changes can be submitted and reviewed as Pull Requests and feedback can be gained regarding the exact changes will be applied.
Happy Terraforming!
Update (7/7/2016)
The pipeline script above has been updated to reflect the use of set +e;
in the Terraform shell scripts to prevent Jenkins from exiting the pipeline early on error codes from Terraform.
Jenkins 2.12 bails on the ‘terraform plan -detailed-exitcode’ for any non-0 exit code. It doesn’t write out the status file in those cases. Is that a new “feature”?
Hi mrz – thanks for pointing that out.
This was an issue that we encountered as well and the fix didn’t make it into this post. I’ve updated the post to reflect the change. The trick is to use ‘set +e;’ at the beginning of the Terraform command to disable Jenkins from prematurely exiting on non-zero command exit codes.
I also facing this issue. Till I am not using set +e in my shell script. And Jenkins is reading SUCCESS even though my `terraform apply` fails, Should i use set +e all time and make it as thumb rule?
🙂
Please note in the above script if you see “>” it should be an explicit “>”
Hi John,
Under what license should the code you’ve posted here be considered?
Thanks!
Hi Brad!
Thanks for the question, you can use this code in accordance with the WTFPL 🙂
Beautiful
Hi John, does this terraform support exist in Jenkins 2.0 as well, or is 2.2 the first supported version?
Hi there I got an error saying ERROR: No com.cloudbees.jenkins.plugins.customtools.CustomTool named Terraform found
Finished: FAILURE and it failed at checklout level
jenkins does not recognize > and status of the below code:
sh “set +e; terraform plan -out=plan.out -detailed-exitcode; echo \$? > status”
Error: gt: command not found
status: command not found