Automating Terraform Projects with Jenkins

Terraform is tool for managing infrastructure as code. It has been a great success in extending the infrastructure knowledge to more team members (which is the whole point of DevOps!). One of my struggles though has been how to effectively manage Terraform projects in a Continuous Integration type workflow. There are a number of considerations when doing this:

  1. Only 1 change should be applied at a time
  2. Need to shared the current infrastructure state w/ all users
  3. Want to to be able to evaluate infrastructure Pull Requests just like with application code

Terraform solves a few of these issues by providing a mechanism for remote state storage. Hashicorp also has a hosted subscript service, Atlas, that can be used for managing Terraform projects. But the price point is often steep for many consumers.

Luckily, the Jenkins CI project has been working on a new mechanism for defining and executing work pipelines and it is now available in the v2.x release of Jenkins.

This post, will show you how to set up a Jenkins Pipeline for planning and applying your Terraform projects. This post was written against the following versions:

  • Jenkins v2.2
  • Terraform v0.6.16

Scripting Remote State Setup

Terraform doesn’t have a way of specifying the remote state configuration in the project files (as of v0.6.16). This causes some pain when trying to share a project amongst a team or with the CI server. I’ve found the following pattern to work fairly well.

First, every Terraform project is expected to have an executable init script in the root of the project.
This script sets up the remote state configuration for the project.

This is an example script I use that will configure the S3 backend for remote state.

#!/bin/bash -e
 
PROJECT="$(basename `pwd`)"
BUCKET="myorgs-infrastructure-state"
 
init() {
  if [ -d .terraform ]; then
    if [ -e .terraform/terraform.tfstate ]; then
      echo "Remote state already exist!"
      if [ -z $IGNORE_INIT ]; then
        exit 1
      fi
    fi
  fi
 
 
  terraform remote config \
    -backend=s3 \
    -backend-config="bucket=${BUCKET}" \
    -backend-config="key=${PROJECT}/terraform.tfstate" \
    -backend-config="region=us-east-1"
 
}
 
while getopts "i" opt; do
  case "$opt" in
    i)
      IGNORE_INIT="true"
      ;;
  esac
done
 
shift $((OPTIND-1))
 
init

Second, we establish the expectation with our team that upon checkout from the repository, this script is executed first to configure the remote state and any subsequent work is preceded by updating the remote state:

$ terraform remote pull

Adding Terraform to Jenkins

Adding Terraform to a Jenkins server is a simple as adding a Custom Tool.

First, go to Manage Jenkins | Global Tool Configuration screen on your Jenkins server (for v2.x). Under Custom Tool, select the Custom Tool installations… button, then Add Custom tool.

Configure the settings like so,
Terraform Custom Tool Setup

Creating a Jenkins Pipeline for Terraform

Now, that we have the tools in place, we can create a new Jenkins Pipeline job to execute our workflow.

The flow of our pipeline should execute a Terraform plan and then prompt a user to either apply or discard the plan.
TF Jenkins Pipeline

To create the job, select New Item from the sidebar, enter the project name, and select Pipeline as the job type.

In the Pipeline box, add the following as the Script definition:

node {
 
    // Mark the code checkout 'Checkout'....
    stage 'Checkout'
 
    // // Get some code from a GitHub repository
    git url: 'git@github.com:myorg/infrastructure.git'
 
    // Get the Terraform tool.
    def tfHome = tool name: 'Terraform', type: 'com.cloudbees.jenkins.plugins.customtools.CustomTool'
    env.PATH = "${tfHome}:${env.PATH}"
    wrap([$class: 'AnsiColorBuildWrapper', colorMapName: 'xterm']) {
 
            // Mark the code build 'plan'....
            stage name: 'Plan', concurrency: 1
            // Output Terraform version
            sh "terraform --version"
            //Remove the terraform state file so we always start from a clean state
            if (fileExists(".terraform/terraform.tfstate")) {
                sh "rm -rf .terraform/terraform.tfstate"
            }
            if (fileExists("status")) {
                sh "rm status"
            }
            sh "./init"
            sh "terraform get"
            sh "set +e; terraform plan -out=plan.out -detailed-exitcode; echo \$? > status"
            def exitCode = readFile('status').trim()
            def apply = false
            echo "Terraform Plan Exit Code: ${exitCode}"
            if (exitCode == "0") {
                currentBuild.result = 'SUCCESS'
            }
            if (exitCode == "1") {
                slackSend channel: '#ci', color: '#0080ff', message: "Plan Failed: ${env.JOB_NAME} - ${env.BUILD_NUMBER} ()"
                currentBuild.result = 'FAILURE'
            }
            if (exitCode == "2") {
                stash name: "plan", includes: "plan.out"
                slackSend channel: '#ci', color: 'good', message: "Plan Awaiting Approval: ${env.JOB_NAME} - ${env.BUILD_NUMBER} ()"
                try {
                    input message: 'Apply Plan?', ok: 'Apply'
                    apply = true
                } catch (err) {
                    slackSend channel: '#ci', color: 'warning', message: "Plan Discarded: ${env.JOB_NAME} - ${env.BUILD_NUMBER} ()"
                    apply = false
                    currentBuild.result = 'UNSTABLE'
                }
            }
 
            if (apply) {
                stage name: 'Apply', concurrency: 1
                unstash 'plan'
                if (fileExists("status.apply")) {
                    sh "rm status.apply"
                }
                sh 'set +e; terraform apply plan.out; echo \$? > status.apply'
                def applyExitCode = readFile('status.apply').trim()
                if (applyExitCode == "0") {
                    slackSend channel: '#ci', color: 'good', message: "Changes Applied ${env.JOB_NAME} - ${env.BUILD_NUMBER} ()"    
                } else {
                    slackSend channel: '#ci', color: 'danger', message: "Apply Failed: ${env.JOB_NAME} - ${env.BUILD_NUMBER} ()"
                    currentBuild.result = 'FAILURE'
                }
            }
    }
}

Once defined, run the job once to get Jenkins to setup the SCM configuration. The job will display like so:
Jenkins Terraform Job

Wrapping Up

The above steps shows you how to setup up a Jenkins Pipeline job for planning and applying changes to the master branch of a Terraform project repository. With a few minor tweaks, you can also create a pipeline job that performs only a plan on Pull Requests that are submitted against the repository. Again, this is useful for team collaboration as infrastructure changes can be submitted and reviewed as Pull Requests and feedback can be gained regarding the exact changes will be applied.

Happy Terraforming!

Update (7/7/2016)
The pipeline script above has been updated to reflect the use of set +e; in the Terraform shell scripts to prevent Jenkins from exiting the pipeline early on error codes from Terraform.

About the Author

Object Partners profile.

One thought on “Automating Terraform Projects with Jenkins

  1. mrz says:

    Jenkins 2.12 bails on the ‘terraform plan -detailed-exitcode’ for any non-0 exit code. It doesn’t write out the status file in those cases. Is that a new “feature”?

    1. John Engelman says:

      Hi mrz – thanks for pointing that out.
      This was an issue that we encountered as well and the fix didn’t make it into this post. I’ve updated the post to reflect the change. The trick is to use ‘set +e;’ at the beginning of the Terraform command to disable Jenkins from prematurely exiting on non-zero command exit codes.

      1. Jayesh Dhandha says:

        I also facing this issue. Till I am not using set +e in my shell script. And Jenkins is reading SUCCESS even though my `terraform apply` fails, Should i use set +e all time and make it as thumb rule?
        🙂

  2. John Engelman says:

    Please note in the above script if you see “>” it should be an explicit “>”

  3. Brad says:

    Hi John,

    Under what license should the code you’ve posted here be considered?

    Thanks!

    1. John Engelman says:

      Hi Brad!
      Thanks for the question, you can use this code in accordance with the WTFPL 🙂

      1. Ron says:

        Beautiful

  4. Brian Pulliam says:

    Hi John, does this terraform support exist in Jenkins 2.0 as well, or is 2.2 the first supported version?

  5. Payal chhillar says:

    Hi there I got an error saying ERROR: No com.cloudbees.jenkins.plugins.customtools.CustomTool named Terraform found
    Finished: FAILURE and it failed at checklout level

  6. smi says:

    jenkins does not recognize &gt and status of the below code:
    sh “set +e; terraform plan -out=plan.out -detailed-exitcode; echo \$? > status”
    Error: gt: command not found
    status: command not found

Leave a Reply to John Engelman Cancel reply

Your email address will not be published.

Related Blog Posts
Natively Compiled Java on Google App Engine
Google App Engine is a platform-as-a-service product that is marketed as a way to get your applications into the cloud without necessarily knowing all of the infrastructure bits and pieces to do so. Google App […]
Building Better Data Visualization Experiences: Part 2 of 2
If you don't have a Ph.D. in data science, the raw data might be difficult to comprehend. This is where data visualization comes in.
Unleashing Feature Flags onto Kafka Consumers
Feature flags are a tool to strategically enable or disable functionality at runtime. They are often used to drive different user experiences but can also be useful in real-time data systems. In this post, we’ll […]
A security model for developers
Software security is more important than ever, but developing secure applications is more confusing than ever. TLS, mTLS, RBAC, SAML, OAUTH, OWASP, GDPR, SASL, RSA, JWT, cookie, attack vector, DDoS, firewall, VPN, security groups, exploit, […]