Managing Rancher container platform with Terraform

This blog post was originally posted at http://rancher.com/managing-container-clusters-terraform-rancher.

Infrastructure as code is a practice of codifying and automating the deployment and management of infrastructure with tooling. This allows for testing, reviewing, approving, and deploying infrastructure changes with the same processes and tools as application code.

Terraform from Hashicorp is a tool for abstracting service and provider APIs into declarative configuration files. It then tracks the state of the infrastructure and converges it to match the specified configuration.

Terraform ships with built-in support for a variety of cloud providers (AWS, CenturyLink Cloud, Google Cloud, Microsoft Azure, OpenStack, VMware vSphere, etc.) and other services such as BitBucket, GitHub, Fastly, Heroku DNSimple, Rancher, etc. The full list of providers can be found at https://www.terraform.io/docs/providers/index.html

Beginning with version 0.8.0, Terraform is able to manage resources within a Rancher cluster using the built-in provider. The full list of supported resources for Rancher can be found here. Any issues should be reported at Github.

Setting up

This post assumes that a Rancher server is available for testing against and the latest version of Terraform is installed. If a Rancher server is not available, one can be started locally using the following Docker command:

$ docker run -d -p 8080:8080 rancher/server:v1.3.3

Configuring the Rancher provider

The first step to managing a Rancher server with Terraform is to configure the <font color="red">rancher</font> provider for Terraform. Create a working directory for this example and place the following code in the <font color="red">rancher.tf</font> within the directory.

provider "rancher" {
  api_url = "http://localhost:8080" //<1>
  access_key = "" //<2>
  secret_key = ""
}

The only required configuration for the <font color="red">rancher</font> provider is <font color="red">api_key</font> (<1>). This should be configured to point to the Rancher server location. Optionally, the API access and secret keys can be configured with the <font color="red">access_key</font> and <font color="red">secret_key</font> properties as shown (<2>). If the Rancher server does _not_ have access control configured, then these settings can be omitted. If access control is enabled, then API credentials are required. Credentials can be created by following the instructions here

Rancher is also able to access these properties using the environment variables <font color="red">RANCHER_URL</font>, <font color="red">RANCHER_ACCESS_KEY</font>, and <font color="red">RANCHER_SECRET_KEY</font> respectively. For security purposes, it is advisable to use the environment variables or Terraform variables to provide credentials.

Creating environments

Now that the provider is configured, Terraform can create new resources. Start by creating a new <font color="red">Environment</font> within Rancher by placing the following code in the <font color="red">demo.tf</font> file in the working directory.

resource "rancher_environment" "demo" { // <1>
  name = "blog-demo" // <2>
  description = "Demonstration environment" // <3>
  orchestration = "cattle" // <4>
}

First, a new resource is declared with the type of <font color="red">rancher_environment</font> and a logical name of <font color="red">demo</font> (<1>). This is used for referencing this resource within the Terraform configuration. The <font color="red">name</font> property is required for the environment (<2>) and specifies the name of the environment in Rancher. Optionally, a description can be provided (<3>). Finally, the orchestration engine to use for the environment is specified (<4>). This value can be <font color="red">cattle</font>, <font color="red">kubernetes</font>, <font color="red">swarm</font>, or <font color="red">mesos</font>. If omitted, the value will default to <font color="red">cattle</font>.

Once specified, Terraform can dry-run and apply the configurations by executing <font color="red">terraform plan</font> and <font color="red">terraform apply</font> respectively.

$ terraform plan
Refreshing Terraform state in-memory prior to plan...
The refreshed state will be used to calculate this plan, but will not be
persisted to local or remote state storage.
 
The Terraform execution plan has been generated and is shown below.
Resources are shown in alphabetical order for quick scanning. Green resources
will be created (or destroyed and then created if an existing resource
exists), yellow resources are being changed in-place, and red resources
will be destroyed. Cyan entries are data sources to be read.
 
Note: You didn't specify an "-out" parameter to save this plan, so when
"apply" is called, Terraform can't guarantee this is what will execute.
 
+ rancher_environment.demo
  description: "Demonstration environment"
  name: "demo"
  orchestration: "cattle"
 
Plan: 1 to add, 0 to change, 0 to destroy.
$ terraform apply
rancher_environment.demo: Creating...
  description: "" => "Demonstration environment"
  name: "" => "demo"
  orchestration: "" => "cattle"
rancher_environment.demo: Creation complete
 
Apply complete! Resources: 1 added, 0 changed, 0 destroyed.
 
The state of your infrastructure has been saved to the path
below. This state is required to modify and destroy your
infrastructure, so keep it safe. To inspect the complete state
use the `terraform show` command.
 
State path: terraform.tfstate

Getting host configuration information

Terraform can create the registration token information for a host to join and environment using the <font color="red">rancher_registration_token</font> resource.
Add the following to the <font color="red">demo.tf</font> file.

resource "rancher_registration_token" "demo-token" { // <1>
  environment_id = "${rancher_environment.demo.id}" // <2>
  name = "demo-token" // <3>
  description = "Host registration token for Demo environment"
}

The environment is declared by creating a new <font color="red">rancher_registration_token</font> resource with the logical name of <font color="red">demo-token</font> (<1>). The token is declared to be created in the <font color="red">demo</font> environment by populating the <font color="red">environment_id</font> property with the <font color="red">id</font> property from the previously created environment (<2>). This uses Terraform’s builtin interpolation and resource dependency system. Finally, the <font color="red">name</font> is a required property (<3>) and an optional <font color="red">description</font> can be provided.

Run <font color="red">terraform plan</font> and <font color="red">terraform apply</font> to view and apply the changes once again. Note that any IDs in the sample output may be different.

$ terraform plan
Refreshing Terraform state in-memory prior to plan...
The refreshed state will be used to calculate this plan, but will not be
persisted to local or remote state storage.
 
rancher_environment.demo: Refreshing state... (ID: 1a7)
The Terraform execution plan has been generated and is shown below.
Resources are shown in alphabetical order for quick scanning. Green resources
will be created (or destroyed and then created if an existing resource
exists), yellow resources are being changed in-place, and red resources
will be destroyed. Cyan entries are data sources to be read.
 
Note: You didn't specify an "-out" parameter to save this plan, so when
"apply" is called, Terraform can't guarantee this is what will execute.
 
+ rancher_registration_token.demo-token
  command: ""
  description: "Host registration token for Demo environment"
  environment_id: "1a7"
  name: "demo-token"
  registration_url: ""
  token: ""
 
Plan: 1 to add, 0 to change, 0 to destroy.
$ terraform apply
rancher_environment.demo: Refreshing state... (ID: 1a7)
rancher_registration_token.demo-token: Creating...
  command: "" => ""
  description: "" => "Host registration token for Demo environment"
  environment_id: "" => "1a7"
  name: "" => "demo-token"
  registration_url: "" => ""
  token: "" => ""
rancher_registration_token.demo-token: Creation complete
 
Apply complete! Resources: 1 added, 0 changed, 0 destroyed.
 
The state of your infrastructure has been saved to the path
below. This state is required to modify and destroy your
infrastructure, so keep it safe. To inspect the complete state
use the `terraform show` command.
 
State path: terraform.tfstate

The <font color="red">rancher_registration_token</font> resource outputs a few useful properties. First it captures the entire <font color="red">command</font> value. This is the value that is visible in the Rancher UI when adding a Custom Host. It contains the entire command for staring the Rancher agent on a machine with Docker to join this environment.

Additionally, the <font color="red">registration_url</font> and <font color="red">token</font> properties are captured. These contain the full URL for the Rancher agent to connect to as specified in the the <font color="red">command</font> property and the environment specific path token from that URL
respectively.

To view the host command, an <font color="red">output</font> can be configured in Terraform to capture the value and write it to the console. Add the following block to the <font color="red">demo.tf</font> file.

output "rancher_agent_command" {
  value = "${rancher_registration_token.demo-token.command}"
}

Now run a <font color="red">terraform apply</font> to configure the <font color="red">output</font> value.

$ terraform apply
rancher_environment.demo: Refreshing state... (ID: 1a7)
rancher_registration_token.demo-token: Refreshing state... (ID: 1c2)
 
Apply complete! Resources: 0 added, 0 changed, 0 destroyed.
 
Outputs:
 
rancher_agent_command = sudo docker run -d --privileged -v /var/run/docker.sock:/var/run/docker.sock -v /var/lib/rancher:/var/lib/rancher rancher/agent:v1.1.3 http://localhost:8080/v1/scripts/A17BDBB62228B4B18B4A:1485619200000:undsWydVkrYSHqd2NNU5fVDdx7s

The <font color="red">rancher_registration_token</font> outputs can be used to configure the launch
parameters for other resources. For example, if Amazon Web Services is being
used to create an Autoscaling Group of instances to join the cluster, the
registration command can be added to the user data for like so.

resource "aws_launch_configuration" "demo-hosts" {
  name = "demo_hosts"
  image_id = "ami-6edd3078" // <1>
  instance_type = "t2.micro"
  root_block_device {
    volume_type = "gp2"
    volume_size = 8
    delete_on_termination = true
  }
  user_data = <<EOF
#cloud-config
packages:
  - ntp
write_files:
  - path: /opt/docker/install.sh // <2>
    permissions: "0755"
    content: |
      #!/bin/bash
      DOCKER_VERSION=$1
      apt-key adv --keyserver hkp://p80.pool.sks-keyservers.net:80 --recv-keys 58118E89F3A912897C070ADBF76221572C52609D
      mkdir -p /etc/apt/sources.list.d/
      echo "deb https://apt.dockerproject.org/repo ubuntu-xenial main" > /etc/apt/sources.list.d/docker.list
      apt-get update
      apt-get install -y linux-image-extra-virtual linux-image-extra-$(uname -r) docker-engine=$DOCKER_VERSION
 
  - path: /opt/rancher/rancher.sh // <3>
    permissions: "0755"
    content: |
      #!/bin/sh
      set -e
      umask 077
 
      ${rancher_registration_token.demo-token.command}
 
runcmd:
  - [ cloud-init-per, once, docker, /opt/docker/install.sh, "1.13.0-0~ubuntu-xenial" ]
  - [ cloud-init-per, once, rancher, /opt/rancher/rancher.sh ]
EOF
}

This block creates a launch configuration for an Ubuntu Xenial image in the <font color="red">us-east-1</font> region in AWS (<1>). It adds boot time scripts to install Docker (<2>), and to start the Rancher agent using the command string from the registration token. Notice that the command property is interpolated into the user data as part of a script file that is written out (<3>). This block is a example and the same pattern can be used for various operating systems or cloud providers.

Controlling Rancher stacks with Terraform

Terraform can additionally create and manage stacks within a Rancher environment. The entire <font color="red">docker-compose.yml</font> and <font color="red">rancher-compose.yml</font> can be specified for the stack, or the Rancher catalog can be used to specify the entry and answers to create a stack. Add the following block to the <font color="red">demo.tf</font> file to create a new stack from the community catalog entry for the Ghost blogging platform.

resource "rancher_stack" "ghost" { // <1>
  environment_id = "${rancher_environment.demo.id}" // <2>
  name = "ghost" // <3>
  description = "Ghost demo stack"
  catalog_id = "community:ghost:0" // <4>
  scope = "user"
  start_on_create = true
  environment { // <5>
    public_port = "80"
  }
}

The block adds a new <font color="red">rancher_stack</font> resource with the logical name of <font color="red">ghost</font> (<1>) within the <font color="red">demo</font> environment previously created (<2>). It specifies the name and description (optional) for the stack (<3>). It then configures the stack to be created from the entry named <font color="red">ghost</font> in the <font color="red">community</font> catalog using the version <font color="red">0</font> of the template (<4>). These values can be discovered by utilizing the “View in API” feature of Rancher. Finally, it configures the answers to the questions defined in the catalog template using the <font color="red">environment</font> block (<5>). Run <font color="red">terraform apply</font> to create the stack in Rancher.

$ terraform apply
rancher_environment.demo: Refreshing state... (ID: 1a7)
rancher_registration_token.demo-token: Refreshing state... (ID: 1c2)
rancher_stack.ghost: Creating...
  catalog_id: "" => "community:ghost:0"
  description: "" => "Ghost demo stack"
  environment.%: "" => "1"
  environment.public_port: "" => "80"
  environment_id: "" => "1a7"
  name: "" => "ghost"
  rendered_docker_compose: "" => ""
  rendered_rancher_compose: "" => ""
  scope: "" => "user"
  start_on_create: "" => "true"
rancher_stack.ghost: Creation complete
 
Apply complete! Resources: 1 added, 0 changed, 0 destroyed.
 
The state of your infrastructure has been saved to the path
below. This state is required to modify and destroy your
infrastructure, so keep it safe. To inspect the complete state
use the `terraform show` command.
 
State path: terraform.tfstate
 
Outputs:
 
rancher_agent_command = sudo docker run -d --privileged -v /var/run/docker.sock:/var/run/docker.sock -v /var/lib/rancher:/var/lib/rancher rancher/agent:v1.1.3 http://localhost:8080/v1/scripts/A17BDBB62228B4B18B4A:1485622800000:E0L4yrjYajfxbg7mKSNxLZGup0

Open your the Rancher dashboard, switch the <font color="red">demo</font> environment and see that the
<font color="red">ghost</font> stack is created and active:

Stack created by Terraform in Rancher

Stack created by Terraform in Rancher

Summary

Terraform can be used to codify infrastructure across many platforms including Rancher. Using a tool like Terraform allows for easily repeating configuration and setup of infrastructure. It also allows for incorporating infrastructure changes into standard development practices and processes such as versioning, reviews, and promotions.

The full list of supported resources for Rancher can be found here. Any issues with the Rancher provider can be reported on Github.

About the Author

Object Partners profile.

One thought on “Managing Rancher container platform with Terraform

  1. Ciaran says:

    Great article, thank you!

Leave a Reply

Your email address will not be published. Required fields are marked *

Related Blog Posts
Using Conftest to Validate Configuration Files
Conftest is a utility within the Open Policy Agent ecosystem that helps simplify writing validation tests against configuration files. In a previous blog post, I wrote about using the Open Policy Agent utility directly to […]
SwiftGen with Image & Color Asset Catalogs
You might remember back in 2015 when iOS 9 was introduced, and we were finally given a way to manage all of our assets in one place with Asset Catalogs. A few years later, support […]
Tracking Original URL Through Authentication
If you read my other post about refreshing AWS tokens, then you probably have a use case for keeping track of the original requested resource while the user goes through authentication so you can route […]
Using Spring Beans in a Kafka Streams ExceptionHandler
There are many things to know before diving into Kafka Streams. If you haven’t already, check out these 5 things as a starting point. Bullet 2 mentions designing for exceptions. Ironically, this seems to be […]