Jan 14, 2020

Keeping Secrets Out of Terraform State

There are many instances where you will want to create resources via Terraform with secrets that you just don’t want anyone to see. These could be IAM credentials, certificates, RDS DB credentials, etc. One problem with creating these resources via Terraform is that the statefile is stored in plain text. Yes, you could treat that statefile as a secret itself, but that may not be good enough depending on your security requirements. For example, you may want to be able to share that statefile with other users that are not authorized to view the secret’s values.

I will show a simple example of one way solve this problem. It may seem obvious, but the various cloud provider’s SDKs and APIs usually include a way to set these values. I’ll show here how to set up a new RDS instance and keep its master password from prying eyes.

First, I will create an RDS DB instance.

resource "aws_db_instance" "default" {
  name                 = "postgres_db"
  engine               = "postgres"
  instance_class       = "db.t2.micro"
  allocated_storage    = 20
  storage_type         = "gp2"
  username             = "foo"
  password             = "password123"

The password here is just a placeholder to initialize the instance. Next, I’ll use the boto3 Python library to update that password to a more secure secret.

#!/usr/env/bin python3
import argparse
import boto3
import secrets
import string
import time
parser = argparse.ArgumentParser(description='Override Terraform password')
parser.add_argument('--db_identifier', type=str, required=True, help="RDS DB instance identifier")
parser.add_argument('--secret_name', type=str, default='db_password', required=False, help="Name for SSM Parameter")
parser.add_argument('--region', type=str, default='us-east-1', required=False, help="AWS region name")
args = parser.parse_args()
def generate_password(string_length=10):
    """Generate a secure random string of letters, digits and special characters """
    password_characters = string.ascii_letters + string.digits + "!#$%^&*()"
    return ''.join(secrets.choice(password_characters) for i in range(string_length))
db_id = args.db_identifier
db_password = generate_password(16)
secret_name = args.secret_name
region = args.region
rds = boto3.client('rds', region_name=region)
db = rds.describe_db_instances(DBInstanceIdentifier=db_id)
while rds.describe_db_instances(DBInstanceIdentifier=db_id)["DBInstances"][0]["DBInstanceStatus"] != "available":
response = rds.modify_db_instance(DBInstanceIdentifier=db_id, MasterUserPassword=db_password, ApplyImmediately=True)
if response:
    print(f"Updated RDS password for {db_id}")

Using the script above, we will no longer have our “real” password stored in state. However, if we ever want to log into this DB we will need to store this somewhere else. This could be whatever tooling you use to manage secrets (Ex. Vault, CyberArk, etc.). If we want to keep this example native to AWS, we could store it in SSM Parameter Store by adding the following to our Python script:

ssm = boto3.client('ssm')
    Description=f'DB Password for {db_id}',

All that’s left is getting Terraform to handle these actions through a null_resource. This could be a local_exec or remote_exec depending where you have dependencies available for the script.

resource "null_resource" "update_password" {
  provisioner "local-exec" {
    command = "python3 update_password.py --db_identifier=${aws_db_instance.default.identifier}"

This Python script above is most useful if you need to store these secrets somewhere outside of AWS, rather than putting them in Parameter Store. A better AWS alternative might be to utilize AWS Secrets Manager to generate a password for RDS. Secrets Manager can also handle automatic rotation. The only real trade off with using Secrets Manager is that you will incur cost.

You can view my directory structure and full text of this example on GitHub here

I hope this sparks some creative ways to maintain your secrecy!

About the Author

Andy Haynssen profile.

Andy Haynssen

Principal Technologist

Andy is a creative person with a passion for software and technology! He has experience in building enterprise platforms with a focus on Infrastructure as Code, public cloud, and Continuous Integration/Continuous Delivery. He has been focusing in the DevOps space for the past 5+ years. In his free time, Andy enjoys creating music, woodworking, and spending time outdoors.

Leave a Reply

Your email address will not be published.

Related Blog Posts
Building Better Data Visualization Experiences: Part 1 of 2
Through direct experience with data scientists, business analysts, lab technicians, as well as other UX professionals, I have found that we need a better understanding of the people who will be using our data visualization products in order to build them. Creating a product utilizing data with the goal of providing insight is fundamentally different from a typical user-centric web experience, although traditional UX process methods can help.
Kafka Schema Evolution With Java Spring Boot and Protobuf
In this blog I will be demonstrating Kafka schema evolution with Java, Spring Boot and Protobuf.  This app is for tutorial purposes, so there will be instances where a refactor could happen. I tried to […]
Redis Bitmaps: Storing state in small places
Redis is a popular open source in-memory data store that supports all kinds of abstract data structures. In this post and in an accompanying example Java project, I am going to explore two great use […]
Let’s build a WordPress & Kernel updated AMI with Packer
First, let’s start with What is an AMI? An Amazon Machine Image (AMI) is a master image for the creation of virtual servers in an AWS environment. The machine images are like templates that are configured with […]