Jan 14, 2020

Keeping Secrets Out of Terraform State

There are many instances where you will want to create resources via Terraform with secrets that you just don’t want anyone to see. These could be IAM credentials, certificates, RDS DB credentials, etc. One problem with creating these resources via Terraform is that the statefile is stored in plain text. Yes, you could treat that statefile as a secret itself, but that may not be good enough depending on your security requirements. For example, you may want to be able to share that statefile with other users that are not authorized to view the secret’s values.

I will show a simple example of one way solve this problem. It may seem obvious, but the various cloud provider’s SDKs and APIs usually include a way to set these values. I’ll show here how to set up a new RDS instance and keep its master password from prying eyes.

First, I will create an RDS DB instance.

resource "aws_db_instance" "default" {
  name                 = "postgres_db"
  engine               = "postgres"
  instance_class       = "db.t2.micro"
  allocated_storage    = 20
  storage_type         = "gp2"
  username             = "foo"
  password             = "password123"
}

The password here is just a placeholder to initialize the instance. Next, I’ll use the boto3 Python library to update that password to a more secure secret.

#!/usr/env/bin python3
 
import argparse
import boto3
import secrets
import string
import time
 
parser = argparse.ArgumentParser(description='Override Terraform password')
parser.add_argument('--db_identifier', type=str, required=True, help="RDS DB instance identifier")
parser.add_argument('--secret_name', type=str, default='db_password', required=False, help="Name for SSM Parameter")
parser.add_argument('--region', type=str, default='us-east-1', required=False, help="AWS region name")
args = parser.parse_args()
 
 
def generate_password(string_length=10):
    """Generate a secure random string of letters, digits and special characters """
    password_characters = string.ascii_letters + string.digits + "!#$%^&*()"
    return ''.join(secrets.choice(password_characters) for i in range(string_length))
 
 
db_id = args.db_identifier
db_password = generate_password(16)
secret_name = args.secret_name
region = args.region
 
rds = boto3.client('rds', region_name=region)
db = rds.describe_db_instances(DBInstanceIdentifier=db_id)
while rds.describe_db_instances(DBInstanceIdentifier=db_id)["DBInstances"][0]["DBInstanceStatus"] != "available":
    time.sleep(3)
response = rds.modify_db_instance(DBInstanceIdentifier=db_id, MasterUserPassword=db_password, ApplyImmediately=True)
if response:
    print(f"Updated RDS password for {db_id}")

Using the script above, we will no longer have our “real” password stored in state. However, if we ever want to log into this DB we will need to store this somewhere else. This could be whatever tooling you use to manage secrets (Ex. Vault, CyberArk, etc.). If we want to keep this example native to AWS, we could store it in SSM Parameter Store by adding the following to our Python script:

ssm = boto3.client('ssm')
ssm.put_parameter(
    Name=secret_name,
    Description=f'DB Password for {db_id}',
    Type='SecureString',
    Value=db_password,
    Overwrite=True
)

All that’s left is getting Terraform to handle these actions through a null_resource. This could be a local_exec or remote_exec depending where you have dependencies available for the script.

resource "null_resource" "update_password" {
  provisioner "local-exec" {
    command = "python3 update_password.py --db_identifier=${aws_db_instance.default.identifier}"
  }
}

This Python script above is most useful if you need to store these secrets somewhere outside of AWS, rather than putting them in Parameter Store. A better AWS alternative might be to utilize AWS Secrets Manager to generate a password for RDS. Secrets Manager can also handle automatic rotation. The only real trade off with using Secrets Manager is that you will incur cost.

You can view my directory structure and full text of this example on GitHub here

I hope this sparks some creative ways to maintain your secrecy!

About the Author

Andy Haynssen profile.

Andy Haynssen

Principal Technologist

Andy is a creative person with a passion for software and technology! He has experience in building enterprise platforms with a focus on Infrastructure as Code, public cloud, and Continuous Integration/Continuous Delivery. He has been focusing in the DevOps space for the past 5+ years. In his free time, Andy enjoys creating music, woodworking, and spending time outdoors.

Leave a Reply

Your email address will not be published. Required fields are marked *

Related Blog Posts
Designing Kubernetes Controllers
There has been some excellent online discussion lately around Kubernetes controllers, highlighted by an excellent Speakerdeck presentation assembled by Tim Hockin. What I’d like to do in this post is explore some of the implications […]
React Server Components
The React Team recently announced new work they are doing on React Server Components, a new way of rendering React components. The goal is to create smaller bundle sizes, speed up render time, and prevent […]
Jolt custom java transform
Jolt is a JSON to JSON transformation library where the transform is defined in JSON. It’s really good at reorganizing the json data and massaging it into the output JSON you need. Sometimes, you just […]
Page Object Model for UI Testing