Jan 14, 2020

Keeping Secrets Out of Terraform State

There are many instances where you will want to create resources via Terraform with secrets that you just don’t want anyone to see. These could be IAM credentials, certificates, RDS DB credentials, etc. One problem with creating these resources via Terraform is that the statefile is stored in plain text. Yes, you could treat that statefile as a secret itself, but that may not be good enough depending on your security requirements. For example, you may want to be able to share that statefile with other users that are not authorized to view the secret’s values.

I will show a simple example of one way solve this problem. It may seem obvious, but the various cloud provider’s SDKs and APIs usually include a way to set these values. I’ll show here how to set up a new RDS instance and keep its master password from prying eyes.

First, I will create an RDS DB instance.

resource "aws_db_instance" "default" {
  name                 = "postgres_db"
  engine               = "postgres"
  instance_class       = "db.t2.micro"
  allocated_storage    = 20
  storage_type         = "gp2"
  username             = "foo"
  password             = "password123"

The password here is just a placeholder to initialize the instance. Next, I’ll use the boto3 Python library to update that password to a more secure secret.

#!/usr/env/bin python3
import argparse
import boto3
import secrets
import string
import time
parser = argparse.ArgumentParser(description='Override Terraform password')
parser.add_argument('--db_identifier', type=str, required=True, help="RDS DB instance identifier")
parser.add_argument('--secret_name', type=str, default='db_password', required=False, help="Name for SSM Parameter")
parser.add_argument('--region', type=str, default='us-east-1', required=False, help="AWS region name")
args = parser.parse_args()
def generate_password(string_length=10):
    """Generate a secure random string of letters, digits and special characters """
    password_characters = string.ascii_letters + string.digits + "!#$%^&*()"
    return ''.join(secrets.choice(password_characters) for i in range(string_length))
db_id = args.db_identifier
db_password = generate_password(16)
secret_name = args.secret_name
region = args.region
rds = boto3.client('rds', region_name=region)
db = rds.describe_db_instances(DBInstanceIdentifier=db_id)
while rds.describe_db_instances(DBInstanceIdentifier=db_id)["DBInstances"][0]["DBInstanceStatus"] != "available":
response = rds.modify_db_instance(DBInstanceIdentifier=db_id, MasterUserPassword=db_password, ApplyImmediately=True)
if response:
    print(f"Updated RDS password for {db_id}")

Using the script above, we will no longer have our “real” password stored in state. However, if we ever want to log into this DB we will need to store this somewhere else. This could be whatever tooling you use to manage secrets (Ex. Vault, CyberArk, etc.). If we want to keep this example native to AWS, we could store it in SSM Parameter Store by adding the following to our Python script:

ssm = boto3.client('ssm')
    Description=f'DB Password for {db_id}',

All that’s left is getting Terraform to handle these actions through a null_resource. This could be a local_exec or remote_exec depending where you have dependencies available for the script.

resource "null_resource" "update_password" {
  provisioner "local-exec" {
    command = "python3 update_password.py --db_identifier=${aws_db_instance.default.identifier}"

This Python script above is most useful if you need to store these secrets somewhere outside of AWS, rather than putting them in Parameter Store. A better AWS alternative might be to utilize AWS Secrets Manager to generate a password for RDS. Secrets Manager can also handle automatic rotation. The only real trade off with using Secrets Manager is that you will incur cost.

You can view my directory structure and full text of this example on GitHub here

I hope this sparks some creative ways to maintain your secrecy!

About the Author

Andy Haynssen profile.

Andy Haynssen

Principal Technologist

Andy is a creative person with a passion for software and technology! He has experience in building enterprise platforms with a focus on Infrastructure as Code, public cloud, and Continuous Integration/Continuous Delivery. He has been focusing in the DevOps space for the past 5+ years. In his free time, Andy enjoys creating music, woodworking, and spending time outdoors.

Leave a Reply

Your email address will not be published. Required fields are marked *

Related Blog Posts
AWS Cloud HSM, Docker and NGINX
There is quite a bit of easily searchable content on the security benefits of leveraging a Hardware Security Module to manage cryptographic keys, so I will leave that to the scope of another article. The […]
Google Professional Machine Learning Engineer Exam 2021
Exam Description A Professional Machine Learning Engineer designs, builds, and productionizes ML models to solve business challenges using Google Cloud technologies and knowledge of proven ML models and techniques. The ML Engineer is proficient in all aspects […]
Designing Kubernetes Controllers
There has been some excellent online discussion lately around Kubernetes controllers, highlighted by an excellent Speakerdeck presentation assembled by Tim Hockin. What I’d like to do in this post is explore some of the implications […]
React Server Components
The React Team recently announced new work they are doing on React Server Components, a new way of rendering React components. The goal is to create smaller bundle sizes, speed up render time, and prevent […]