Jan 14, 2020

Keeping Secrets Out of Terraform State

There are many instances where you will want to create resources via Terraform with secrets that you just don’t want anyone to see. These could be IAM credentials, certificates, RDS DB credentials, etc. One problem with creating these resources via Terraform is that the statefile is stored in plain text. Yes, you could treat that statefile as a secret itself, but that may not be good enough depending on your security requirements. For example, you may want to be able to share that statefile with other users that are not authorized to view the secret’s values.

I will show a simple example of one way solve this problem. It may seem obvious, but the various cloud provider’s SDKs and APIs usually include a way to set these values. I’ll show here how to set up a new RDS instance and keep its master password from prying eyes.

First, I will create an RDS DB instance.

resource "aws_db_instance" "default" {
  name                 = "postgres_db"
  engine               = "postgres"
  instance_class       = "db.t2.micro"
  allocated_storage    = 20
  storage_type         = "gp2"
  username             = "foo"
  password             = "password123"
}

The password here is just a placeholder to initialize the instance. Next, I’ll use the boto3 Python library to update that password to a more secure secret.

#!/usr/env/bin python3
 
import argparse
import boto3
import secrets
import string
import time
 
parser = argparse.ArgumentParser(description='Override Terraform password')
parser.add_argument('--db_identifier', type=str, required=True, help="RDS DB instance identifier")
parser.add_argument('--secret_name', type=str, default='db_password', required=False, help="Name for SSM Parameter")
parser.add_argument('--region', type=str, default='us-east-1', required=False, help="AWS region name")
args = parser.parse_args()
 
 
def generate_password(string_length=10):
    """Generate a secure random string of letters, digits and special characters """
    password_characters = string.ascii_letters + string.digits + "!#$%^&*()"
    return ''.join(secrets.choice(password_characters) for i in range(string_length))
 
 
db_id = args.db_identifier
db_password = generate_password(16)
secret_name = args.secret_name
region = args.region
 
rds = boto3.client('rds', region_name=region)
db = rds.describe_db_instances(DBInstanceIdentifier=db_id)
while rds.describe_db_instances(DBInstanceIdentifier=db_id)["DBInstances"][0]["DBInstanceStatus"] != "available":
    time.sleep(3)
response = rds.modify_db_instance(DBInstanceIdentifier=db_id, MasterUserPassword=db_password, ApplyImmediately=True)
if response:
    print(f"Updated RDS password for {db_id}")

Using the script above, we will no longer have our “real” password stored in state. However, if we ever want to log into this DB we will need to store this somewhere else. This could be whatever tooling you use to manage secrets (Ex. Vault, CyberArk, etc.). If we want to keep this example native to AWS, we could store it in SSM Parameter Store by adding the following to our Python script:

ssm = boto3.client('ssm')
ssm.put_parameter(
    Name=secret_name,
    Description=f'DB Password for {db_id}',
    Type='SecureString',
    Value=db_password,
    Overwrite=True
)

All that’s left is getting Terraform to handle these actions through a null_resource. This could be a local_exec or remote_exec depending where you have dependencies available for the script.

resource "null_resource" "update_password" {
  provisioner "local-exec" {
    command = "python3 update_password.py --db_identifier=${aws_db_instance.default.identifier}"
  }
}

This Python script above is most useful if you need to store these secrets somewhere outside of AWS, rather than putting them in Parameter Store. A better AWS alternative might be to utilize AWS Secrets Manager to generate a password for RDS. Secrets Manager can also handle automatic rotation. The only real trade off with using Secrets Manager is that you will incur cost.

You can view my directory structure and full text of this example on GitHub here

I hope this sparks some creative ways to maintain your secrecy!

About the Author

Andy Haynssen profile.

Andy Haynssen

Principal Technologist

Andy is a creative person with a passion for software and technology! He has experience in building enterprise platforms with a focus on Infrastructure as Code, public cloud, and Continuous Integration/Continuous Delivery. He has been focusing in the DevOps space for the past 5+ years. In his free time, Andy enjoys creating music, woodworking, and spending time outdoors.

Leave a Reply

Your email address will not be published.

Related Blog Posts
Natively Compiled Java on Google App Engine
Google App Engine is a platform-as-a-service product that is marketed as a way to get your applications into the cloud without necessarily knowing all of the infrastructure bits and pieces to do so. Google App […]
Building Better Data Visualization Experiences: Part 2 of 2
If you don't have a Ph.D. in data science, the raw data might be difficult to comprehend. This is where data visualization comes in.
Unleashing Feature Flags onto Kafka Consumers
Feature flags are a tool to strategically enable or disable functionality at runtime. They are often used to drive different user experiences but can also be useful in real-time data systems. In this post, we’ll […]
A security model for developers
Software security is more important than ever, but developing secure applications is more confusing than ever. TLS, mTLS, RBAC, SAML, OAUTH, OWASP, GDPR, SASL, RSA, JWT, cookie, attack vector, DDoS, firewall, VPN, security groups, exploit, […]