Sep 24, 2020

Setup Spring DataSource from values stored in AWS Secret Manager

I was tasked migrating an application to AWS recently. The company wanted the application to store the database credentials in AWS’s Secret Manager. The issue that I had is that if I setup a configuration with a custom DataSource, I lose some of the auto configuration. It’s possible to define my own DataSource but that added code I didn’t need. This also became apparent with trying to get Flyway to work too. I then found a workaround that worked for the use case I wanted.

To start, we need to setup a secret inside AWS. For this post I have a username and password item defined using the same secret name. We could store everything we need in here, including the URL, the driver, connection pool info, flyway connection details etc. Here is an example screenshot of a a secret defined for the database:

AWS ECR Task Configuration

Now that the secret is defined, we need a way to send this information in. For our purposes, I’ll be using a ECR container and define this in the secret section. The AWS task definition would look something like this:

[
  {
      "name": "my_app",
      "image": "someUserID.dkr.ecr.us-east-1.amazonaws.com/env/app:1.2.3",
      "memoryReservation": 512,
      "essential": true,
      "portMappings": [
          {
              "containerPort": 8080,
              "protocol": "tcp"
          }
      ],
      "secrets": [
          {
            "name": "DB_FLYWAY_CREDS_SECRET",
            "valueFrom": "arn:aws:secretsmanager:us-east-1:1:secret:/env/flyway_db_connection_info"
          },
          {
            "name": "DB_APP_CREDS_SECRET",
            "valueFrom": "arn:aws:secretsmanager:us-east-1:1:secret:/env/db_connection_info"
          }
      ],
      "environment": [
          {
            "name": "DB_HOST",
            "value": "somehost"
          },
          {
            "name": "DB_PORT",
            "value": "5432"
          }
      ]
  }
]

The values we’re sending in the secret section is the ARN of actual secret. What gets sent in is actually a JSON string with the key/value pairs of the secret. Just make sure your task is given the secretsmanager:GetSecretValue permission for the ARN of the key(s) you’re using. Make sure this is defined in the secret section so they aren’t visible in the AWS UI. Now onto the Java code.

Java Code

First of all I need to define my own properties. I want this app to work in AWS but also for local test and therefore I don’t want to rewrite the world. If the secret is set, great we’ll use that. If not, I just want it to work as-is. To get around this, I extend the properties that Spring come with and add a special awsDbSecret property which will be the JSON object that AWS sends in:

@Getter
@Setter
@ConfigurationProperties(prefix = "spring.datasource.my-app-custom")
public class CustomDataSourceProperties extends DataSourceProperties {
    /**
     * The AWS secret is JSON format containing the database connection (username/password)
     * information.
     */
    private String awsDbSecret;
}
@Getter
@Setter
@ConfigurationProperties(prefix = "spring.flyway.my-app-custom")
public class CustomFlywayProperties extends FlywayProperties {
    /**
     * The AWS secret is JSON format containing the flyway database connection (username/password)
     * information.
     */
    private String awsDbSecret;
}

Now that we have custom properties defines. Lets set up the application.yml file:

spring:
  datasource:
    driver-class-name: org.postgresql.Driver
    // Here I'm injecting DB_HOST and DB_PORT from AWS. This can be hardcoded or can be stored in the AWS secret as well.
    url: jdbc:postgresql://${DB_HOST}:${DB_PORT}/db-schema
    my-app-custom:
      aws-db-secret: ${DB_APP_CREDS_SECRET}
  flyway:
    my-app-custom:
      aws-db-secret: ${DB_FLYWAY_CREDS_SECRET}

Now we have properties we want. Let’s setup a custom @Configuration which does the magic for us.

import lombok.RequiredArgsConstructor;
import lombok.extern.slf4j.Slf4j;
import org.springframework.boot.autoconfigure.flyway.FlywayProperties;
import org.springframework.boot.autoconfigure.jdbc.DataSourceProperties;
import org.springframework.boot.configurationprocessor.json.JSONException;
import org.springframework.boot.configurationprocessor.json.JSONObject;
import org.springframework.boot.context.properties.EnableConfigurationProperties;
import org.springframework.context.annotation.Bean;
import org.springframework.context.annotation.Configuration;
import org.springframework.context.annotation.Primary;
import org.springframework.core.Ordered;
import org.springframework.core.annotation.Order;
import org.springframework.core.env.ConfigurableEnvironment;
import org.springframework.core.env.PropertiesPropertySource;
import org.springframework.util.StringUtils;

import java.util.Properties;


@RequiredArgsConstructor
@Slf4j
@Configuration
@EnableConfigurationProperties({CustomDataSourceProperties.class, CustomFlywayProperties.class})
@Order(Ordered.HIGHEST_PRECEDENCE)
public class DatabaseConfig {

    private final ConfigurableEnvironment environment;

    @Primary
    @Bean
    public DataSourceProperties customDataSourceProperties(CustomDataSourceProperties customDataSourceProperties, DataSourceProperties dataSourceProperties) {
        boolean usedAwsSecret = false;
        try {
            if (StringUtils.hasText(customDataSourceProperties.getAwsDbSecret())) {
                JSONObject dbCredentials = new JSONObject(customDataSourceProperties.getAwsDbSecret());
                dataSourceProperties.setUsername(dbCredentials.getString("username"));
                dataSourceProperties.setPassword(dbCredentials.getString("password"));

                // Needed since Hikari reads properties directly
                Properties properties = new Properties();
                properties.setProperty("spring.datasource.username", dataSourceProperties.getUsername());
                properties.setProperty("spring.datasource.password", dataSourceProperties.getPassword());
                environment.getPropertySources().addFirst(new PropertiesPropertySource("aws-custom-datasource-properties", properties));

                usedAwsSecret = true;
            }
        } catch (JSONException e) {
            log.error("Datasource AWS secret property was set but an error occurred parsing the value");
        }

        if (!usedAwsSecret) {
            log.info("No AWS credentials secret was configured. Falling back to properties set for username/password.");
        }

        return dataSourceProperties;
    }

    @Primary
    @Bean
    public FlywayProperties customFlywayProperties(CustomFlywayProperties customFlywayProperties, FlywayProperties flywayProperties) {
        boolean usedAwsSecret = false;
        try {
            if (StringUtils.hasText(customFlywayProperties.getAwsDbSecret())) {
                JSONObject dbCredentials = new JSONObject(customFlywayProperties.getAwsDbSecret());
                flywayProperties.setUser(dbCredentials.getString("username"));
                flywayProperties.setPassword(dbCredentials.getString("password"));

                // Needed since Hikari reads properties directly
                Properties properties = new Properties();
                properties.setProperty("spring.flyway.user", flywayProperties.getUser());
                properties.setProperty("spring.flyway.password", flywayProperties.getPassword());

                environment.getPropertySources().addFirst(new PropertiesPropertySource("aws-custom-flyway-properties", properties));

                usedAwsSecret = true;
            }
        } catch (JSONException e) {
            log.error("Flyway AWS secret property was set but an error occurred parsing the value");
        }

        if (!usedAwsSecret) {
            log.info("No AWS flyway credentials secret was configured. Falling back to properties set for username/password.");
        }

        return flywayProperties;
    }

}

Final Notes

And that’s it! This works because we are defining our own custom DataSourceProperties file and making it the primary bean. Now Spring will use the properties that have the username and password set. By extending the class, we get all of Spring’s current and future configuration items plus our extra decoder.

You may ask: Why is the @Primary annotation needed if our class extends DataSourceProperties? This is because Springs expect one and only one DataSourceProperties to be defined. It will fail to start if it detects two or more beans (ConfigurationProperties) defined. This gets around the framework by adding a little extra processing to Spring’s properties before it’s used anywhere else.

  • Auto configuration can’t occur until it injects the DataSource
  • DataSource can’t be created until it has the DataSourceProperties
  • DataSourceProperties can’t be injected until the bean marked as @Primary is defined first

The @ConfigurationProperties prefix can be whatever you like. It does not need to start with spring.datasource if you want to isolate the property namespace to be something more custom.

One final note. You can also use Spring’s spring-cloud-starter-aws-secrets-manager-config as well. In my use case I could not since I needed access to multiple secrets due to the company terraform modules I needed to use. spring-cloud-starter-aws-secrets-manager-config only allows access to one secret and thus wasn’t suitable for my case.

About the Author

Jeff Torson profile.

Jeff Torson

Sr. Consultant

Jeff is a full stack developer with experience in the government/defense and finance industry. He has experience ranging from thick client Eclipse RCP programs to microservices using Spring Boot for data access and Elasticsearch. He enjoys learning about anything related to the IT field and has even managed Linux and Windows servers and setup deployment pipelines. He is a true believer in that if something is worth doing, then it’s worth doing right the first time and fully unit/integration tested.

Leave a Reply

Your email address will not be published. Required fields are marked *

Related Blog Posts
TICK Stack Monitoring for the Non-Technical
TICK – Telegraf, Influx, Chronograf, and Kapacitor – is a method of monitoring your systems and applications. In this article, I discuss in non-technical terms what the difference is between TICK and Prometheus Grafana A […]
Design Systems, Part 1 • Introduction
Business leaders need a practical guide to plan and execute Design System Initiatives. The aim of this series is to be that guide. This installment introduces terms and definitions as a primer on Design Systems.
ML for Translating Dysarthria Speech (Pre-Part 1)
What is Dysarthria? Per the Mayo Clinic, Dysarthria occurs when the muscles you use for speech are weak or you have difficulty controlling them. Dysarthria often causes slurred or slow speech that can be difficult […]
Develop Quality Code
As software continues to dominate every facet of our lives, developers are faced with an ever-increasing pressure to produce bug free code. The responsibility of clean quality software falls upon everyone that is involved in […]