Sep 24, 2020

Setup Spring DataSource from values stored in AWS Secret Manager

I was tasked migrating an application to AWS recently. The company wanted the application to store the database credentials in AWS’s Secret Manager. The issue that I had is that if I setup a configuration with a custom DataSource, I lose some of the auto configuration. It’s possible to define my own DataSource but that added code I didn’t need. This also became apparent with trying to get Flyway to work too. I then found a workaround that worked for the use case I wanted.

To start, we need to setup a secret inside AWS. For this post I have a username and password item defined using the same secret name. We could store everything we need in here, including the URL, the driver, connection pool info, flyway connection details etc. Here is an example screenshot of a a secret defined for the database:

AWS ECR Task Configuration

Now that the secret is defined, we need a way to send this information in. For our purposes, I’ll be using a ECR container and define this in the secret section. The AWS task definition would look something like this:

[
  {
      "name": "my_app",
      "image": "someUserID.dkr.ecr.us-east-1.amazonaws.com/env/app:1.2.3",
      "memoryReservation": 512,
      "essential": true,
      "portMappings": [
          {
              "containerPort": 8080,
              "protocol": "tcp"
          }
      ],
      "secrets": [
          {
            "name": "DB_FLYWAY_CREDS_SECRET",
            "valueFrom": "arn:aws:secretsmanager:us-east-1:1:secret:/env/flyway_db_connection_info"
          },
          {
            "name": "DB_APP_CREDS_SECRET",
            "valueFrom": "arn:aws:secretsmanager:us-east-1:1:secret:/env/db_connection_info"
          }
      ],
      "environment": [
          {
            "name": "DB_HOST",
            "value": "somehost"
          },
          {
            "name": "DB_PORT",
            "value": "5432"
          }
      ]
  }
]

The values we’re sending in the secret section is the ARN of actual secret. What gets sent in is actually a JSON string with the key/value pairs of the secret. Just make sure your task is given the secretsmanager:GetSecretValue permission for the ARN of the key(s) you’re using. Make sure this is defined in the secret section so they aren’t visible in the AWS UI. Now onto the Java code.

Java Code

First of all I need to define my own properties. I want this app to work in AWS but also for local test and therefore I don’t want to rewrite the world. If the secret is set, great we’ll use that. If not, I just want it to work as-is. To get around this, I extend the properties that Spring come with and add a special awsDbSecret property which will be the JSON object that AWS sends in:

@Getter
@Setter
@ConfigurationProperties(prefix = "spring.datasource.my-app-custom")
public class CustomDataSourceProperties extends DataSourceProperties {
    /**
     * The AWS secret is JSON format containing the database connection (username/password)
     * information.
     */
    private String awsDbSecret;
}
@Getter
@Setter
@ConfigurationProperties(prefix = "spring.flyway.my-app-custom")
public class CustomFlywayProperties extends FlywayProperties {
    /**
     * The AWS secret is JSON format containing the flyway database connection (username/password)
     * information.
     */
    private String awsDbSecret;
}

Now that we have custom properties defines. Lets set up the application.yml file:

spring:
  datasource:
    driver-class-name: org.postgresql.Driver
    // Here I'm injecting DB_HOST and DB_PORT from AWS. This can be hardcoded or can be stored in the AWS secret as well.
    url: jdbc:postgresql://${DB_HOST}:${DB_PORT}/db-schema
    my-app-custom:
      aws-db-secret: ${DB_APP_CREDS_SECRET}
  flyway:
    my-app-custom:
      aws-db-secret: ${DB_FLYWAY_CREDS_SECRET}

Now we have properties we want. Let’s setup a custom @Configuration which does the magic for us.

import lombok.RequiredArgsConstructor;
import lombok.extern.slf4j.Slf4j;
import org.springframework.boot.autoconfigure.flyway.FlywayProperties;
import org.springframework.boot.autoconfigure.jdbc.DataSourceProperties;
import org.springframework.boot.configurationprocessor.json.JSONException;
import org.springframework.boot.configurationprocessor.json.JSONObject;
import org.springframework.boot.context.properties.EnableConfigurationProperties;
import org.springframework.context.annotation.Bean;
import org.springframework.context.annotation.Configuration;
import org.springframework.context.annotation.Primary;
import org.springframework.core.Ordered;
import org.springframework.core.annotation.Order;
import org.springframework.core.env.ConfigurableEnvironment;
import org.springframework.core.env.PropertiesPropertySource;
import org.springframework.util.StringUtils;

import java.util.Properties;


@RequiredArgsConstructor
@Slf4j
@Configuration
@EnableConfigurationProperties({CustomDataSourceProperties.class, CustomFlywayProperties.class})
@Order(Ordered.HIGHEST_PRECEDENCE)
public class DatabaseConfig {

    private final ConfigurableEnvironment environment;

    @Primary
    @Bean
    public DataSourceProperties customDataSourceProperties(CustomDataSourceProperties customDataSourceProperties, DataSourceProperties dataSourceProperties) {
        boolean usedAwsSecret = false;
        try {
            if (StringUtils.hasText(customDataSourceProperties.getAwsDbSecret())) {
                JSONObject dbCredentials = new JSONObject(customDataSourceProperties.getAwsDbSecret());
                dataSourceProperties.setUsername(dbCredentials.getString("username"));
                dataSourceProperties.setPassword(dbCredentials.getString("password"));

                // Needed since Hikari reads properties directly
                Properties properties = new Properties();
                properties.setProperty("spring.datasource.username", dataSourceProperties.getUsername());
                properties.setProperty("spring.datasource.password", dataSourceProperties.getPassword());
                environment.getPropertySources().addFirst(new PropertiesPropertySource("aws-custom-datasource-properties", properties));

                usedAwsSecret = true;
            }
        } catch (JSONException e) {
            log.error("Datasource AWS secret property was set but an error occurred parsing the value");
        }

        if (!usedAwsSecret) {
            log.info("No AWS credentials secret was configured. Falling back to properties set for username/password.");
        }

        return dataSourceProperties;
    }

    @Primary
    @Bean
    public FlywayProperties customFlywayProperties(CustomFlywayProperties customFlywayProperties, FlywayProperties flywayProperties) {
        boolean usedAwsSecret = false;
        try {
            if (StringUtils.hasText(customFlywayProperties.getAwsDbSecret())) {
                JSONObject dbCredentials = new JSONObject(customFlywayProperties.getAwsDbSecret());
                flywayProperties.setUser(dbCredentials.getString("username"));
                flywayProperties.setPassword(dbCredentials.getString("password"));

                // Needed since Hikari reads properties directly
                Properties properties = new Properties();
                properties.setProperty("spring.flyway.user", flywayProperties.getUser());
                properties.setProperty("spring.flyway.password", flywayProperties.getPassword());

                environment.getPropertySources().addFirst(new PropertiesPropertySource("aws-custom-flyway-properties", properties));

                usedAwsSecret = true;
            }
        } catch (JSONException e) {
            log.error("Flyway AWS secret property was set but an error occurred parsing the value");
        }

        if (!usedAwsSecret) {
            log.info("No AWS flyway credentials secret was configured. Falling back to properties set for username/password.");
        }

        return flywayProperties;
    }

}

Final Notes

And that’s it! This works because we are defining our own custom DataSourceProperties file and making it the primary bean. Now Spring will use the properties that have the username and password set. By extending the class, we get all of Spring’s current and future configuration items plus our extra decoder.

You may ask: Why is the @Primary annotation needed if our class extends DataSourceProperties? This is because Springs expect one and only one DataSourceProperties to be defined. It will fail to start if it detects two or more beans (ConfigurationProperties) defined. This gets around the framework by adding a little extra processing to Spring’s properties before it’s used anywhere else.

  • Auto configuration can’t occur until it injects the DataSource
  • DataSource can’t be created until it has the DataSourceProperties
  • DataSourceProperties can’t be injected until the bean marked as @Primary is defined first

The @ConfigurationProperties prefix can be whatever you like. It does not need to start with spring.datasource if you want to isolate the property namespace to be something more custom.

One final note. You can also use Spring’s spring-cloud-starter-aws-secrets-manager-config as well. In my use case I could not since I needed access to multiple secrets due to the company terraform modules I needed to use. spring-cloud-starter-aws-secrets-manager-config only allows access to one secret and thus wasn’t suitable for my case.

About the Author

Jeff Torson profile.

Jeff Torson

Principal Technologist

Jeff is a full stack developer with experience in the government/defense and finance industry. He has experience ranging from thick client Eclipse RCP programs to microservices using Spring Boot for data access and Elasticsearch. He enjoys learning about anything related to the IT field and has even managed Linux and Windows servers and setup deployment pipelines. He is a true believer in that if something is worth doing, then it’s worth doing right the first time and fully unit/integration tested.

Leave a Reply

Your email address will not be published. Required fields are marked *

Related Blog Posts
Getting Started with CSS Container Queries
For as long as I’ve been working full-time on the front-end, I’ve heard about the promise of container queries and their potential to solve the majority of our responsive web design needs. And, for as […]
Simple improvements to making decisions in teams
Software development teams need to make a lot of decisions. Functional requirements, non-functional requirements, user experience, API contracts, tech stack, architecture, database schemas, cloud providers, deployment strategy, test strategy, security, and the list goes on. […]
JavaScript Bundle Optimization – Polyfills
If you are lucky enough to only support a small subset of browsers (for example, you are targeting a controlled set of users), feel free to move along. However, if your website is open to […]
Creating Mocks For Unit Testing in Go
Unit testing is an important part of any project, and Go built its framework with a testing package; making unit testing part of the language. This testing framework is good for most scenarios, but you […]