Sep 24, 2020

Setup Spring DataSource from values stored in AWS Secret Manager

I was tasked migrating an application to AWS recently. The company wanted the application to store the database credentials in AWS’s Secret Manager. The issue that I had is that if I setup a configuration with a custom DataSource, I lose some of the auto configuration. It’s possible to define my own DataSource but that added code I didn’t need. This also became apparent with trying to get Flyway to work too. I then found a workaround that worked for the use case I wanted.

To start, we need to setup a secret inside AWS. For this post I have a username and password item defined using the same secret name. We could store everything we need in here, including the URL, the driver, connection pool info, flyway connection details etc. Here is an example screenshot of a a secret defined for the database:

AWS ECR Task Configuration

Now that the secret is defined, we need a way to send this information in. For our purposes, I’ll be using a ECR container and define this in the secret section. The AWS task definition would look something like this:

[
  {
      "name": "my_app",
      "image": "someUserID.dkr.ecr.us-east-1.amazonaws.com/env/app:1.2.3",
      "memoryReservation": 512,
      "essential": true,
      "portMappings": [
          {
              "containerPort": 8080,
              "protocol": "tcp"
          }
      ],
      "secrets": [
          {
            "name": "DB_FLYWAY_CREDS_SECRET",
            "valueFrom": "arn:aws:secretsmanager:us-east-1:1:secret:/env/flyway_db_connection_info"
          },
          {
            "name": "DB_APP_CREDS_SECRET",
            "valueFrom": "arn:aws:secretsmanager:us-east-1:1:secret:/env/db_connection_info"
          }
      ],
      "environment": [
          {
            "name": "DB_HOST",
            "value": "somehost"
          },
          {
            "name": "DB_PORT",
            "value": "5432"
          }
      ]
  }
]

The values we’re sending in the secret section is the ARN of actual secret. What gets sent in is actually a JSON string with the key/value pairs of the secret. Just make sure your task is given the secretsmanager:GetSecretValue permission for the ARN of the key(s) you’re using. Make sure this is defined in the secret section so they aren’t visible in the AWS UI. Now onto the Java code.

Java Code

First of all I need to define my own properties. I want this app to work in AWS but also for local test and therefore I don’t want to rewrite the world. If the secret is set, great we’ll use that. If not, I just want it to work as-is. To get around this, I extend the properties that Spring come with and add a special awsDbSecret property which will be the JSON object that AWS sends in:

@Getter
@Setter
@ConfigurationProperties(prefix = "spring.datasource.my-app-custom")
public class CustomDataSourceProperties extends DataSourceProperties {
    /**
     * The AWS secret is JSON format containing the database connection (username/password)
     * information.
     */
    private String awsDbSecret;
}
@Getter
@Setter
@ConfigurationProperties(prefix = "spring.flyway.my-app-custom")
public class CustomFlywayProperties extends FlywayProperties {
    /**
     * The AWS secret is JSON format containing the flyway database connection (username/password)
     * information.
     */
    private String awsDbSecret;
}

Now that we have custom properties defines. Lets set up the application.yml file:

spring:
  datasource:
    driver-class-name: org.postgresql.Driver
    // Here I'm injecting DB_HOST and DB_PORT from AWS. This can be hardcoded or can be stored in the AWS secret as well.
    url: jdbc:postgresql://${DB_HOST}:${DB_PORT}/db-schema
    my-app-custom:
      aws-db-secret: ${DB_APP_CREDS_SECRET}
  flyway:
    my-app-custom:
      aws-db-secret: ${DB_FLYWAY_CREDS_SECRET}

Now we have properties we want. Let’s setup a custom @Configuration which does the magic for us.

import lombok.RequiredArgsConstructor;
import lombok.extern.slf4j.Slf4j;
import org.springframework.boot.autoconfigure.flyway.FlywayProperties;
import org.springframework.boot.autoconfigure.jdbc.DataSourceProperties;
import org.springframework.boot.configurationprocessor.json.JSONException;
import org.springframework.boot.configurationprocessor.json.JSONObject;
import org.springframework.boot.context.properties.EnableConfigurationProperties;
import org.springframework.context.annotation.Bean;
import org.springframework.context.annotation.Configuration;
import org.springframework.context.annotation.Primary;
import org.springframework.core.Ordered;
import org.springframework.core.annotation.Order;
import org.springframework.core.env.ConfigurableEnvironment;
import org.springframework.core.env.PropertiesPropertySource;
import org.springframework.util.StringUtils;

import java.util.Properties;


@RequiredArgsConstructor
@Slf4j
@Configuration
@EnableConfigurationProperties({CustomDataSourceProperties.class, CustomFlywayProperties.class})
@Order(Ordered.HIGHEST_PRECEDENCE)
public class DatabaseConfig {

    private final ConfigurableEnvironment environment;

    @Primary
    @Bean
    public DataSourceProperties customDataSourceProperties(CustomDataSourceProperties customDataSourceProperties, DataSourceProperties dataSourceProperties) {
        boolean usedAwsSecret = false;
        try {
            if (StringUtils.hasText(customDataSourceProperties.getAwsDbSecret())) {
                JSONObject dbCredentials = new JSONObject(customDataSourceProperties.getAwsDbSecret());
                dataSourceProperties.setUsername(dbCredentials.getString("username"));
                dataSourceProperties.setPassword(dbCredentials.getString("password"));

                // Needed since Hikari reads properties directly
                Properties properties = new Properties();
                properties.setProperty("spring.datasource.username", dataSourceProperties.getUsername());
                properties.setProperty("spring.datasource.password", dataSourceProperties.getPassword());
                environment.getPropertySources().addFirst(new PropertiesPropertySource("aws-custom-datasource-properties", properties));

                usedAwsSecret = true;
            }
        } catch (JSONException e) {
            log.error("Datasource AWS secret property was set but an error occurred parsing the value");
        }

        if (!usedAwsSecret) {
            log.info("No AWS credentials secret was configured. Falling back to properties set for username/password.");
        }

        return dataSourceProperties;
    }

    @Primary
    @Bean
    public FlywayProperties customFlywayProperties(CustomFlywayProperties customFlywayProperties, FlywayProperties flywayProperties) {
        boolean usedAwsSecret = false;
        try {
            if (StringUtils.hasText(customFlywayProperties.getAwsDbSecret())) {
                JSONObject dbCredentials = new JSONObject(customFlywayProperties.getAwsDbSecret());
                flywayProperties.setUser(dbCredentials.getString("username"));
                flywayProperties.setPassword(dbCredentials.getString("password"));

                // Needed since Hikari reads properties directly
                Properties properties = new Properties();
                properties.setProperty("spring.flyway.user", flywayProperties.getUser());
                properties.setProperty("spring.flyway.password", flywayProperties.getPassword());

                environment.getPropertySources().addFirst(new PropertiesPropertySource("aws-custom-flyway-properties", properties));

                usedAwsSecret = true;
            }
        } catch (JSONException e) {
            log.error("Flyway AWS secret property was set but an error occurred parsing the value");
        }

        if (!usedAwsSecret) {
            log.info("No AWS flyway credentials secret was configured. Falling back to properties set for username/password.");
        }

        return flywayProperties;
    }

}

Final Notes

And that’s it! This works because we are defining our own custom DataSourceProperties file and making it the primary bean. Now Spring will use the properties that have the username and password set. By extending the class, we get all of Spring’s current and future configuration items plus our extra decoder.

You may ask: Why is the @Primary annotation needed if our class extends DataSourceProperties? This is because Springs expect one and only one DataSourceProperties to be defined. It will fail to start if it detects two or more beans (ConfigurationProperties) defined. This gets around the framework by adding a little extra processing to Spring’s properties before it’s used anywhere else.

  • Auto configuration can’t occur until it injects the DataSource
  • DataSource can’t be created until it has the DataSourceProperties
  • DataSourceProperties can’t be injected until the bean marked as @Primary is defined first

The @ConfigurationProperties prefix can be whatever you like. It does not need to start with spring.datasource if you want to isolate the property namespace to be something more custom.

One final note. You can also use Spring’s spring-cloud-starter-aws-secrets-manager-config as well. In my use case I could not since I needed access to multiple secrets due to the company terraform modules I needed to use. spring-cloud-starter-aws-secrets-manager-config only allows access to one secret and thus wasn’t suitable for my case.

About the Author

Object Partners profile.
Leave a Reply

Your email address will not be published.

Related Blog Posts
Natively Compiled Java on Google App Engine
Google App Engine is a platform-as-a-service product that is marketed as a way to get your applications into the cloud without necessarily knowing all of the infrastructure bits and pieces to do so. Google App […]
Building Better Data Visualization Experiences: Part 2 of 2
If you don't have a Ph.D. in data science, the raw data might be difficult to comprehend. This is where data visualization comes in.
Unleashing Feature Flags onto Kafka Consumers
Feature flags are a tool to strategically enable or disable functionality at runtime. They are often used to drive different user experiences but can also be useful in real-time data systems. In this post, we’ll […]
A security model for developers
Software security is more important than ever, but developing secure applications is more confusing than ever. TLS, mTLS, RBAC, SAML, OAUTH, OWASP, GDPR, SASL, RSA, JWT, cookie, attack vector, DDoS, firewall, VPN, security groups, exploit, […]