Framework for a Multi-stage Spring Property Loader Extension Allowing Dynamic Updates of Properties via JMX

Overview

It is commonplace in enterprise applications to allow application properties to be loaded from configuration files. When leveraging Spring, this is typically achieved using a PropertyPlaceholderConfigurer instance within the application. As is stated in the JavaDocs for this class,  it is “A property resource configurer that resolves placeholders in bean property values of context definitions. It pulls values from a properties file into bean definitions.” (see http://static.springsource.org/spring/docs/3.0.x/api/org/springframework/beans/factory/config/PropertyPlaceholderConfigurer.html)

Properties are loaded at application startup, prior to loading other spring beans for the application. Other beans in the application can then leverage the properties as part of their initialization, allowing externalization of various properties. This can be accomplished within Spring context files using placeholders containing the desired property keys, or via annotations in the classes themselves.

A common and vital pattern leveraged in many applications is to allow for environment-specific properties file. This means that different properties files may get loaded depending on the environment in which the application is running. There are a handful of strategies for supporting this pattern, and a full discussion is outside of the scope of this discussion. The one strategy that will be discussed is as follows: specify a location external to the application itself (such as a configuration folder living on the system where the application will be deployed) that will contain a consistently named property file (e.g. myApp.properties). When configuring the PropertyPlaceholderConfigurer spring bean, specify that location as the location of the properties file to load. This might look like:


<bean id="placeholderConfigurer" class="org.springframework.beans.factory.config.PropertyPlaceholderConfigurer">
    <property name="location" value="file:/my/app/config/myApp.properties" />
</bean>

Even more useful is to leverage multiple properties files, allowing for property overrides. This might look like:


<bean id="placeholderConfigurer" class="org.springframework.beans.factory.config.PropertyPlaceholderConfigurer">
    <property name="locations">
        <list>
            <value>classpath:defaults.properties</value>
            <value>file:/my/app/config/myApp.properties </value>
         </list>
    </property>
</bean>

When using a list of property files for a PropertyPlaceholderConfigurer, entries later in the list take precedence over entries earlier in the list when there are property key collisions. In the above example, we specify an internal property file within the distribution called default.properties, and a property file external to the application. In the event that there is overlap of property keys between the files, the values in the external properties file would be used (because it is last in the list). Furthermore, the full set of properties loaded would be the union of the properties contained in the two files (e.g., there is no stipulation that the properties contained in the files correlate at all, unless you desire it). There are many advantages to this simple approach to property loading, including:

  • The default (internal) property file can contain default values for all properties, if desired, ensuring that there will not be “missing” properties in your application
  • The external property file can be updated manually on the server without requiring redeployment. However, an application restart would be required to pick up the modifications to the file.
  • Properties that should never be modified without an application redeployment can be exclusively contained within the internal properties file. This might be important if there are properties that you need to be configurable for unit testing purposes, but otherwise need to be consistent in all deployments regardless of environment.

At this point a developer with some knowledge of Spring is probably saying “so what”. This is nothing that isn’t already outlined in Spring’s documentation and dozens of examples out there on the web. So, here is where we can introduce some extensions that make the configuration framework more flexible.

Enabling Multi-stage Property Loading

At a recent client, there was a requirement for an application that we were writing that many of it’s properties be configurable from a central configuration database. Essentially, the database represented servers, applications on those servers, and key-value property pairs for each application/server combination. The idea is that it is a one-stop shop for managing the configurations for all applications (and in fact all instances of each application) in the enterprise. In contrast to that requirement (or in conjunction with it), from a development and testing perspective we wanted:

  • An internal properties file to represent defaults and properties that should not change from environment to environment (e.g. should not be configurable without an application redeployment, as they may substantially affect some piece of logic within the application)
  • An external, environment-specific properties file that could be changed at will (at least in local and dev environment), and could contain overrides for a number of the properties in the internal properties file.

Given these three sources for properties, we needed to put together an extension to the property placeholder framework that could:

  • Load properties from multiple sources
    • Including a database
  • Allow staged property loading with flexible override ordering. In other words, we wanted to be able to specify whether the internal, external, or database properties should take precedence when configuration keys overlap.

As we saw in the previous section, dealing with the internal and external properties files is easy. However, dealing with the database properties files takes a little more work..

For beginners, we need a basic extension to PropertyPlaceholderConfigurer to work with. We will start with the following class, StagedPropertyPlaceholderConfigurer, which extends PropertyPlaceholderConfigurer.


public class StagedPropertyPlaceholderConfigurer extends PropertyPlaceholderConfigurer {

    private boolean loadFromDB = true;
    private boolean loadFromExternalPropertiesFile = true;
    private boolean loadFromInternalPropertiesFile = true;

    private Resource internalProperties;
    private Resource externalProperties;

    public StagedPropertyPlaceholderConfigurer(Resource internalProperties, Resource externalProperties) {
        super();
        this.internalProperties = internalProperties;
        this.externalProperties = externalProperties;

        // if this constructor is used, a data source will not be used for property loading
        loadFromDB = false;

        if(internalProperties == null) {
            loadFromInternalPropertiesFile = false;
        }

        if(externalProperties == null) {
            loadFromExternalPropertiesFile = false;
        }
    }
}

As you can see, this class contains properties for the internal and external property files, and a basic constructor taking these arguments. The configurer is set up to be used without the database at all, for the moment. Some flags to indicate where to load properties from for reference during the actual loading.

Next, we need to override the loadProperties method from the PropertyPlaceholderConfigurer class. The Spring framework will call this method to load the properties. For the moment, lets make it simple, since we are not considering the database quite yet:


@Override
protected void loadProperties(final Properties props) throws IOException {
    List<Resource> resources = new ArrayList<Resource>();

    if(loadFromInternalPropertiesFile) {
        resources.add(internalProperties);
    }

    if(loadFromExternalPropertiesFile) {
        resources.add(externalProperties);
    }

    Resource[] locations = new Resource[resources.size()];
    resources.toArray(locations);
    setLocations(locations);

    super.loadProperties(props);
}

At this point, we have essentially re-invented the wheel, so lets go a bit further. Now we are going to add the ability to configure a data source to load properties from, and specify how to load them. In order to do this in as generic a fashion as we can, we will define a DatasourceProperyLoaderStrategy interface as follows:



/**
 * Interface to be implemented for defining a data source-based property loading strategy to use in conjunction
 * with the StagedPropertyPlaceholderConfigurer.
 *
 */
 public interface DatasourcePropertyLoaderStrategy {

    /**
     * provide any configuration needed to connect to the data source. This should include creation of the data source
     * and any other initialization necessary.
     */
     public void configure(Properties properties);

    /**
     * Load properties from the configured data source. If overrideExisting is true,
     * replace any existing properties in the {@link Properties} instance with the ones loaded from
     * the data source. Otherwise, the existing properties take precedence in the case of property key collisions
     *
     * @param properties {@link Properties} instance containing any propreties that have already been loaded
     * @param overrideExisting if true, the properties loaded from this data source take precedence in the case of key collisions.
     */
    public void loadProperties(Properties properties, boolean overrideExisting);

    /**
     * Any cleanup required after properties are loaded, for instance disposing of the data source
     */
    public void tearDown(Properties properties);

}

An implementation of this interface is necessary for defining how properties would be loaded from a data source. This implementation would need to define:

  • How to connect to the data source (e.g., would need to be aware of data source attributes such as the database urls, drivers, credentials, and so on).
  • How to load properties from the database (e.g. where are properties loaded, in what format, how to process them)
  • How to merge properties pulled from the data source with the other properties (e.g., is manipulation of keys required, are there case-sensitivity concerns, should the properties loaded from the database override the previously loaded properties)
  • How to clean up the data source and dispose of it when property loading is completed.

Note that all methods on this interface take the Properties instance. Obviously this is needed to merge properties from the database with the previously loaded properties. However, the fact that they are passed to the configure() and teardown() methods allows previously loaded properties to be leveraged while determining how to load the additional properties. For instance, we can put the data source attributes in the external, environment-specific property file. This file would be loaded first, and therefore the data source attributes contained within that configuration file can be used to establish the data source to pull additional properties from the database.

Another useful aspect of this interface is that it would not be limited to pulling properties from a database. You could easily create an implementation that pulled configuration from another source, such as a web service.

If you were to implement a strategy to load from a database, the org.springframework.jdbc.datasource.DriverManagerDataSource class is a safe bet for creating a data source.

Okay, now back to the StagedPropertyPlaceholderConfigurer itself. With an implementation of DatasourcePropertyLoaderStrategy assumed to be defined, lets make the following modifications (in bold):


public class StagedPropertyPlaceholderConfigurer extends PropertyPlaceholderConfigurer {

    private boolean loadFromDB = true;
    private boolean loadFromExternalPropertiesFile = true;
    private boolean loadFromInternalPropertiesFile = true;
    private boolean databaseTakesPrecedence = true;

    private Resource internalProperties;
    private Resource externalProperties;

    private DatasourcePropertyLoaderStrategy databasePropertyLoaderStrategy;

    public StagedPropertyPlaceholderConfigurer(Resource internalProperties, Resource externalProperties, DatasourcePropertyLoaderStrategy databasePropertyLoaderStrategy, boolean databasePrecedence) {
        super();
        this.databasePropertyLoaderStrategy = databasePropertyLoaderStrategy;
        this.internalProperties = internalProperties;
        this.externalProperties = externalProperties;
        this.databaseTakesPrecedence = databasePrecedence;

        if(databasePropertyLoaderStrategy == null) {
            loadFromDB = false;
            databaseTakesPrecedence = false;
        }

        if(internalProperties == null) {
            loadFromInternalPropertiesFile = false;
        }

        if(externalProperties == null) {
            loadFromExternalPropertiesFile = false;
        }
    }

    public StagedPropertyPlaceholderConfigurer(Resource internalProperties, Resource externalProperties) {
        super();
        this.internalProperties = internalProperties;
        this.externalProperties = externalProperties;
        loadFromDB = false;

        if(internalProperties == null) {
            loadFromInternalPropertiesFile = false;
        }

        if(externalProperties == null) {
            loadFromExternalPropertiesFile = false;
        }
    }

    @Override
    protected void loadProperties(final Properties props) throws IOException {
        List<Resource> resources = new ArrayList<Resource>();

        if(loadFromInternalPropertiesFile) {
            resources.add(internalProperties);
        }

        if(loadFromExternalPropertiesFile) {
            resources.add(externalProperties);
        }

        Resource[] locations = new Resource[resources.size()];
        resources.toArray(locations);
        setLocations(locations);
        loadProperties(props);

        if(loadFromDB) {
            try {
                databasePropertyLoaderStrategy.configure(props);
                databasePropertyLoaderStrategy.loadProperties(props, databaseTakesPrecedence);
                databasePropertyLoaderStrategy.tearDown(props); 
            } catch (Exception e) {
                // just log the exception? This happen if the datasource is unavailable or the data returned
                // cannot be processed correctly. In this case, the DatasourcePropertyLoaderStrategy should be responsible for
                // any necessary exception handling.
            } 
        } 
    }
}

So far so good. We have a property loader that can take properties from multiple sources and merge them. It can pull property from an external data source, whether that is a database, a web service, or anything else. It can be configured to allow different property sources to take precedence over others. And the only real work involved is in implementing the DatasourcePropertyLoaderStrategy. But, as long as we have gone this far, lets go a bit further.

Exposing a Configuration Bean for Use Throughout the Code Base

What has been discussed up to this point allows for configuration modifications that would only be picked up with an application restart. It would be nice to be able to dynamically update properties via JMX without an application restart. This can be very useful for tuning applications (if you have properties that are considered “tunable”), or allowing dynamically modifiable behavior. For example, if your application posts content over http to some URL, you might want to be able to modify that URL without restarting the application or having to monkey around with the network.

In order to facilitate this, we will further modify the property placeholder configuration framework, enabling the creation of a “configuration bean” that can be leveraged throughout the application. This configuration bean would get populated with the final output of the StagedPropertyPlaceholderConfigurer. With some modifications to how you access your properties in your code, the bean can then be leveraged to get access to properties. And the bean can be exposed as a managed resource to allow dynamic modification of the properties.

First, lets will define a ConfigurationBean interface as follows:


public interface ConfigurationBean {

    public Map<String, String> getProperties();
    public void setProperties(final Properties props);
    public String getValue(String key);
    public Integer getIntValue(String key);
    public Long getLongValue(String key);
    public Boolean getBooleanValue(String key);
    public String setProperty(String key, String value);
    public void loadIntoCollection(String key, final Collection target, Class classType) throws Exception;
}

And a base implementation of this interface:


public class ConfigurationBeanImpl implements ConfigurationBean {

    /** The props configuration key-value pairs */
    private Map<String, String> props = new HashMap<String, String>();

    /**
     * Gets the properties.
     *
     * @return the properties
     */
    public Map<String, String> getProperties() {
        if (props == null) {
            return null;
        }

        return Collections.unmodifiableMap(props);
    }

    /**
     * Sets the properties for the configuration bean based on Properties loaded by the Property Placeholder Configurer
     *
     * @param props
     */
    public void setProperties(final Properties props) {
        for (Iterator iterator = ((Map) props).keySet().iterator(); iterator.hasNext();) {
            String key = (String) iterator.next();
            this.props.put(key, props.getProperty(key));
        }
    }

    /**
     * Returns the value for the specified key, or null if no such key exists.
     *
     * @param key
     * @return the value
     */
    public String getValue(String key) {
        if (props == null) {
            return null;
        }

        return props.get(key);
    }

    /**
     * Gets the value for the specified key as an Integer
     *
     * @param key
     * @return the value
     */
    public Integer getIntValue(String key) {
        if (props == null) {
            return null;
        }

        String val = props.get(key);
        if (val == null) {
            return null;
        }

        return Integer.valueOf(val);
    }

    /**
     * Gets the value for the specified key as a Long
     *
     * @param key
     * @return the value
     */
    public Long getLongValue(String key) {
        if (props == null) {
            return null;
        }

        String val = props.get(key);
        if (val == null) {
            return null;
        }

        return Long.valueOf(val);
    }

    /**
     * Gets the value for the specified key as a Boolean
     *
     * @param key
     * @return the value
     */
    public Boolean getBooleanValue(String key) {
        if (props == null) {
            return null;
        }

        String val = props.get(key);
        if (val == null) {
            return null;
        }

        return Boolean.valueOf(val);
    }

    /**
     * Sets the specified property with the specified value
     *
     * @param key
     * @param value
     * @return the previous value of the property
     */
    public String setProperty(String key, String value) {
        if (props == null) {
            props = new HashMap<String, String>();
        }

        return props.put(key, value);
    }

    /**
     * Used to populate the collection target, with values of type classType. classType must have a single String argument constructor to create new instances.
     *
     * Example usage
     *
     * List<Long> toFill = new ArrayList<Long>();
     *
     * loadIntoCollection("propertyKey",toFill, Long.class);
     *
     * @param key
     * @param target The collection to load config data into cannot be null
     * @param classType The class type to cast the config's String value to. Should be the class type assigned to the Collection target
     *
     * @throws Exception
     **/
    public void loadIntoCollection(String key, final Collection target, Class classType) throws Exception {
        if (props == null) {
            return;
        }

        if (target == null) {
            throw new IllegalArgumentException("Argument target cannot be null");
        }

        if (classType == null) {
            throw new IllegalArgumentException("Argument classType cannot be null");
        }

        String val = props.get(key);
        if (val == null || val.length() == 0) {
            return;
        }

        String[] vals = val.split(",");
        Constructor con;

        try {
            con = classType.getConstructor(String.class);
        } catch (NoSuchMethodException e) {
            throw new IllegalArgumentException(classType + " does not have a single String argument constructor ");
        }

        for (int i = 0; i < vals.length; i++) {
            try {
                target.add(con.newInstance(vals[i]));
            } catch (Exception e) {
                throw new Exception("Config Value: " + vals[i] + " cannot be converted to " + classType.getName());
            }
        }
    }
}

This interface can of course be extended to handle additional data types, or to provide more granular methods for getting and setting specific properties. In fact, it would be a good idea for any application using this framework to create a ConfigurationBean implementation that extends ConfigurationBeanImpl, just for the flexibility it would provide.

Now we need to define a singleton instance of the ConfigurationBean that will be used within the StagedPropertyPlaceholderConfigurer (and elsewhere in the application). For instance:


<bean id="configBean" class="com.objectpartners.sandbox.config.ConfigurationBeanImpl" scope="singleton"/>

Next, the StagedPropertyPlaceholderConfigurer needs to be updated to populate the ConfigurationBean instance. There are a few things that need to be added to this class to support this:

  • We need a handle on the bean factory, as we will need to look up the ConfigurationBean instance. This can be accomplished by:
    • Defining a private data member for the BeanFactory (There is a BeanFactory instance on the superclass PropertyPlaceholderConfigurer, but it is hidden):
    • 
      private BeanFactory beanFactory;
      
    • Overriding the setBeanFactory method as follows:
    • 
      @Override
      public void setBeanFactory(BeanFactory beanFactory) throws BeansException {
          this.beanFactory = beanFactory;
          super.setBeanFactory(beanFactory);
      }
      
      
  • We need to know the id of the ConfigurationBean singleton in order to be able to find it in the bean factory.
    • Add the string name as a data member:
    • 
      private String configurationBeanName;
      
      
    • Update all constructors to take the configuration bean name as a parameter. E.g.
    • 
      
      public StagedPropertyPlaceholderConfigurer(Resource internalProperties, Resource externalProperties,
                                                 DatasourcePropertyLoaderStrategy databasePropertyLoaderStrategy,
                                                 boolean databasePreceduence, String configurationBeanName) {
          super();
          this.configurationBeanName = configurationBeanName;
          this.databasePropertyLoaderStrategy = databasePropertyLoaderStrategy;
          this.internalProperties = internalProperties;
          this.externalProperties = externalProperties;
          this.databaseTakesPrecedence = databasePreceduence;
      
          if(databasePropertyLoaderStrategy == null) {
              loadFromDB = false;
              databaseTakesPrecedence = false;
          }
      
          if(internalProperties == null) {
              loadFromInternalPropertiesFile = false;
          }
      
          if(externalProperties == null) {
              loadFromExternalPropertiesFile = false;
          }
      }
      
      
  • We need to implement a method to populate the ConfigurationBean instance with the output of the property loading process:


/**
 * Populates the Configuration bean with the properties that have been loaded.
 *
 * @param props the props
 */
protected void loadServerConfigBean(final Properties props) {
    ((ConfigurationBean) beanFactory.getBean(this.configurationBeanName, ConfigurationBean.class)).setProperties(props);
}

  • And, finally, we need to add a call to loadServerConfigBean after all of the properties have been loaded/merged. Add the following as the last line of code in the loadProperties method:

loadServerConfigBean(props);

Okay, that was a lot of little things, but now we are done with the property placeholder configurer and populating the configuration bean. The next step is to add a bean to the Spring context to define the property placeholder.  This bean definition, along with the configuration bean definition and the definition of a DatasourcePropertyLoaderStrategy, might look like:



<bean id="configBean" class="com.objectpartners.sandbox.config.ConfigurationBeanImpl" scope="singleton"/>

<bean id="datasourcePropertyLoaderStrategy" class="com.objectpartners.sandbox.config.DatabasePropertyLoaderStrategy" scope="singleton"/>

<bean id="placeholderConfig" class="com.objectpartners.sandbox.config.StagedPropertyPlaceholderConfigurer">
    <constructor-arg index="0" value="classpath:internal.properties"/>
    <constructor-arg index="1" value="file:/my/app/config/external.properties"/>
    <constructor-arg index="2" value="datasourcePropertyLoaderStrategy "/>
    <constructor-arg index="3" value="true"/>
    <constructor-arg index="4" value="configBean"/>
</bean>

After application startup, the configuration bean would now contain the current values of all of the application properties. In fact, you could extend the framework so that the configuration bean will ultimately contain more than the union of the contents of all of the configuration files and data sources. One example might be to load in configuration values for the host ip and host name into configuration based on an InetAddress instance created during application startup.

Now we need to discuss the usage of the configuration bean. This will require a little change in programming paradigm where you want to leverage the configuration bean. First of all, in order to access properties in the configuration bean, you need to know the property keys. Therefore, it is useful to create a “configuration constants” class or interface whose sole purpose is to contain constants for every configuration key that will be of interest. For instance, if you have a property with a key of “my.first.property”, define a constant:


public static final String MY_FIRST_PROPERTY_KEY = “my.first.property”;

Next, let suppose that we have a class Foo that leverages the current value of “my.first.property” in a method bar. We can get leverage the configuration bean by wiring it into the class and using the methods on the configuration bean to access the value. For instance



package com.objectpartners.sandbox.config;

import org.springframework.beans.factory.annotation.Autowired;

public class Foo {

    @Autowired
    private ConfigurationBean configurationBean;

    public static final String MY_FIRST_PROPERTY_KEY = "my.first.property";

    public void bar() {
        System.out.println("value of my.first.property = " + configurationBean.getValue(MY_FIRST_PROPERTY_KEY));
    }
}

The Last Step – JMX

Now we are down to the last step – allowing the configuration values to be updated dynamically via JMX. The quickest way to do this will be to make the Configuration Bean a managed resource.

Add the following to your Spring configuration:


<context:mbean-export />
<context:annotation-config />
<context:component-scan base-package=<package> />

Next, add the following to your implementation of the ConfigurationBean, right before the class declaration (it is assumed that you have subclassed ConfigurationBeanImpl, extended the ConfigurationBean interface, or both):


@Component
@ManagedResource(objectName="example:name=configuration", description = "Configuration")

Finally, for any methods that you want to be exposed via JMX, add the following before the method declaration:


@ManagedAttribute

For any getters or setters decorated with the @ManagedAttribute, you will now be able to view the return values of those methods as attributes on a configuration mbean using your favorite JMX client (I would recommend JVisualVM for day-to-day use). Additionally, any getters and setters decorated with the @ManagedAttribute annotation will be presented as an operation on this mbean. Lets suppose that we go with the base implementation of ConfigurationBean, and add these annotations (entire class not listed, just the pertinent portions):


@Component
@ManagedResource(objectName="example:name=configuration", description = "Configuration")
public class ConfigurationBeanImpl implements ConfigurationBean {

    /** The props configuration key-value pairs */
    private Map<String, String> props = new HashMap<String, String>();

    /**
     * Sets the specified property with the specified value
     *
     * @param key
     * @param value
     * @return the previous value of the property
     */
    @ManagedAttribute
    public String setProperty(String key, String value) {
        if (props == null) {
            props = new HashMap<String, String>();
        }

        return props.put(key, value);
    }

    /**
     * Returns the value for the specified key, or null if no such key exists.
     *
     * @param key
     * @return the value
     */
    @ManagedAttribute
    public String getValue(String key) {
        if (props == null) {
            return null;
        }

        return props.get(key);
    }

In the JMX console, you will now be able to retrieve the current value for any property key using the getValue() operation, and will be able to dynamically update the value of any property via the setProperty() operation.

Additional Considerations and Final Thoughts:

There are some considerations that should be kept in mind if you plan on leveraging this framework or extending it:

  • Allowing any configuration value to be updated dynamically via JMX might not be desired functionality. If there are certain values that you might want to update via JMX, and certain ones that you would not want to, then you can implement more granular methods in your configuration bean, and only expose those methods in JMX.
  • It should be noted that the @ManagedAttribute annotation is only valid for getters and setters. If you want to expose an operation for some other type of method, use the @ManagedOperation annotation.
  • There are certainly situations where it does not make sense to allow dynamic updates of configuration values, or where it would have no effect. For instance, suppose your database driver is a configuration value, and it is used to set up your data source. That is probably only going to happen once, at application start up. Changing the configuration value after application startup is not going to have any affect in such a situation.
  • There would also be a multitude of types of configuration that you would not want to be available for dynamic update at all. This is why it generally would make sense to extend (or replace) the ServerConfig interface, and only include those operations that you absolutely need exposed in JMX for your application.
  • Remember that configuration values do not need to be single values. It is common to have comma-separated lists for configuration values. Managing these types of configurations via JMX may take some specialized operations (e.g., implementation of some operations from the List interface, exposed as JMX operations)
  • This is by no means a perfect framework, and could be refined and extended in many ways. For instance:
    • Rather than only allowing one internal property file, one external property file, and one Datasource property loading strategy, you could easily update the framework to allow multiples of each

It should also be noted that there have been a lot of updates in relation to how properties can be loaded within Spring 3.  Those capabilities are well worth looking at as well.

Leave a Reply

Your email address will not be published. Required fields are marked *

*

*