Integrating SE Configuration with Java EE

Integrating SE Configuration with Java EE

A lot of people asked about integration of configuration mechanisms with Java EE, e.g. within Java EE 8. So I write down my current thinkings on this topic here, looking forward for your opinions and thinkings.

Nature of a Deployment Descriptor

Let's start analyzing quickly what the essence of a typical EE configuration file is, e.g. a web.xml deployment descriptor, a JPA persistence.xml or a CDI beans.xml , a log4j.properties or a container specific descriptor for a datasource. What all this rather different configurations have in common? Fortunately there are quite a few common aspects:

• all configurations mentioned are files.
• all configurations must be deployed to some well defined locations.
• all its contents is basically of literal nature, in MIME types text/xml or text/plain.
• all adhere to some standards or defined formats

Advantages of Literal Configuration Files

As a consequence we have some interesting options:

• literal configuration can be placed somewhere else
• literal configuration can be "enhanced" with some kind of resolution mechanism (e.g. using expression language)

In other words, we can place our configuration at more or less any arbitrary location, e.g.

• we can support additional locations on the classpath
• we can support additional file locations, e.g. as part of a domain deployment
• we can read the configuration files from a simple web server
• we can call some restful services to get them
• we can manage them within a database
• we can deploy Java classes that act as factories for providing the descriptors

Integrating with Java EE

The current Java EE stack is quite simple to benefit from a configuration mechanism, when some rather simple preconditions are met:

• First of all backward compatibility must be ensured. So it must be possible to read configuration from the same places as before.
• With the new configuration mechanism in place it must be defined, how additional (external) configuration is related to probably already existing configuration
• With convention-over-configuration in mind external configuration should override configuration at the default locations, because without external configuration everything works as expected. With external configuration provided the defaults are expected to be overridden without any further flags to be set.
• As another variant external configuration may also extend default configuration.
• Nevertheless due to security concerns some constraints may be possible, these must be configured similarly.

Basically existing JSRs also can use a configuration service in different ways:

Integration on file or decriptor level

The most simplest way for integration is on file or decriptor level. Hereby each JSR can define and register the default locations of the deployment descriptors within the configuration service, e.g. using "meta-configuration":

  <? xml version="1.0" encoding="UTF-8" ?>
  <configuration>
    <config name="jpa.persistence-units" type="input-files">
       <location>classpath*:META-INF/persistence.xml</location>
       <override-policy>override</override-policy>
     </config>
  </configuration>

Each JSR then can use the configuration service to read the descriptors, e.g. JPA could read its persistence.xml files as follows:

  Configuration config = ...;

  Collection<InputStream> persistenceUnits =

       config.get("jpa.persistence-units", Collection.class);

  for(InputStream is:persistenceUnits){

       // read persistence unit

  }

The biggest advantage of this approach is that the existing JSRs require only a few minimal code changes to benefit from the new configuration service. Another advantage is that the configuration service does not any kind of knowledge on the internals of the configuration, it simply manages files.
Nevertheless dynamic configuration resolution may be more complicated, since the configuration service does not have any know how about the internal configuration structure. Nevertheless dynamic resolution is still possible (see later in this post).

Note: the example above uses InputStream as abstraction of a file resource. Alternate types are to be discussed here, such as String or URI.

Full Integration

Another approach would be to use the configuration API to map a JSR's configuration directly. E.g. a persistence.xml can be mapped as follows:

persistence-unit.myapp.provider=\
                        org.hibernate.ejb.HibernatePersistence
persistence-unit.myapp.jta-data-source=java:/DefaultDS

persistence-unit.myapp.properties.myprop=myPropValue
persistence-unit.myapp.properties.myprop2=myPropValue2

In this case all types of configuration mechanism can be used to provide such a configuration. Basically it would also be possible to map an existing descriptor/configuration format by providing some according conversion classes.

Dynamic Value Resolution

In many cases dynamic resolution of values simplifies things a lot (used too widely things get more complicated similarly). Also in many cases it would be sufficient to provide a basic configuration template and allow the configuration service to manage the dynamic replacements. This can be achieved, e.g. by allowing EL to be used within a configuration, e.g.

<persistence>  <persistence-unit name="myapp">    <provider>              
      org.hibernate.ejb.HibernatePersistence
    </provider>    <jta-data-source>$[config.myApp.myappDS]           </jta-data-source>
    <properties>

      ... 
    </properties>  </persistence-unit>
</persistence>

or

   persistence-unit.myapp.jta-data-source=$[config.myApp.myappDS]

Summary

Summarizing there are a couple of possible integration points. Basically all kind of EE descriptor files can be deployed at arbitrary locations, locally or remotedly, encrypted or plain or whatever is needed. With EL included dynamic resolution can be easily achieved. Finally existing JSRs can integrate with configuration services either on descriptor/file level or define a full configuration and key mapping.

Separation of concerns into a basic configuration service and how the services are integrated with Java EE, enable also a flexible collaboration and open discussion with all the other specifications. It is also important that things are discussed early, so the other specifications have time to adopt the new features. As another side effect the same configuration mechanism can similalry be used for administrative resources or alternate environments, e.g. Tomcat, Jetty etc.

Configuration Formats and Locations

Configuration Formats and Locations

Configuration Formats

Configuration data can be stored in various formats. With the JDK a few possible formats are included by default:

• .properties files, readable with java.util.Properties enable storing simple key, value pairs in an ISO-8859-1 encoded text file, also supporting Unicode escapes.
• The same java.util.Properties class also provides a corresponding .xml formatted variant, which benefit from all the xml encoding options.
• Parameters passed with -Dkey=value on the Java command line are accessible from System.getProperties().
• Finally environment properties inherited by the underlying runtime platform are accessible from System.getenv().

All this mechanisms are provided by the Java SE platform out of the box and therefore are widely used. But there are for sure more possible formats that might be used as source of configuration, e.g. other xml formats, JSON or databases. Therefore it makes sense to model the configuration format explicitly, so custom (or legacy) formats can be supported easily:

public interface ConfigurationFormat{

    String getFormatName();

    boolean isAccepted(URI resource);

    Map<String,String> readConfiguration(URI resource);

}

Implementations of this class can be simply registered using different component loading mechanism, such as java.util.ServiceLoader or, in case of Java EE, alternately as CDI managed bean. Access to the formats can be obtained by a corresponding singleton, which provides 

• access to common formats, such as property, or xml-property files.
• access to other (registered) formats by name
• access to all currently registered format names
• access to a matching format given an URI of a resource.

public final class ConfigFormats{

    private ConfigFormats(){}

    public static ConfigurationFormat getFormat(                                          String formatName);

    public static Collection<String> getFormatNames();

    public static ConfigurationFormat getFormat(URI resource);

    public static ConfigurationFormat getPropertiesFormat();

    public static ConfigurationFormat getXmlPropertiesFormat();

}

Also the singleton accessor for accessing predefined maps can be easily enriched by corresponding methods (though with increasing complexity and similar method signatures building a Builder maybe more appropriate):

public final class PropertyMaps{

    private PropertyMaps(){ }

    // factory methods

    ...

    public static PropertyMap fromPaths(

              Map<String,String> metaInfo,
              ConfigurationFormat format, String... paths);

    public static PropertyMap fromPaths(ConfigurationFormat format, 

              String... paths);

}

Configuration Locations

Similar to the fact that configuration data can be formatted differently, configuration can be also be read/accessed from different locations:

• as class path resources
• as files on the locale file system
• as resources accessible from a web server (or configuration server)
• as remote data accessible from a configuration bean (EJB, managed bean, ...)
• ...

In the above examples/API we simply pass a literal path to locate/define a configuration. Hereby the idea is that the path is formatted in a way, so multiple location mechanisms (called readers) can be transparently added/registered to the configuration system. A configuration resource then can be defined as <reader>:<location> (the ones, who know Spring will possibly see some similarities with Spring's Resource API). Examples of valid configuration resources can be:

• classpath:cfg/test-*.xml
• classpath*:cfg/${STAGE}/*.xml
• file:${APP_DIR}/cfg/envconfig/*.xml
• url:http://myconfigserver.intra.net/config/${STAGE}/get?appID=MyApp
• ds:[ConfigDS]SELECT a.key, a.value FROM Config a WHERE a.appID="MyApp"

Hereby 

• classpath uses ClassLoader.getResource(String), also supporting Ant-like path expressions
• classpath* uses ClassLoader.getResources(String), also supporting Ant-like path expressions
• file locates files on the local file system, also supporting Ant-like path expressions
• url uses new URL(String), in the example above calling a Restful service
• ds accesses configuration data using an OQL query, reading from the ConfigDS datasource.

The exact syntax for path expressions, of course, can be discussed and improved. Dynamic parts basically can be implemented using expression language (EL) extensions.

A ConfigurationReader hereby can be modeled by a simple interface as illustrated below:

public interface ConfigurationReader{

    String getReaderName();

    Map<String,String> readConfiguration(String readerPath);

}

Similarly to formats, readers can be managed and accessed/tested from a ConfigurationReader singleton:

public final class ConfigReaders{

    private ConfigReaders(){}

    public static ConfigurationReader getReader(                                          String readerName);

    public static Collection<String> getReaderNames();

}

So given this interfaces and accessors our configuration model now is capable of supporting more or less every type of configuration, as long as its mappable to Map<String,String>. It does not imply any constraints, how configuration must be stored and managed in an enterprise, nor does it constrain the format of the input source. 

But even with that, there are additional things that must be considered:

• Configuration may also change. Mechanisms must be provided so configuration changes can be propagated to interested parties, both locally optionally also remotely. Such changes might also be propagated across VM boundaries e,g, by passing a serialized ChangeSet or Configuration over the network.
• All examples as of now were programmatically defining the configuration to be used. Typically in an enterprise context this is determined by some configuration meta-model (aka meta-configuration).
• Even worse within an application server running multiple enterprise / web applications several classloaders are active. As a consequence configuration that is provided on the classpath must be isolated along the corresponding classloader and its child class loaders.
• Also we have not yet discussed how our configuration service can interoperate / being integrated in more detail within an EE environment. Integration hereby must be enabled on a global or domain level, e.g. for configuring administrative resources, but also interoperate with CDI, enabling powerful injection of configuration.

These are the topics for the upcoming blogs, so stay tuned!

Composite Configuration

Composite Configuration

Modeling Common Aspects

Looking at Configuration my working analysis was to model it mainly as a Map<String,String> with additional meta data added. As we have seen this concept comes with several advantages:

• The basic API ( java.util.Map) is already defined by the JDK.
• Since keys as well as values are simple Strings, we inherit all the advantages of the final and immutable  String class, like type and thread safety.
• since we constraint our API to this simple types, we ensure no or minimal overlaps with CDI in the EE context.
• our model is fully compatible with Java SE, providing therefore maximal compatibility also with the SE platform.

Applied to the configuration format we would define two distinct artifacts:

• a PropertyMap, which models the minimal requirements for a configuration map.
• a Configuration, which extends PropertyMapand provides additional functionalities, such as extension points, type support etc.

public interface PropertyMap extends Map<String,String>{

    Set<String> getSources();

    Map<String,String> getMetaInfo(String key);

    Map<String,String> getMetaInfo();

    void reload();

    boolean isMutable();

}

public interface Configuration extends PropertyMap{

    String getConfigId();

    Boolean getBoolean(String key);

    Boolean getBooleanOrDefault(String key,
                               Boolean defaultValue);

    Byte getByte(String key);

    ...

    <T> T getAdapted(String key, PropertyAdapter<T> adapter);

    <T> T getAdaptedOrDefault(String key,
                    PropertyAdapter<T> adapter, T defaultValue);

    <T> T get(String key, Class<T> type);

    <T> T getOrDefault(String key, Class<T> type, 
                                               T defaultValue);

    Set<String> getAreas();

    Set<String> getTransitiveAreas();

    Set<String> getAreas(Predicate<String> predicate);

    Set<String> getTransitiveAreas(Predicate<String> predicate);

    boolean containsArea(String key);

    Configuration with(ConfigurationAdjuster adjuster);

    <T> T query(ConfigurationQuery<T> query);

}

A configuration instance then can be built using a PropertyMap, e.g.

PropertyMap myPropertyMap = ...;

Configuration config = new BuildableConfiguration
                                       .Builder("myTestConfig")
                   .withUnits(myPropertyMap);

So we can provide partial configurations by just implementing the PropertyMap interface. For convenience an AbstractPropertyMap class can be defined that additionally supports implementing this interface. 

public class MyPropertyMap extends AbstractPropertyMap{

    protected Map<String,String> initContentDelegate(){

      // in reality, provide something useful here...

      return Collections.emptyMap();

   }

}

Using Composites to Build Complex Configurations

Given the simple basic PropertyMap interface we can start thinking on how building more complex configurations by combining existing combinations. Basically the ingredients required are:

• two (or more) existing configurations
• a combination algorithm or policy

Now thinking on mathematical sets, we may provide similar functionality when combining configurations:

• union
• intersection
• subtraction

Additionally we have to think ow we should resolve conflicts (different values with the same key), most important policies are:

• ignore duplicates (keeping the original values from former entries)
• override existing previous values by later values
• throw an exception, when conflicting entries are encountered

This can be modeled by a corresponding policy enum:

public enum AggregationPolicy{

    IGNORE,

    OVERRIDE,

    EXCEPTION

}

Finally we can provide a factory class that provides as 

• commonly used property maps by reading from resolvable paths, using common configuration formats, e.g. .property-files (the resolution capabilities hereby can be extended by implementing and registering a corresponding SPI)
• most commonly used compositions of partial configurations (maps)

This can be modeled with a simple singleton as follows:

public final class PropertyMaps{

    private PropertyMaps(){ }

    // factory methods

    public static PropertyMap fromArgs(

              Map<String,String> metaInfo, String... args);

    public static PropertyMap fromPaths(

              Map<String,String> metaInfo, String... paths);

    public static PropertyMap from(

              Map<String,String> metaInfo, 

              Map<String,String> map);

    public static PropertyMap fromArgs(String... args);

    public static PropertyMap fromPaths(String... paths);

    public static PropertyMap from(Map<String,String> map);

    public static PropertyMap fromEnvironmentProperties();

    public static PropertyMap fromSystemProperties();

    // combinations

    public static PropertyMap unionSet(

              PropertyMap... propertyMaps);

    public static PropertyMap unionSet(

              AggregationPolicy policy,
              PropertyMap... propertyMaps);

    public static PropertyMap intersectedSet(
              PropertyMap... propertyMaps);

    public static PropertyMap subtractedSet(

              PropertyMap target, PropertyMap... subtrahendSets);

    public static PropertyMap filterSets(

              Predicate<String> filter, PropertyMap propertyMap);

}

With the given mechanism we are able to define complex configurations, realizing some complex override and configuration rules quite easily:

String[] cliArgs = ...;

Map<String,String> defaultMap = ...;

Configuration config = new BuildableConfiguration.Builder(
                            "myTestConfig").withUnits(

       PropertyMaps.from(defaultMap),

       PropertyMaps.fromPaths("classpath:test.properties"),

       PropertyMaps.fromPaths("classpath:cfg/test.xml"),

       PropertyMaps.fromSystemProperties(),

       PropertyMaps.fromPaths(
                  "url:http://1.2.3.4/remoteCfg.xml"),

       PropertyMaps.fromArgs(cliArgs),

      )

      .build();

Basically the above creates a full fledged Configuration instance that:

• is built from properties contained in the given default map.
• that may be overridden by entries in test.properties, read from the classpath
• that may be overridden by entries in cfg/test.xml, using the JDKs xml property format (also read from the classpath)
• that may be overridden by entries from the resource loaded from http://1.2.3.4/remoteCfg.xml
• that may be overridden by entries  from the CLI arguments

Of course, this example uses always the same keys for all different partial configuration sources, which might not be a realistic setup. But adding a mapping of provided keys to some other keys is basically a trivial task.

Summary

Summarizing separating configuration into a simple basic interface (PropertyMap) and a more complex extended variant (Configuration), allows us to easily build composite configurations by combining more simpler partial property maps. Most commonly configuration locations, formats and combination strategies can also provided easily by according factory classes. Also in most cases, implementing the more simpler PropertyMapinterface should completely sufficient.

Putting all this to reality, we have defined a quite powerful mechanism, that allows us to implement also complex use cases with only a few abstractions.

Adding Additional Type Support to Configuration

Adding Additional Type Support to Configuration

In my previous blog I discussed roughly what configuration is. I also stopped on a working assumption that configuration mainly is a Map<String,String> type with additional metadata. In this blog I would like to elaborate, how non literal types can be supported. Summarizing I would define the following requirements:

• it should be possible to access configuration as non literal type
• all types contained in java.lang should be supported.
• nevertheless arbitrary other types should also be enabled
• it should be possible to register "converters"
• it should also be possible to pass a matching "converter" programmatically

First of all we have to think about, what kind of functionality we want to add here to the basic Configuration interface (this is also the reason why converter is written in italic face above).

Basically adding type support requires a configuration entry's value, that is a String to be compatible with some arbitrary type:

 "...allows the interface of an existing class to be used from another interface."

This is exactly the GoF's adapter pattern. So let as define an adapter:

@FunctionalInterface

public interface PropertyAdapter<T>{

  <T> T adapt(String value);

}

Now we can extend Configuration to allow access on configuration using such an adapter:

public interface Configuration{

   ...

   <T> T getAdapted(String key, PropertyAdapter<T> adapter);

}

Obviously this is a very minimalistic approach. Now thinking on all basic types defined in java.lang, it is a good idea to add corresponding convenience methods:

public interface Configuration{

   ...

   Character getCharacter(String key);
   Byte getByte(String key);
   Short getShort(String key);
   Integer getInteger(String key);
   Long getLong(String key);
   Float getFloat(String key);
   Double getDouble(String key);     

}

By default, I would suggest throwing a RuntimeException, if a value is missing, is a good idea, so these methods never will return null values. Additinoally it might b e a good idea to let also default values to be returned, so we add also the following methods:


   Character getCharacterOrDefault(String key,
                                   Character defaultValue);
   Byte getByteOrDefault(String key, Byte defaultValue);
   Short getShortOrDefault(String key, Short defaultValue);
   Integer getIntegerOrDefault(String key, Integer defaultValue);
   Long getLongOrDefault(String key, Long defaultValue);
   Float getFloatOrDefault(String key, Float defaultValue);
   Double getDoubleOrDefault(String key, Double defaultValue);

   <T> T getAdaptedOrDefault(String key, Adapter<T> adapter,
                             T defaultValue); 
With the above signatures passing null as a default value is completely valid. So one might write:

Byte myNumber = config.getByte("minNumber", null);
if(myNumber==null){
   // do whatever needed
}

Though such code above is quite common, it might be worthwil to think on additional utility functionality, e.g. using JDK 8 features:

Configurator configurator = Configurator.of(config);
configurator.forByte("minNumber", MyNumbers::configure};

Though it would be interesting to investigate this area more, I would like to go one step back and ask, if a single String configuration value is always enough to create/implement every type of adapted interface. We have seen that a configuration entry is basically

• a configuration key
• a configuration value
• configuration entry metadata

So to cover that we would extend our interface slightly as follows:

@FunctionalInterface

public interface PropertyAdapter<T>{

  <T> T adapt(String key, String value,
              Map<String,String> metadata);

}

 
But this is still not optimal, we would be much more flexible by passing the Configuration instance, on which the adapter is used, to the adapter implementation:

@FunctionalInterface

public interface PropertyAdapter<T>{

  <T> T adapt(String key, Configuration config);

}

More generally the above is simply a specialization of a query against a configuration, but just for one specific key:

@FunctionalInterface

public interface ConfigurationQuery<T> {
   T queryFrom(Configuration config);
}

Now these concepts can be useful, when thinking on collection support. Image the following configuration:

foo.a=aValue

foo.b=bValue
foo.c=cValue

Given this configuration foo can be adapted in different ways:

• It can be adapted to a Map with a=aValue, b=bValue, c=cValue
• But it can also be mapped to a List (interpreting a,b,c as ordering predicate) with aValue, bValue, cValue.
• Similarly it could also be mapped to a tree:
            (root)
           /  |   \
         a    b    c
         |    |    |   
    aValue  bValue cValue

With the extended adapter definition all that is possible.

With that our Configuration is already very flexible, for example think on the following entry: 

javax.persistence.unit.MyUnit=https://myconfigserver.net/myApp/persistence/MyUnit.xml?myinstance=${instance.host} 

This entry could reference a persistence.xml, that is provided remotely. With ${instance.host} EL could be used to enable also dynamic aspects included into the configuration.
Now an adapter might return a String, containing the descriptor file, but this would load the configuration completely into memory. As an alternate we might allow to return an InputStream:

InputStream myUnitConfigStream = config.getAdapted(
                           "javax.persistence.unit.MyUnit",
                           URLResolver.of());

URLResolver would implements hereby an adapter that creates an URL and tries to load it, returning the InputStream. It is accessed using a static factory method:

public final class URLResolver implements PropertyAdapter<InputStream>{

  private static final URLResolver INSTANCE = new URLResolver();








  private URLResolver(){}
  
  public InputStream adapt(String key, Configuration config){
    try{
      URL url = new URL(config.getValueOrDefault(key, null));
      if(url!=null){
        return url.openStream();
      }
    }
    catch (Exception e){
      // log error

    }
    return null;
  }

  public static URLResolver of(){
    return INSTANCE;
  }

}
  
So we have seen that with rather small additional to the Configuration interface, we already have gained much flexibility, what we can do with it. Thinking on the new features of Java 8 configuration will be for sure get much more fun than it was in the past.
I would be interested, on what you think would be useful scenarios, using the mechanism presented in this blog. So you are invited to leave your comments and ideas!