A Proposal for Java EE Configuration

Requirements for Java EE Configuration

In this post I want to outline, what I see a reasonable scope for a Java EE Configuration JSR. I would first try to summarize different use cases, people want to have to be covered. In a next part I try to deduce corresponding requirements. Finally I will try to discuss, what is needed to cover these aspects.

Use Cases

First I will summarize the use cases that were mentioned so far in the discussion, especially also here.

1. An enterprise wants to centrally control their deployments (could be cloud, but is not restricted to). Hereby according configuration should be managed externally within a database or some other kind of data container. The configuration should be accessible with an administrative console and it should be possible to change configuration and remotedly update running instances affected by a configuration change. Especially administrative resources, such as data sources, thread pools, users, roles etc. should be configurable, so these aspects can be managed in a central location.
2. For different stages or runtime environments different CDI components should be loaded that reflect the different runtime environments. This includes configuring CDI, Enterprise Java Beans and similar aspects (basically everything that can be configured with some kind of deployment constructor deployed with an ear or war archive).
3. Depending on the stage and some additional aspects different configuration must be used (including, but not limited to deployment profiles). Advanced use cases include network zones, host, tier, server instance and more. Hereby the possible properties are not constraint and may heavily differ between applications and especially enterprises (runtime environments). Many applications use a file based configuration approach, which is often cumbersome and error prone.
4. It should be possible to adapt/reconfigure COTS applications to better accomodate them into the concrete environments. It should be possible to deploy them on different stages, without having to redeploy anything (also because traceability of the deployment may be a legal requirement).
5. It should be possible to provide configuration that goes beyond what is defined by the current EE standards. E.g. there are companies that redeploy the exact some jars in different setups, but reconfigure them (e.g. in multitenant, multiproduct scenarios). Thinking in extremes that could mean having a single application deployed multiple times very differently.
6. Configuration should be accessible both using a Java API, as well as being injected using CDI mechanisms.
7. It should be easily possible to use this configuration mechanism also in testing. Developers should be capable of easily configure their test setup that should be loaded.
8. Similar to views in relational database configuration should be scoped, so only a subset is visible. This can be required for example because of security concerns.
9. Finally for multi-tenant/SaaS setups configuration must be dynamic, so a SaaS provider can lookup configuration programmatically.

Requirements

This list is quite probably not complete. Nevertheless it is possible to deduce some requirements:

1. It must be possible to store and manage configuration at an arbitrary place independent of the target deployments.
2. It must be possible to store and manage configuration along the target deployments.
3. It must be possible to provide configuration with deployment artifacts (jars, wars, ears)
4. It must be possible to mix configuration from different locations.
5. It must be possible to priorize configuration.
6. It must be possible to combine configuration in different ways (override, extend, ignore duplicates, ...).
7. It must be possible to change configuration and trigger the changes to interesting observers.
8. Configuration must be loadable within early boot of a system, where no EE context is available.
9. Configuration must be adaptable depending on the current runtime context.
10. Configuration must be accessible/injectable using CDI mechanisms.
11. Configuration must be accessible using a Java based API.
12. Configuration services must allow to configure administrative resources, such as datasources, users, roles.
13. The runtime environment must be modeled in a flexible way, because different companies have different deployment/runtime and consequenty configuration needs.
14. Configuration must support staging. It would be possible to define a minimal staging type applied to an environment, But such a concept would require to allow concrete sub-stages to be modellable to accommodate also complex runtime environments.
15. Configuration should be able to support a wide range of EE.  If some areas would be excluded, we would stop half-way and destroying much of the benefits a powerful configuration mechanism can provide, since we in such case have to do workarounds as before.
16. Configuration may be marked as read-only. An simple accessor should allow to determine if configuration is mutable.
17. Configuration must be thread-safe.
18. Whereas configuration itself must not be serializable, it must be possible to extract a current state of configuration and serialize this state (e.g. as immutable configuration). This enables sending configuration easily over network connections.
19. Configuration must also support aspects that are currently not part of any EE standard, especially it must be possible to also configure application aspects. Without that you simply have to do workarounds to mitigate this lacking feature, distrying the benefits of other configurations.

The list is for sure not yet complete, but I tend to say the most important aspects are covered. If you are something missing, let me know, so I can extend the list above.

Conclusions

Give the huge areas of use cases above, one may think, it it will not be possible to define a useful scope for a JSR. I would fully agree, if one would try to solve everything from scratch modeling every aspect explicitly. But why should we? Configuration is per se quite an abstract concept, as we have seen also in previous blogs here. So lets keep configuration abstract and simple (KISS principle):

Configuration = Map<String,String> + Metadata

Metadata = Name + Map<String,String>

Finally we need some mechanism to access configuration. Given the requirements above it is not useful, to define configuration access as an interface injectable by CDI only, since it will prevent us of using configuration at early stages, where CDI is not loaded yet. Similarly it would not be possible to use configuration for configuring administrative resources. But if we would fail to cover these requirements, it is quite useless to file a JSR for this topic.

So basically configuration must IMO be provided by some simple accessor as a global service. Now some people might argue that this is not EE styled. So I would ask, what EE is all about, so I look up Wikipedia:

"Java EE provides an API and runtime environment for developing and running enterprise software, includingnetwork and web services, and other large-scale, multi-tiered, scalable, reliable, and secure network applications. Java EE extends the Java Platform, Standard Edition (Java SE), providing an API for object-relational mapping, distributed and multi-tier architectures, and web services."

So there is nothing said that is can not be done in that way. Additionally, we have some additional benefits:

• There is a clear separation of concerns. The configuration service has a clear, well defined API, how it can be accessed and what kind of features it must provide. The configuration service itself does basically not neet to know anything about its EE context, which is the way how modularized systems are built as of today.
• We have defined a simple interface, that allows us to hook in different kind of implementations and solutions to provide configuration. For example any vendor can provide additional configuration management tools and products that allow to manage your application server configuration centrally thus also creating additional benefits (and profits).
• Basically it can be left over to the implementations how the configuration provided is stored and retrieved. The JSR should focus on the mechanisms and interfaces, not on how different companies should manage their configurations.
• The interface and the access points must be defined by the JSR, but the effective data feeds can still be adapted to the needs of the concrete usage scenarios.
• Said that, this also enables us to support configuration in SE cases, especially for testing. Especially when CDI would also be configurable, it can be used (e.g. with CDI 2.0) in combination with configuration services also in test environments very effectively, which would simplify testing of EE a lot.
• Over the years the former complex and over-engineered EJB design was transformed into a lean POJO (POJI) based easy to use component model, where additional services and cross-cutting concerns can simply be annotated. It would look very strange if we would add a configuration mechanism that again only works within a EE context.

Additionally there are other possible things that might be useful:

• Finally configuration of application aspects is also required. Defining a generic and easily accessible configuration service allows to cover these aspects similarly, without having to file another JSR. Defining some additional annotations to be used with CDI for injecting configuration into managed beans should be rather trivial. Advanced use cases hereby may be ommitted and left over for future releases.
• provide configuration in different versions.
• node and cluster based configuration.

So there is IMO a strong coincidence for going that way, because

• the mechanism is simple
• the mechanism builds on top of existing interfaces (Map), hereby also supporting features such as Lambdas and streams out of the box
• the mechanism is not intersecting with CDI, because it is constraint.
• String keys and values, can easily be combined and filtered, so aspects included above can be easily supported:
• filtering/views
• templating/defaults
• intersections
• unions
• ...

One of the following key aspects would be, how we then integrate this new functîonality with all the existing JSRs? Basically there are two different possible ways to go forward:

1. The JSR itself defines, how each JSR must support adaptable configuration.
2. The JSR defines the basic mechanism. The JSRs to be included into EE8 should define on their own, how they want to make use the new mechanisms. Hereby the EE8 umbrella JSR group, lead by Bill Shannon and Linda DeMichels are coordinating the initiative.

I personally think going for (1) is not the way, it should be. I think each JSR itself has the best know how, what might be feasible points of integration with the new configuration services. Additionally this would also help to reduce the scope of the Configuration JSR drastically. As a precondition to that, the API to be used should be defined and stable very early in the overall process. My idea would be to have it defined the latest until end of this year (assuming we can file the JSR within next few weeks). So we have two years time to integrate it into the existing standards, which looks for me a reasonable time frame. At the same time I would suggest the configuration JSR provides some kind of guidelines and corresponding templates, so the other JSRs can easily adopt the new mechanism. As an example a template could look something like that:

Configuration for JSR 220
Configuration Key	Configuration Type	Availability	Expected Content
javax.jpa.persistenceUnits	java.lang.String	application/ear startup	Comma separated lists of names of the registered persistence units.
javax.jpa.persistenceUnit.[name]	java.io.InputStream	application/ear startup	Deployment Descriptor (persistence.xml) as described in section XXX

Of course, the example above is not probably not complete. But it shows the main intend quite well. Nevertheless it would still be feasible that the configuration JSR makes proposals, how the other JSRs can be accomodated to the new mechanism, if other JSRs wish us to do so.

If then finally a JSR is nevertheless failing to use the new APIs, the configuration JSR and the EE overall planning would not be affected. This greatly reduces the risk for EE8 to be delayed, but still there is a reasonable chance that we can achieve a new flexible EE platform.

Proposal and Scope

Given all the aspects described above a configuration JSR will cover the following:

Next:

• Open the JSR

Until end of 2014:

• Define the Java API as far as possible
• Define a simple map based configuration abstraction, as basically outlines in my previous posts. 
• Define a simple, but flexible and extendible Configuration Service (accessor) API. It consists of an accessor that is also loadable outside of CDI.
• Define a SPI to configure/back the implementation used by the accessor API.
• Define a configuration description template. Send it out to each JSR included in EE8, so each JSR can individually decide how the want to be configurable. The results then can be included either into updated JSRs, or/and in summarized form into the configuration JSR.

Basically this implies that the JSR will publish its early draft review until end of 2014.

In 2015:

• Discuss and coordinate configuration support with other JSRs.
• Define configuration keys/support for administrative resources.
• Provide fully functional reference implementation, so it can be included into Glassfish as one of the first modules.

This will be a reasonable scope for a Public Review during 2015.

In 2016:

• Finish the TCK and RI work and go final.

Comparing with the Proposal done by Mike Keith at last Java One

Mike Keith did several presentations during the last years at different conferences, e.g. at JavaOne 2013. 

Mike hereby did quite a good job, basically the use cases match very well, what is defined above here. So also this proposal does not target things completely differently, but given the experience of Credit Suisse, CloudBees and other companies that run huge application server farms and cloud or cloud like infrastructures, there is one crucial aspect, where I really disagree:

• Do not define another archive format to provide configuration!

There are a couple of reasons, why I think, we should not follow the idea of configuration archives as proposaed by Mike Keith:

1. We already know from experience with ears and wars that handling of archives is not always easy and often is cumbersome (especially during development). Whereas also in Credit Suisse, application configuration is delivered along the code as jar artifacts, it is only covering parts of the deployment. 
2. Additionally the JSR would have to define the format of the archive, define the stages and other sub schemas/folders so the archive covers all kind of aspects required. This would add complexity to the configuration JSR, without benefit for users.
3. In Credit Suisse Datasources, thread pools, users, security roles, certificates and other resources are not managed as file resources primarly. Within the deployment process according files are generated, by this makes the deployment inflexible, since even simple changes require a complete redeployment. Similary the deployment packages have to rebuilt for each stage. 
4. Often configuration changes are assummed to be more easily deployable, but experience has shown that this must not be the case. Additionally there might be cases, where operations has to adapt a configuration, e.g. because a database or other resources has been moved. With archives the application has to be redeployed, which is exactly, what we want to prevent.

Instead, I would let it open, how the Configuration Service effectively is locating its configuration. As shown above we can define different configurations (and locations) more effectively as shown above in the template, but without implying constraints how this configuration must be deployed.

This point has been proven in discussions during the last year with leading engineers and Java Champions. Also other colluegues have written blogs on this e.g. see http://blog.loof.fr/2013/10/configuration-management-jsr.html

So my proposal would be slightly adapted:

Higher Level Services

As already mentioned also at https://java.net/jira/browse/JAVAEE_SPEC-19 and also as commented by Arjan Tijms, we must be capable of providing higher level services on top of that very basic concepts. Most prominent use cases are the deployment descriptors, located at several different locations throughout the system, but there are also other settings that may be considered. I outlined in the previous blog, some of the possibilities to add additional functionalities using generic functional extensions. Tijms has outlined, that an additional higher level API for providing configuration information to the different components in a EE container would be useful, which may go beyond adding specialized configuration adapters. I would not disagree on that topic. 
Basically the template idea above could then similarly extended, so instead of modelling the template entries on the low level, as described above, we would model a high level API (e.g. as an interface?), that is finally provided by the configuration JSRs. This would shield existing components from the details of the configuration service.

Since different components in a EE container are configured rather differently, it might require to define multiple interfaces to be provided. I would like to postpone this at this point, since this topic is worth it's own blog entry ;-)

Feedback Required

I would like to encourage all readers to give feedback on this blog. Please do not only raise your concerns, but also give positive feedbacks, what you think. It would help to see, if this would be a feasible way to go ahead.