Use SpEL to Conditionally Suppress JSON Fields During Serialisation

If you use the Jackson Framework to serialise objects to Json, exposed through a Spring REST API,  then you might often come across a requirement to suppress the field dynamically during serialisation when a condition is met. This example shows how to extend the framework by implementing the PropertyFilter stereotype to use Spring Expression Language (SpEL) expressions to conditionally write out the field and value at the point of serialisation.

The Jackson framework provides customisation to conditionally filter out Java object values with the PropertyFilter interface. This is implemented within the framework as a BeanPropertyFilter and SimpleBeanPropertyFilter which provides some static functions to filter out different properties on the target bean being serialised. This is accomplished by adding the filter to the Object Mapper within the context and then telling Jackson to apply that filter to certain beans by annotating the bean class with the JsonFilter stereotype .

These basic functions are useful but don't allow the developer to provide any more complex conditions and logic. For example, you might wish to only include an integer Json field and value in output if the value in the object is greater than a predefined value.

Spring Expression Language is perfect for this job; it provides a framework to inject complex conditional logic through an expression represented as a String. Using this in a new annotation which can be added to serializable beans will allow us to easily define expressions which conditionally suppress Json properties.

I first created a new stereotype to mark a Java field and define an expression used during serialisation.



I then created an implementation of the Jackson PropertyFilter interface which looks for fields marked for conditional suppression and applies the expression to that field value.



The function injects the bean instance into the SpEL evaluation context and ensures that the expression evaluates as a boolean and throws out a SpEL Exception with an appropriate error code if not.

SpEL provides convenient syntax to refer to values within the context at runtime. However, typing these out in an expression specifically for a field seems overly clunky so I abbreviated that syntax with a substitute character and replaced it during the evaluation.



I added some further configuration to ensure the SpEL Filter is injected into the Object Mapper in the Spring Context, whether it's already there or not, and an annotation to enable the functionality.



The filter works as intended. When coupled with the existing JsonFilter annotation it is very easy to define complex logic which conditionally suppresses Json elements.



The full source can be found on my public GitHub or integrated into your project directly from Jitpack

Spring Boot Caching

In a microservices environment caching is an important none functional requirement which can help reduce load overhead, latency and improve the user experience. Where ever data is loaded, particularly from an HTTP endpoint on another service, caching should be considered. In Spring Boot it is easy to implement caching on any function and the cache itself can be implemented independently of the service in which it is applied.

When to Cache?

In a microservice environment multiple calls are made to and from different services within the context of a single request to a gateway. In a 'database per service' architecture the same data is retrieved from the same end point multiple times throughout the course of the initial request. Where this is done in the same thread it might be tempting to wrap the data and pass it down the chain. However, this approach breaks the single responsibility rule and restricts the extensibility of our independent services. Caching is a convenient answer but introduces potential problems. 

The amount of data cached, length of time it is cached for and amount of times it is used are all  variables which must be traded off for the price of improved performance. If we cache data for too long or don't refresh it often enough then we run the risk of error or even perceived corruption within the system. Conversely, if we don't cache long enough or refresh too often, we don't get any benefit from the cache.

Let's first consider how to enable caching in the Spring Boot application and introduce a simple cache to improve the performance of the system. In this example an application uses a Feign client to retrieve the same data multiple times from an endpoint on another service via HTTP. These are expensive and caching will clearly help improve performance.

Enable Caching in the Application

To enable caching we need to bring in the relevant dependencies and annotate a class loaded into the Spring container. As usual Spring Boot provides a convenient starter.

Add the enable caching annotation on the initialising class with the main method.

Caching the Data

The @Cacheable annotation tells Spring to cache the data returned by the method against a generated key on the first request and then takes that data from the cache whenever the same parameters are used. In this case a repository method uses a Feign client to call the endpoint. It is this method which we annotate and tell Spring to cache the data it returns against a key generated from the method parameters.

So far we've introduced the concept of caching to the system and told Spring to cache beans returned by the annotated method in the cache called 'mycache'. Now we need to implement and configure that cache. By default Spring will implement a simple in memory Map via the default CacheManager bean. We can override that by defining our own CacheManager implementation in some custom configuration. Spring provides many different manager implementations as hooks for implementing different well known cache providers, such as Ehcache.

Implementing a Cache

In the above example we have told the application to cache data returned from a method the first time it is called with the given parameter and then retrieve that from the cache each time that parameter is used after that. Currently this doesn't solve any of the trade offs mentioned earlier. We only want the same data to be kept in the cache for the duration of time it will be used for. If that data changes after a period of time we want to ensure the call to the method refreshes whats in the cache with data from the actual service that provides it. To solve this we must implement some constraints on the cache to ensure we refresh the data when required and prevent the cache growing too large which could cause resource issues.

Configuration for Ehcache requires us to introduce a config xml file to the classpath. If you're not a fan of this and prefer code configuration, as is the Spring Boot standard then a handy SourceForge library provides some alchemy for wrapping the Ehcache config into a Spring CacheManager.

The max entries, TTL and TTI values are loaded from our applicaiton properties/yml file or from a config server if we're using one. Max entries limits the number of objects in the cache and a 'Least Recently Used' policy manages eviction behind the scenes. Time To Live limits the maximum amount of time objects can reside in the cache. Time to Idle limits the time an object can reside in the cache without being used and should be set to a lower value than TTL. Both these values are critical to the success of the cache. Set them too low and we reduce the effectiveness of caching. Set too high and we run the risk of using data which isn't fresh and causing unintentional error downstream. This trade off can only be optimised through performance testing.

Cache Keys

Along with other configuration we can create a customised KeyGenerator bean. By default Spring uses a SimpleKeyGenerator class which either uses the hashCode of the a single parameter or combines multiple parameters to store the objects against in the cache. If you're using some kind of correlation Id to uniquely identify user request to the gateway then it might be usesful to pass this into the method as a parameter or create a custom KeyGenerator bean to key data so that we ensure different user requests and threads don't use the same cached data. However, it really does depend on your use case.

Testing 

It might be tempting to test that caching is working by using a Mock framework to mock the feign client. I've found this doesn't work because the Mock proxy intercepts the method call before the Cachable interceptor. I used Wiremock to stub the client call instead and verified the number of calls made doesn't increase after the first method request. We can also autowire the CacheManager into the test to access the cache and test its contents.

Spring Cloud AWS - Parameter Store

Externalising configuration is key to managing and automating the deployment of an application into different environments. Since Spring Boot all application configuration can be managed in one single properties or yaml file which can be baked into an image, added to a container or pulled from a source at runtime. Spring Cloud AWS now integrated with the System Manager which allows application configuration to be externalised into the Parameter Store: a simple name value store which allows access to parameter values to be controlled with IAM roles. Parameter values are read from the store and automatically injected into the application by the property placeholder configurator, negating the need for application properties to be stored in a file.

As with all cloud services, the key to its effectiveness is governance. We can restrict access to different parameters, or different named parameter groups, using IAM policies attached to the resource running our application. Coupled with Spring Profiles, this allows us to create a powerful process for the management of environments within the same account and VPC.

In this simple example I'll create a Spring Boot application which will run on an EC2 instance and read its configuration from the System Manager Parameter Store. I will show how to ensure that the application accesses only those parameters specific to the environment in which its deployed using Spring Profiles and IAM policies.

First we'll create the simple Spring Boot application and import the Spring Cloud AWS SSM dependency. It's worth ensuring that we only import the Spring Cloud dependencies we need otherwise convention over configuration will attempt to access lots of other AWS resources we don't want to use.

No we'll create two parameters in the SSM store with the same property name but for different environments: development and production. Creating a new parameter is quite simple and requires a key, description and a value. Spring Cloud AWS requires the key to be named in a specific format which can be configured in a bootstrap.yml file but defaults to:

/config/application.name/

The key must start with a forward slash, can be configured in a boot strap parameter named aws.paramstore.prefix and defaults to config. The second path element of the key defaults to the name of the application as defined in parameter spring.application.name

The rest of the key, after the application name element, must be the name of the parameter identified in the application. e.g.

/config/application.name/some.parameter.name

In the application this could be referenced with the following annotation:


The application name element of the key can be suffixed with an environment label allowing us to specify different values for development and production, e.g.

/config/application.name-production/some.parameter.name

/config/application.name-development/some.parameter.name

Now that we have two different parameters for each environment referenced by the same name we can put some access control in to ensure that only those values can be accessed by resources running in the appropriate environment. The following IAM policy will be attached to the production EC2 instances, on which the Spring Boot app will run, and will ensure that only production parameter values cannot be read by production resources. A similar policy can be created for development resources.

When we run the application we can activate specific Spring Profiles: development or production and Spring will automatically attempt to access the named parameter suffixed with that profile name. For example, to run the application in production:

java -jar -Dspring.profiles.active=production my-app.jar

So long as we can ensure the EC2 instance, or container, running the application has the correct IAM role associated we can be certain that only those parameter resources for that environment will be accessed.