Exposing functionality, through a web service, is no different to exposing it through a web site and therefore the security applied to the web service should be no different. On the web, the user navigates the site by accessing links to end points. Data is mutated through calls to different HTTP methods at the end point. The user opens a browser, 'GET's the media which is rendered, mutates data, and 'POST's it back to the server: Create, Read, Update, Delete, the foundation of Resource Oriented Architecture. That's no different if our client is a program making requests to those resources exposed as web service. Think of the client as 'hand rolled' browser. A browser handles the session and a web service client will also need to deal with this programmatically. The session, controlled by the server, should be created, delivered to the client and used no differently whether it is serving up requests from a browser or a web service client.
In an ideal world we could expose the exact same controllers we use for a web site as RESTful web services. Why not? it's just the same data, accessed through the same functions, on the same service. In the real world this is unlikely to happen. Controllers in our web application may need to deal with certain nuances of the view, a web page. Next time you build a web site using MVC keep in mind whether you could access them from a client other than just a browser. Could you return pure JSON objects rather than marshaling objects to a view? It's possible, I'm sure it is, I've never done it in practice but I maintain the idea of one web application being able to expose resource representations independently of the client.
On a web site, we use the cookie mechanism to allow the browser to tie independent requests together to form a session. The cookie contains a token generated upon initial authentication and is used by the server to ensure subsequent requests are coming from that authenticated principle. The servlet container generates the JSESSIONID which is included in the cookie. This ties requests together but is also persisted on the client side. Login to a web application, receive a JSESSIONID, close your browser then open it again and go back the web site. Your previous session persists, albeit only in memory, and your are automatically authenticated. In our web service we need a mechanism to do the same. We could ask the client to authenticate with the same credentials each time. If our security is the same as our web site we may ask for the username and password upon each request. There's some obvious reasons why that isn't ideal. So, this leads us to the conclusion that we need to create some kind of token upon initial authentication, return it to the client and ask them to supply that with each subsequent request. Let's just use the cookie again.
Spring Security supports persistent cookie based authentication 'out the box' with the rememberMe service. In order to implement this we need to build on the 'vanilla' implementation we would use as a basis for authentication and authorisation across our application infrastructure. You will probably already have defined this, implemented UserDetails, GrantedAuthority and the UserService interface in your system, but I will go through the basics of that to start with.
Let's start with the user, the representation of the actor within our system. In my example system I persist these detail in a database and hence these entities map directly to database tables with JPA. I also like to serialize these with simpleframework annotations.
In my model I have two implementations of GrantedAuthority. Role and Permission, the authority of each is the name. Permissions are grouped into Roles and Roles associated directly to the user. You should note in the User class, my getAuthorities implementation returns both the associated roles AND their permissions which gives flexibility when securing my resources. Here's the Role.
And the Permission
The UserDetailsService is the only other interface needed to be implemented in order to get Spring Security up and running. This is something which is not solely used by Spring Security and will probably expose functionality to applications used to manage users and associated data. The service performs this business logic, delegates CRUD operations to repository objects, (DAOs implemented with JPA), and maintains transaction control. Here's the code snippet from the service which we implement from the framework.
Now we have our implementations we can begin to build the security context which will be loaded into the web application through the usual Spring context loader. Here's the basic stanza for the authentication mechanism.
It's worth saying here that I usually like a physical split between the web and service tiers. Spring allows you to easily facilitate this with the Spring Remote package. I use the HttpInvoker to do this which exposes the service interfaces as web services. I guess you might be thinking: Why do we need an MVC controller at all when we can just expose the service functionality through Spring Remote? This is a reasonable argument. However, the service I'm exposes as a remote interface also hosts some functionality I might not want exposing. I could of course restrict access to those functions with security annotations, as we'll see later, but there's a good argument to have that functionality hosted on a private subnet and only allow access to the top tier from the outside world. The controllers in our MVC application, mapped to URIs, simply marshal requests to the remote service interfaces and we can pick and choose exactly what we expose to the outside world.
On the service tier I expose the remote interfaces
And on my MVC context I override the userService bean as an Http client to the remote service, Spring Remote does the rest, (excuse the bad pun!).
In a web application which renders a UI through a browser we'd probably just define a basic authentication processing filter, another 'out the box' component. In our web service application we will need to customize this feature and we will revist this later.
Now we can look at the authorisation. In Spring Security this is facilitated by the Access Decision Manager which uses a list of voters. There are three choices out the box: Affirmation, Consensus or Unanimous based. For affirmation, only one voter in the list needs to vote positively to allow the authenticated principle access to the requested resource. Consensus; the majority. Unanimous; ... you get the picture. I only have one voter in the list so the manager type doesn't matter too much but you can see how extensible this can be. The RoleVoter bean checks the requests resource is configured with an authority granted to the authenticated principle.
All that's left to do is associate the authorities with the resources in our system. There's two ways to do this. In a web application those resources are defined by a URI and method and we use the http filter to associate the resources to authorities. We could also make the association declaratively using the @Secured annotation on a method. Below is an example of the http filter defined in the security context. The granularity of our GrantedAuthority implementation model allows us to associate permissions with those methods and package them up into Roles.
So far we've defined a model and service and made the configuration for the basic 'vanilla' Spring Security implementation. Now it's a case of adding to this to implement the cookie and make the changes required to access the end points from a client other than a browser.
We can reuse the same Authentication filter which would be used in a web site application, the UsernamePasswordAuthenticationFilter class. However, we will need to override the default username and password parameter names with whatever we want those parameter names to be. We could use the default but I don't like it as it gives away the fact we are using Spring and Java.
Utilizing the cookie is a simple case of defining a RememberMe service and plugging it into the Authentication Filter. The work is done inside the AbstractRememberMeServices, there are options here when choosing which concrete type to use. The main decision is whether to persist the cookie data or not. If so, then how to accomplish that. I favour persistence and I like to use JPA. This means defining our own token entity, dao and service. This also gives us the opportunity to take of a problem which arises from the fact we are defining our services as remote interfaces.
The option to persist the token data is facilitated in the framework with the PersistentTokenBasedRememberMeServices class. This stores a POJO, PersistentRememberMeToken, via the PersistentTokenRepository. Again there are options here on implementation of the repository, either in memory or JDBC to a defined table. The problem is that the PersistentRememberMeToken is not serializable and therefore cannot be passed across a remote interface. We can start customizing the RememberMe functionality by defining our own POJO, an entity with an association to the User model entity we defined earlier.
We can create a DAO for that model object which defines CRUD operations which could be used when managing instances outside of the relationship with the user.
We now create own own RememberMe service by sub-typing the AbstractRememberMeServices class. The two methods we are forced to implement deal with creating and reading the token model object through the repository: onLoginSuccess and processAutoLoginCookie deal with the initial authentication and cookie generation, and the subsequent authentication with the cookie accordingly.
The AbstractRememberMeServices class deals with setting the generated series and token values on the response for a successful initial authenitcation via the addCookie and setCookie functions. Ultimately the generated token ends up in a value in the Set-Cookie header with a name defined by the RememberMe service class. By default his is SPRING_SECURITY_REMEMBER_ME_COOKIE. The cookie looks a little like this -
Set-Cookie: SPRING_SECURITY_REMEMBER_ME_COOKIE=12345abcde Expires=Wed, 13 Jan 2015 22:23:01 GMT; Secure; HttpOnly
There are a couple of loose ends to tie up before we can build and deploy. For a web based application we need to specify how Spring Security should handle a request to a secured resource before authentication has taken place. The entry-point-ref attribute in the http filter specifies reference to a bean which handles this request. In our web service we don't really care about redirecting to a login page so we just send a HTTP 401 to tell the client to authenticate.
Similarly, in the authentication processing filter declaration we specify a success handler which redirect back to the originally requested resource after a successful authenitcation. Again, in our web application we don't need to bother so we just send a HTTP 200 on every successful authentication request.
Here's the success handler, it's pretty simple, I won't bother showing the entry point, that's just as simple.
And so that's it. We should now be able to run a client, such as SoapUI and call the authentication end point with the username and password credentials as parameters, see the status 200 OK returned along with the Set-Cookie value in the response header. For subsequent requests to the end point URLs secured in our http filter declaration, we will need to use the Set-Cookie header and use the token value received in the response to the initial authentication on the request. In SoapUI you can write a nice Groovy script to take the cookie value and set it as a global property.
Friday, 20 March 2015
Wednesday, 18 February 2015
Mapping Enumerated Column Values with Hibernate Annotations
If you use JPA to map table data to your entity classes using annotations then you may have come across difficulty when you want to map a Java Enum to a column. JPA offers two ways to map an enum using the EnumType enum. You have choice of mapping your enum as either ORDINAL or STRING.
With an ORDINAL type, the value persisted in the column will the ordinal integer of the enumerated value. With the STRING type, the value of enum.name() will be persisted. This can be somewhat limiting to a developer faced with an enum type which holds other declared fields. You may be working with a legacy database and the values in the column don't reflect the ordinal values or names of the enumerated type you want to use to constrain the data. In this case the JPA EnumType enum is no good to you and you may well end up having to use an unconstrained String or Integer value in the entity class. However, if your underlying JPA implementation is Hibernate then you can make use of the UserType and ParameterizedType interface in the Hibernate core package. Using these will tie your system to Hibernate, not directly as I will show, but you will need the Hibernate core as a dependency in your build.
Let's start by looking at an example situation with a database table with a column you want to map to an enum. My example uses an enum which reflects a country, this might not be something you'd use an enum for in real life, you'd probably want to read country information from a table of its own as it is something which can change. The column contains the actual country name as a VARCHAR but I want to constrain that in my code with the enum. The enum uses the two character code defined in ISO 3166-1. The enum looks like this:
The fields are defined as the country name, ISO 3166-1 3 char code, the numeric ISO 3166-1 code and a boolean flag used for something else. The database column I'm going to map this enum to hold the country name, e.g. Afghanistan, and I want to ensure I tightly constrain that to this enum when annotating the value in my entity class.
I'll start by implementing the ParameterizedType and UserType Hibernate interfaces in an abstract utility class which will allow me to specify the enum field in a concrete sub type. Here's what that class looks like:
The important function here is the setParameterValues method. this is called when the annotation we will implement is loaded and sets the type of Enum which will be instantiated when JPA maps column data. In our Country example the type will be the Country enum class itself. This abstract class is generic and can be implemented for each type of Enum class we wish to map. We now create a sub type for our Country enum which looks like this:
In my concrete sub type I state the Oracle data type that the mapper should expect, in this case our country names column is a VARCHAR. I then implement the two abstract methods which perform the get and set of the enum object based on the values in the column or the value of the enum. My Country enum has a utlity method which returns the enum for a given country name and the nullSafeGet function utilizes this. In the nullSafeSet function I simply pass the country name from the enum into the prepared statement.
Now we can use this class in our annotation in the entity like so:
And that's it. My DAO object which reads/writes the entity instances will do the rest and populate the country field with a valid Country enum. If it can't find an enum with a country name field which matches it'll just set a null. You might want to add some error handling around that depending on your use case. The getCountryByName function could throw an exception if the given name doesn't match any enum country name values. You'd then just catch that in the nullSafeGet implementation and throw one of the exceptions on that method signature.
With an ORDINAL type, the value persisted in the column will the ordinal integer of the enumerated value. With the STRING type, the value of enum.name() will be persisted. This can be somewhat limiting to a developer faced with an enum type which holds other declared fields. You may be working with a legacy database and the values in the column don't reflect the ordinal values or names of the enumerated type you want to use to constrain the data. In this case the JPA EnumType enum is no good to you and you may well end up having to use an unconstrained String or Integer value in the entity class. However, if your underlying JPA implementation is Hibernate then you can make use of the UserType and ParameterizedType interface in the Hibernate core package. Using these will tie your system to Hibernate, not directly as I will show, but you will need the Hibernate core as a dependency in your build.
Let's start by looking at an example situation with a database table with a column you want to map to an enum. My example uses an enum which reflects a country, this might not be something you'd use an enum for in real life, you'd probably want to read country information from a table of its own as it is something which can change. The column contains the actual country name as a VARCHAR but I want to constrain that in my code with the enum. The enum uses the two character code defined in ISO 3166-1. The enum looks like this:
The fields are defined as the country name, ISO 3166-1 3 char code, the numeric ISO 3166-1 code and a boolean flag used for something else. The database column I'm going to map this enum to hold the country name, e.g. Afghanistan, and I want to ensure I tightly constrain that to this enum when annotating the value in my entity class.
I'll start by implementing the ParameterizedType and UserType Hibernate interfaces in an abstract utility class which will allow me to specify the enum field in a concrete sub type. Here's what that class looks like:
The important function here is the setParameterValues method. this is called when the annotation we will implement is loaded and sets the type of Enum which will be instantiated when JPA maps column data. In our Country example the type will be the Country enum class itself. This abstract class is generic and can be implemented for each type of Enum class we wish to map. We now create a sub type for our Country enum which looks like this:
In my concrete sub type I state the Oracle data type that the mapper should expect, in this case our country names column is a VARCHAR. I then implement the two abstract methods which perform the get and set of the enum object based on the values in the column or the value of the enum. My Country enum has a utlity method which returns the enum for a given country name and the nullSafeGet function utilizes this. In the nullSafeSet function I simply pass the country name from the enum into the prepared statement.
Now we can use this class in our annotation in the entity like so:
And that's it. My DAO object which reads/writes the entity instances will do the rest and populate the country field with a valid Country enum. If it can't find an enum with a country name field which matches it'll just set a null. You might want to add some error handling around that depending on your use case. The getCountryByName function could throw an exception if the given name doesn't match any enum country name values. You'd then just catch that in the nullSafeGet implementation and throw one of the exceptions on that method signature.
Working with Collections
The collections framework is a core part of the Java platform yet often misunderstood or misused by developers. The framework contains plenty of functionality for working with and manipulating collections of objects. Often though, you may want to perform an operation on a collection of objects which isn't naively supported by the framework. In such cases you end up having to code the solution into your function or hand roll your own utility to perform the generic operation.
The Jakarta Commons Collections project is a dependency found in many projects. If your using a major framework in your project, such as Spring or Struts then you'll find this dependency in your build. Commons Collections is a very under used framework considering the wealth of functionality it provides and something I think developers should make more use of instead of coding their own solutions to common challenges.
In this post I will demonstrate the use of a predicate to filter a collection of objects. Something which would be easy to code by using elements of the core Java framework but even easier to accomplish in one line with Commons Collections.
Let's start with a requirement: I have a collection of Fruit instances, the Fruit contains a field which defines the type of fruit it is and I want to filter that collection based on type. Here's my simple Fruit class: I'm using the commons-collections4 framework which makes use of generics. Here's my build dependency: Now we create a Predicate which will be used to evaluate the instances of Fruit in our collection and return a boolean based on the function we implement. My function is simply going to check the type of the given Fruit instance is equal to that of the predefined type I set when instantiating the Predicate implementation. I'm not going to be strict, I'm just going to check equality without regarding case sensitivity. Note that I'm returning false if the type value is null, even though the only constructor in the Fruit class enforces that value is set it is best to be safe and I don't want any null Fruit type instances in my results. Now I can use my FruitPredicate in my logic and perform operations on collections of Fruit instances. First, we'll look at a basic function to filter the collection for a specific type of Fruit: We can also use the same Predicate to select the inverse: The collection returned from our select functions, which filtered for specific types of fruit, is still modifiable and we can add another Fruit of any type after the filtering has occurred. Another useful utility allows us to predicate the result preventing other types of Fruit being added after the filtering has occurred. There's a whole host of functions which you can play around with and undoubtedly find use for in your code.
The Jakarta Commons Collections project is a dependency found in many projects. If your using a major framework in your project, such as Spring or Struts then you'll find this dependency in your build. Commons Collections is a very under used framework considering the wealth of functionality it provides and something I think developers should make more use of instead of coding their own solutions to common challenges.
In this post I will demonstrate the use of a predicate to filter a collection of objects. Something which would be easy to code by using elements of the core Java framework but even easier to accomplish in one line with Commons Collections.
Let's start with a requirement: I have a collection of Fruit instances, the Fruit contains a field which defines the type of fruit it is and I want to filter that collection based on type. Here's my simple Fruit class: I'm using the commons-collections4 framework which makes use of generics. Here's my build dependency: Now we create a Predicate which will be used to evaluate the instances of Fruit in our collection and return a boolean based on the function we implement. My function is simply going to check the type of the given Fruit instance is equal to that of the predefined type I set when instantiating the Predicate implementation. I'm not going to be strict, I'm just going to check equality without regarding case sensitivity. Note that I'm returning false if the type value is null, even though the only constructor in the Fruit class enforces that value is set it is best to be safe and I don't want any null Fruit type instances in my results. Now I can use my FruitPredicate in my logic and perform operations on collections of Fruit instances. First, we'll look at a basic function to filter the collection for a specific type of Fruit: We can also use the same Predicate to select the inverse: The collection returned from our select functions, which filtered for specific types of fruit, is still modifiable and we can add another Fruit of any type after the filtering has occurred. Another useful utility allows us to predicate the result preventing other types of Fruit being added after the filtering has occurred. There's a whole host of functions which you can play around with and undoubtedly find use for in your code.
Friday, 6 February 2015
SolrCloud With Zookeeper Ensemble on the AWS
1.0 Background
Solr is a faceted text based search
service backed by Apache Lucene. Data is indexed allowing your UI to
display results for a search query very quickly. You may already be running solr as a standalone application on a single instance which is also
host to the Lucence indexes. The nature of the indexes means that the
application is difficult to cluster and your Solr service is vulnerable.
If the service was lost then recovery would take time to re-index rendering your application unusable for that period.
The latest stable version 4.10.6 includes SolrCloud which allows the
Lucence index data to be shared across multiple Solr instances fronted by a
load balancer. This provides both resilience and performance enhancements to
the search service. Should one instance of the service be lost the service as a
whole will continue to function. The service can be scaled to meet demand when
required.
This post outlines the complete installation and
architecture of SolrCloud running in the AWS.
2.0 Install & Set Up
2.1 Build the Basic Application Server Image
The first step is to build a server image
which we will use as a template for launching our SolrCloud instances as nodes
in the cluster. As we will see in 2.2
our SolrCloud instances will be launched into a private subnet within a VPC.
However, as a start, we will launch an EC2 classic instance and use this as a
temporary machine on which to build the configuration and create an image as a
template. So, take the first step an launch an image with appropriate security
into EC2 from a basic Amazon Linux AMI.
Now we have a running instance we can
configure the server, install tomcat, Solr and Zookeeper and create an image
which we will use as a template the nodes in our cluster. Everyone has their own flavour of where and
how to install these things, you may also visit the AMI market place and launch
an instance from a pre-configured image.
2.1.1 JDK 1.7
- Download the JDK, I used jdk-7u71-linux-x64.gz
- Unzip into /opt/
- Add a java.sh file into /etc/profile.d/ which sets the JAVA_HOME environment variable and adds the bin/ directory to the path
- Reboot and check the new SDK bin directory is set on the path and the JAVA_HOME environment variable correctly set.
- Download tomcat 7, I used apache-tomcat-7.0.57.tar.gz
- Unzip into /usr/share/tomcat7
- Set the UTF-8 attribute in the Connector element in server.xml
<Connector port="8080" protocol="HTTP/1.1"
connectionTimeout="20000"
redirectPort="8443" URIEncoding="UTF-8"/>
- Set the management-gui role, user and password in conf/tomcat-users.xml
<role rolename="manager-gui"/>
<user username="username" password="password" roles="manager-gui"/>
- install apr-devel & http-devel using the package manager
yum –y apr-devel httpd-devel
- Add a tomcat user and set the home directory, remember to chown that to the new user.
useradd –d /user/share/tomcat7 –s/bin/bash tomcat
chown –R tomcat.tomcat /usr/share/tomcat7
- Build the Commons Daemon JSVC script to run the tomcat process. This will allow us to run tomcat on boot up and is the standard way to run tomcat as a daemon process. Follow this guide but ensure you configure the server with the following additional steps when running configure.
- When running configure use the installed JDK and APR and copy the compiled APR libraries into the runtime environment.
./configure –with-apr=/usr –with-java-home=/opt/jdk1.7.0_71
cp /usr/local/apr/lib/libtcnative-1.s* /opt/jdk1.7.0_71/jre/lib/amd64/server
- Copy the daemon.sh file, which runs the jsvc script you have built and copied into the bin/ directory, into /etc/init.d/tomcat7 and set the run level using chkconfig so it boots up on start up.
# chkconfig: - 20 80
# description: starts tomcat - /usr/share/tomcat7 under user tomcat
- Now set the environment variables for tomcat, there are various options for doing this but I just add another file under /etc/profile.d containing the following.
export CATALINA_HOME=/usr/share/tomcat7
export JSVC_OPTS="-Xms2048m -Xmx2048m"
You will now have a Tomcat server platform on which to add the SolrCloud application. You might want to create an Amazon Machine Image of the instance you have configured which you can use as a base template for a generic tomcat server.
2.1.3 Deploy Solr 4.10.3
Solr is simply a web application which runs in tomcat. The Solr distribution comes with a pre-built .war file which you can deploy into tomcat in the usual way. I always create a deployer context file and set any environment variable, such as the Solr home directory, in that context. This also allows us to deploy different versions of the application into the same location within tomcat when we want to upgrade.
Solr is simply a web application which runs in tomcat. The Solr distribution comes with a pre-built .war file which you can deploy into tomcat in the usual way. I always create a deployer context file and set any environment variable, such as the Solr home directory, in that context. This also allows us to deploy different versions of the application into the same location within tomcat when we want to upgrade.
- Download the Solr package, I used Solr-4.10.3.tgz
- Unpack the file into your home directory, we don’t need to upack into any particular location as we are going to cherry pick files within the distribution to deploy into tomcat. Find the solr.war file in:
solr-4.10.3/example/webapps/solr.war
- Create a Solr Home directory somewhere outside of the tomcat application server. This will host the cores, including indexes and other core related configuration. I created my Solr Home in:
/var/lib/solr/
- Ensure that the tomcat user has access to solr home
chown –R tomcat.tomcat /var/lib/solr
- Now we can create the core. To do this I simply copied the example core folder from the solr distribution into my solr home directory. This includes the example schemas, configuration and indexes but we can overwrite and get rid of them later.
cp –r solr-4.10.3/example/solr /var/lib/solr/
- Create a deployer context for the solr.war file place the following file in:
/usr/share/tomcat7/conf/Catalina/localhost/solr.xml
<?xml version="1.0" encoding="UTF-8"?>
<Context path="/solr/home"
docBase="/usr/share/tomcat7/deploy/solr.war">
<Environment name="solr/home" override="true" type="java.lang.String" value="/var/lib/solr" />
</Context>
As you can see, I create a deploy directory within tomcat home where I place my web application resource files.
- Now edit the solr.xml configuration in /var/lib/solr/solr.xml. This is the main solr configuration file and defines our cloud formation. We will revisit this file a number of times. For now we only have one machine running so we will set up just for localhost for now. The solr.xml I have is pretty much the same as the example with the addition of the zkhost configuration element, here’s what the solrcloud stanza looks like in my solr.xml:
<solrcloud>
<str name="host">${host:}</str>
<int name="hostPort">8080</int>
<str name="hostContext">${hostContext:solr}</str>
<int name="zkClientTimeout">${zkClientTimeout:30000}</int>
<bool name="genericCoreNodeNames">${genericCoreNodeNames:true}</bool>
<str name="zkHost">127.0.0.1:2181</str>
</solrcloud>
- Now you can start tomcat which will deploy the solr.war web application into the webapps/ directory. You might want to check the catalina.out log and ensure the application has deployed, if not then add a logging file as specified below and up the log level to DEBUG. Once successfully deployed and running you can shut tomcat down again.
- Use the log4j.properties included in the resources/ directory in the Solr example. Copy the file into the WEB-INF/classes/ directory of the deployed solr web app within tomcat. You may want to change the logging properties and specify a solr.log file somewhere appropriate, e.g. /var/log/solr/solr.log
- In the core within the solr home directory you’ll find a core.properties file which contains a single entry specifying the name of the collection. This should reflect the name of your collection, the same name as the directory in which the core configuration and indexes are kept. In our case this is collection1 and the value in core.properties should reflect this.
- Restart tomcat, confirm the Solr application has booted up cleanly. You can now open up a browser and login to your solr admin page and view the core.
 |
| Single instance of Solr |
2.1.4 Deploy Zookeeper 3.4.6
Now we have the application running it is time to turn our attention to the underlying data itself and how this is will be kept in sync across the SolrCloud cluster. SolrCloud uses Zookeeper to keep the config and index data synchronised across the instances. In order to ensure we don’t have a single point of failure within the system we will cluster Zookeeper as well, this is known as the ensemble. In fact, the best option is to run Zookeeper on the each instance also running Solr within tomcat. This way we can have only single image which we can use as a template for all instances within the cluster. Zookeeper needs a majority number of instances to be up in order to maintain the service; therefore an odd number of instances should be deployed. For this reason we will have 3 instances, each running Solr and Zookeeper in the cluster.
- Download Zookeeper zookeeper-3.4.6.tar.gz unzip into /usr/share/ zookeeper-3.4.6
- The main zookeeper configuration file is in the conf/ directory called zoo.cfg. There are various configurations here which we can tweak later, for now we will keep the defaults but we need to specify the zookeeper data directory. You should add this directory outside of the zookeeper application home, my dataDir configuration looks like this
dataDir=/var/lib/zookeeper/
I then create this directory
mkdir /var/lib/zookeeper
- Whilst we are editing zoo.cfg note the client port value. This is the port Solr, the client, will use to communicate to zookeeper. By default this is set to port 2181 and we will need to ensure we specify this in our AWS security group later.
- We need to ensure each zookeeper instance in the ensemble knows how to communicate with the other. Although we are currently installing one instance and no nothing about the other instances we will run in our cluster we can prepare the configuration. Again we do this in zoo.cfg and add the following lines:
# zookeeper ensemble
server.1=localhost:2191:3888
#server.2=x.x.x.x:2191:3888
#server.3=x.x.x.x:2191:3888
I have left the configuration for server 2 & 3 commented out, we will need to specify these later on when we bring up our instances within the AWS VPC environment.
Again, note the ports. 2191 and 3888 are ports used within the zookeeper ensemble are different to the client port value. We will need to ensure these are specified in the AWS security group later on.
- Each zookeeper instance in the ensemble needs to know which server it is. In the configuration we specified in zoo.cfg we have said that this instance, localhost, is server id 1. Zookeeper will look for a file called myid in the data directory. Within that file is a number which details which server this is. I created the file and edited it, adding a single value: 1 we will do this later on our other instances with values 2 & 3.
touch /var/lib/zookeeper/myid
- Whilst in the conf/ directory edit the log4j.properties file or create one if not already present. I set a rolling log in /var/log/zookeeper/zookeeper.out
- We now need to make a zookeeper start script allowing the server to be run as a daemon process. I hacked the zkServer.sh script in the bin/ directory of the distribution and set the run levels as we did with tomcat and placed it in /etc/init.d.
# chkconfig: - 20 80
# description: starts zookeeper - /usr/share/zookeeper-3.4.6 under user root
I then added the script and turned it on using the chkconfig command.
chkconfig –add zookeeper
chkconfig zookeeper on
- This is a good point to test the zookeeper installation. We can boot up the server using the new start script in /etc/init.d/zookeeper or just run the zkServer.sh script in bin/ using the start command.
bin/zkServer.sh start
- You should now be able to connect to the zookeeper server using the zkcli.sh script in the bin directory like so:
bin/zkCli.sh
This will connect and allow you to run zookeeper commands.
- Now the server is running exit from the zkCli command line and we can upload the Solr configuration and core so that Zookeeper can do its thing and synchronise the indexes across multiple instances. We are still running just one instance so we’ll have to revisit this when we come to run it across multiple instances in the VPC. We do this step here just to allow us to boot up solr and ensure it runs cleanly with Zookeeper. We need to execute 3 commands using the zkCli.sh tool which comes with the Solr distribution we downloaded.
/home/ec2-user/solr-4.10.3/example/scripts/cloud-scripts/zkcli.sh
We will use this tool to upload the solr configuration files found in solr home, link the uploaded collection to the configuration and bootstrap zookeeper. Execute the following commands:
/home/ec2-user/solr-4.10.3/example/scripts/cloud-scripts/zkcli.sh -zkhost 127.0.0.1:2181 -cmd upconfig -d /var/lib/solr/collection1/conf/ -n collection1_conf
/home/ec2-user/solr-4.10.3/example/scripts/cloud-scripts/zkcli.sh -zkhost 127.0.0.1:2181 -cmd linkconfig -collection collection1 -confname collection1_conf -solrhome /var/lib/solr/
/home/ec2-user/solr-4.10.3/example/scripts/cloud-scripts/zkcli.sh -zkhost 127.0.0.1:2181 -cmd bootstrap -zkhost 127.0.0.1:2181 -solrhome /var/lib/solr/
- We can now restart tomcat and run Solr with as a single node cluster to test everything is correct. Start the server with the tomcat start script and check the solr and zookeeper logs to ensure everything is configured correctly. Once you’re happy shut it down again.
We can now concern ourselves with the actual data we are going to index, the schema and configuration of the data we are serving up. If you are migrating from an old single solr instance you will probably want to copy these files across into the conf/ directory of the core in solr home. Depending upon which version of Solr you are migrating from there will need to be some changes to these files. If you are using the default example core, named collection1 then you won’t need to do this but I am assuming you will want to progress to using a schema of your own at some point.
- Copy the following files into the conf/ directory of the core within the solr home.
schema.xml
synonyms.xml
solrconfig.xml
stopwords.xml
spellings.xml
synonyms.xml
solrconfig.xml
stopwords.xml
spellings.xml
- If you are upgrading from an older version of Solr prior to version 4 then you will need to add the following line to the schema.xml.
<field name="_version_" type="long" indexed="true" stored="true"/>
- If you wish to test Solr and view the details in the Solr admin page then you will need to reload the data into zookeeper. Use the zkcli.sh tool to issue the upconfig, linkconfig and bootstrap commands outlined in the last step of the previous section. You can then reboot tomcat and browse the Solr admin page on your running instance.
You should now image your instance. We will use this as a
template from which we will launch instances within the VPC and create the
cluster. Once you have created your AMI you can terminate the instance on which
you created the template.
2.2 Build the AWS VPC Environment
- Create a new subnet and associate to the new VPC. Give the subnet a CIDR block as a sub set of the VPC, I gave it 10.0.0.0/24 giving me 256 possible private IP addresses.
- Create a Routing Table, name it SolrCloud-RT or something like that and associate with your VPC.
- Now create a security group which will be associated to each instance we launch into our VPC. From our configuration in the previous sections we know which ports are needed for communication between the instances in the cluster and machines outside the VPC. We said that we would use EC2 Classic link from one EC2 Classic instance to the VPC this will facilitate SSH access to our instances. When you create a security group it will give you a predefined id, prefixed with the letters ‘sg’ when you specify a security group id in the inbound rules it means any instance associated with that group has access on that port to instances associated with the security group you are creating. This allows us to use a self-referencing set of rules for intercommunication between our instances without prior knowledge of instance IP addresses.
Here’s my inbound rules, note the two security groups are this group, for self-reference, and the security group of my EC2 Classic instance.
|
Protocol
|
Port
|
Source
|
|
SSH
|
22
|
sg-ec2classicSG
|
|
HTTP
|
80
|
sg-ec2classicSG
|
|
Custom TCP
|
8080
|
sg-this
|
|
Custom TCP
|
2181
|
sg-this
|
|
Custom TCP
|
2191
|
sg-this
|
|
Custom TCP
|
3888
|
sg-this
|
- Create a load balancer. This will be used to marshal the HTTP requests to Solr from outside the VPC. When we launch instances we will associate them to this load balancer. Create the load balancer inside the SolrCloud VPC and configure the listener to forward HTTP requests on port 80 to HTTP port 8080 on the instances, this is the port our tomcat instance is running on.
The load balancer needs a mechanism to ensure the instances are up and running. It does with a Health Check ping. We want to ensure the Solr application is running so we choose a URL within that application. I simply set the path to the favicon.ico file in the Solr application root with the default timeouts and thresholds.
HTTP 8080 /solr/favicon.ico
Select the Solr subnet we set up previously. We could have the option to create multiple subnets in different availability zones within our VPC to give us high availability. This would be a good idea but for now let’s keep it simple and stick with one subnet.
Select the security group we created in the previous step, we will add our instances later so go ahead and create the load balancer.
- Create the EC2 Classic link between the EC2 Classic instance and the VPN. As previously stated, we don’t expose Solr to the outside world. Access to Solr will be performed from another application which will make HTTP requests to the Solr load balancer within the VPC. My application resides on an EC2 Classic instance and it’s a simple case of selecting the instance and linking to the VPC.
2.3 Launch The Instances and Create the Cloud
Now it’s time to create and launch the cluster. We will launch 3 instances from the AMI we made in section 2.1. We will need to amend some of the Zookeeper configurations we intentionally left commented out in section 2.1.4, the Solr.xml configuration which defines the zkhosts and also upload and link the Solr configurations into Zookeeper. We need to do this on each instance we launch and the configurations are subtly different on each, getting these right is important to ensure the cluster works as intended.
- In the EC2 Dashboard select the AMI we made earlier and hit launch. The type of instance is up to you, I used t2.small which gives us ample memory and power to run Solr and Zookeeper. Select the number of instances to 3 and ensure we select the VPC and subnet we created in the previous section. I give it a volume of 8 Gig but again this will depend on the size of the index you intend to run in Solr. Give the instances a name, we can tweak this later and add numbers to each individual instance later. Select the security group we created previously and launch the instance with your key. Note, you will need to put this key on the EC2 classic instance from where you will hop your ssh session from.
When each instance is up and running note the private IP addresses, write them down somewhere noting which is which server using a pencil and piece of paper!
- SSH into your EC2 Classic link the open a session to the first VPC instance. Check tomcat and zookeepr running by running the following commands, remember they should autostart on boot.
ps –ef | grep tomcat
ps –ef | grep zookeeper
Shut down tomcat:
/etc/init.d/tomcat7 stop
- Edit the hosts file in /etc/hosts change the local IP address to that of the machines private IP. You will now need to reload the network interface or reboot the machine. If you reboot you’ll need to SSH back in from your EC2 instance.
/etc/init.d/network restart or shutdown –r now
- Configure solr, open up the file in /var/lib/solr/solr.xml and add your three private IP addresses to the solrcloud configuration stanza.
<str name="zkHost">x.x.x.x:2181,x.x.x.x:2181,x.x.x.x:2181</str>
- Configure zookeeper. Open up the file in /usr/share/zookeeper-3.4.6/conf/zoo.cfg and uncomment the lines for server 2 & 3 in the zookeeper ensemble section. Add the ip addresses in for server 1, 2 & 3 paying particular attention to this and ensure you know which server is which! My configuration looks like this:
# zookeeper ensemble
server.1=localhost:2191:3888
server.2=x.x.x.x:2191:3888
server.3=x.x.x.x:2191:3888
Remember, localhost will be different on each machine you do this so don’t just copy it from one instance to the other!
- Use the zkcli.sh script to upload the Solr configuration to Zookeeper by issuing the same commands we did back in 2.1.4
/home/ec2-user/solr-4.10.3/example/scripts/cloud-scripts/zkcli.sh -zkhost 127.0.0.1:2181 -cmd upconfig -d /var/lib/solr/collection1/conf/ -n collection1_conf
/home/ec2-user/solr-4.10.3/example/scripts/cloud-scripts/zkcli.sh -zkhost 127.0.0.1:2181 -cmd linkconfig -collection collection1 -confname collection1_conf -solrhome /var/lib/solr/
/home/ec2-user/solr-4.10.3/example/scripts/cloud-scripts/zkcli.sh -zkhost 127.0.0.1:2181 -cmd bootstrap -zkhost 127.0.0.1:2181 -solrhome /var/lib/solr/
- Restart tomcat and the instance is now running and ready. Repeat all these steps in section 2.3 on each instance launched into your VPC. Tail the solr and zookeeper logs on each instance as you bring them up and watch the cluster work J
The cluster will now look something like the image below.
Using the configuration we implemented in section 2.2 means
you won’t be able to connect to the Solr admin page via the load balancer. However,
feel free to open up the security group to the outside but remember; this means
your Solr implementation is open to the world. If you do, I suggest confining
it to your own IP address. You could also just launch your instances into EC2
Classic rather than a VPC and set them behind a public load balancer, again
remember to detail your own IP address in your security group rather than the
entire Internet. Browse to the admin
page using the public domain of the load balancer and you’ll see something like
this.
 |
| SolrCloud running 3 instances |
Now our SolrCloud, Zookeeper Ensemble cloud is up and running we should think about adding some monitoring. We don’t autoscale the instances behind our load balancer simply because we need to configure each instance in the cluster using the private IP addresses of each instance. We don’t know them until we launch the instance and although there may be some good potential of future work to deal with this; we can’t set that configuration automatically when an instance comes up. It’s easy enough to just set some monitors on our cluster so should we lose an instance we can be notified and manually bring another up without any loss of service.
Without going into great detail about AWS monitors all I will say is configure some alarms on the VPC load balancer to send an yourself an email should the number of instance drop below 3. There are many other metrics to which you could attach alarms and it’s up to you how much you want to do.
3.0 Future work
I’m sure after going through this you’ll see potential
to change and improve some things. Here’s a few things I would like to do to
improve performance and resilience of the service.
We only created one subnet within our VPC. This means all the instances we launch are in the same availability zone. Should anything happen to that data centre then we will lose our Solr service. The idea of SolrCloud is to provide greater resilience and get away from single point failure. To that end it might be prudent to create three private subnets in each availability zone and launch one instance in each. We could then scale the cluster up to two or three instance in each zone giving improved performance and resilience.
3.2 Auto Setting IPs when launching instances
As I mentioned before our cluster is fairly static and we cannot auto scale the instances behind the load balancer. Although that’s not a great issue it would be really nice if the entire system was completely passive and scaled up and down automatically. I’m not 100% certain on how to use subnet CIDRs but it could be possible to either define all the possible IP addresses in our configuration and create one generic image which could then be used in the autoscaling script, or get the auto launched instance to receive a predefined private IP specified in the launch configuration.
Subscribe to:
Posts (Atom)