Elasticsearch EC2 Autoscaled Implementation

Elasticsearch has now superseded Solr as the defacto search server. Both systems have Lucene at the heart, Elastic has various benefits and features above Solr but this post will focus on implementing a scaleable search service cluster on an AWS. In a previous post I outlined a similar solution for a SolrCloud cluster. Although this is robust it isn't truly scaleable because each instance needs to be preconfigured to have reference to the other. It may be possible to get around this, inject configuration at runtime and autoscale the cluster. However, Elasticsearch makes this very easy to do directly out of the box and create a stateless, highly scalable search service cluster in a fraction of the time.

Amazon offers two search systems as Platform as a Service: CloudSearch and Elasticsearch Service. Unsurprisingly, these are essentially SolrCloud and Elasticsearch implemented on EC2 behind the scenes and offered as a service. Of course there are advantages to using these but if you want complete control over your search cluster implementation and functionality then you need to build your own from the ground up. The good news is that this is very simple but there are some stumbling blocks which are not obvious in any documentation I've come across.

Setting up the Environment

Before getting into the Elasticsearch configuration we need to focus on the environment. You will probably already have your own VPC and network configuration for Dev, Test and Production. The cluster is a simple set up; Elasticsearch instances running in each availability zone behind a load balancer serving HTTP requests. Each instance needs to know about the other in the cluster and this is done by automatic discovery. Each instance must be associated to the same security group which is used to control access and importantly for filtering discovered instances on the same network. This will be covered later.

Autoscaled cross availability zone Elasticsearch EC2 instances with associated volumes 

Elasticsearch instances will, by default, be capable of automatically discovering each other and forming a dynamic cluster. This is accomplished using network multicast. However, this is disabled in AWS and most other cloud hosts and prevents Elasticsearch from clustering out of the box. To create the cluster we must install the Cloud-aws plugin which utilizes the AWS-API for unicast discovery of other Elasticsearch instances on the network. In order to use this we must provide IAM account credentials in our Elasticsearch configuration file. The account must have permission to use the EC2 service by granting - ec2:DescribeInstances

Installing Elasticsearch from the Package Manager

Once happy with the network setup we can begin building out the temporary instance which will be used to form the baseline image. Many of the tutorials and examples I have come across built out on Ubuntu but I decided to go with the initial Amazon Machine Linux baseline. I started a t2.medium instance and added an additional volume of 25Gig. This will be mounted on a path where we will store the indexed data so size is dependent on what you intend to index and how you want to shard it.

The easiest way to install Elasticsearch is from an rpm which can be downloaded the Elastic.co web site and will need to be added to the package manager
Edit the file and add the Elastic repository
Now install Elastic from the package manager, run it as a service and set the JVM heap size as an environment variable. Add the following line, set the heap size to whatever you wish, within the constraints of your desired EC2 instance type -

Installing The AWS Plugin and Configuring Elasticsearch

Now configure Elasticsearch itself. First, install the cloud-aws plugin which is required to enable auto discovery.
Now edit the main elastic yaml config file found in /etc/elasticsearch/elasticsearch.yml With the exception of logging, all settings are configured in this file. The intention is to avoid any predefined state, such as hostnames or IP addresses, which will require any manual intervention at runtime. There are many properties which could be configured here, however the configuration outlined below is enough to get a basic cluster up and running on EC2.
Now start the Elasticsearch service to ensure it boots up and runs on the configuration we have made. By default the logs are set to level INFO, if you edit the logging.yml file prior to starting and set the default level to DEBUG you'll be able to get some more meaningful information out should you come across any problems. When running the server will log to /var/log/elasticsearch/elastic-cluster.log
You should now be able to make client HTTP requests to the running service either from your local browser or the command line and get information about the cluster and nodes. This will report the state of the cluster with a JSON response and should look similar to this

Create the Autoscaled Cluster

Assuming the service starts without a problem you will now have a single instance from where you can create the image which will be used in the launch configuration. Simply image the running instance and name the AMI appropriately.

Create an Elastic Load Balancer, set the ping check to root on port 9200. Add a listener to receive on port 80 and forward to 9200 on the instances. Either create anew security group or use the same one associated to the instance you created earlier. If creating a new one ensure the instance security group has reference to it so that traffic from the load balancer can make contact with the instances. Enable cross Availability Zone balancing so we can ensure the cluster is highly available.

Create the Launch Configuration and associated Autoscaling Group. these are straight forward, the Launch Config should simply instantiate the image we created earlier on a suitable instance type. I chose t2.medium which gives me ample memory to designate the 1 gig heap size I defined in my environment variable. associate the instance security group and create the Autoscaling Group which you can play with to scale as you require. I just set it to maintain a cluster of 3, which should ensure we have at least one instance in each Eu-west zone. If you wish to launch your instances with an associated volume on which to store the search index files then you must specify the volume in the launch configuration and pass the mount command as user data to the launching instance. You should also specify the path.data value in the elasticsearch.yml file with reference to the directory where the volume is mounted.

Once the cluster has initialized you should be able to make another HTTP request to the load balancer and confirm the number of running nodes This will report the state of the cluster with a JSON response and should look similar to this