Recently I’ve been working with Elasticsearch as I look to implement a logging solution using Logstash.  Elasticsearch is a popular, flexible open source real-time search and analytics engine and I’ve been looking at making use of the engine to perform full text searches and analytics against the mountains of log data we have on our platform.

As part of this work I’ve been setting up an Elasticsearch cluster of three Windows Server 2012 R2 Datacenter edition virtual machines in Windows Azure.  We’ve chosen Windows based virtual machines as the team who will support it have more experience and knowledge of Windows OS than Linux.  To get up an running I’ve used the following blog post from Thomas Ardal, which has a further link to info on installing Elasticsearch on the VMS – Running ElasticSearch in a cluster on Azure.

One of the elements of configuring the cluster concerns the discovery of nodes in the cluster.  By default Elasticsearch makes use of multicast to discover the nodes in the cluster, however multicast doesn’t appear to work fully in Azure at present, therefore Thomas’ blog post details making changes to the elasticsearch.yml configuration file to disable multicast and add in the node ip addresses manually. 

However on Friday (31st Jan 2014) Elasticsearch announced an Azure Cloud Plugin for Elasticsearch and the goal for this plugin is to enable automatic unicast discovery of Windows Azure Virtual Machines in the Cloud Service.  The article details how to set the plugin up in Linux Virtual Machines, but as I’ve already mentioned we are using Windows Server 2012 R2 for all of our virtual machines in the cluster.  Therefore I set off today to configure the plugin for running Elasticsearch on Windows.

1. Create Windows Azure VMs as detailed in Thomas’ post, installing Elasticsearch on each Virtual Machine you wish to use as a node.
2. Create a self-signed certificate for working with the Azure REST API using makecert (You will need the Windows SDK for this in order to make use of makecert).
3. Upload the .cer file to the Management Certificates (Settings->Management Certificates) store for your Azure subscription in the Azure Portal. 
4. Using Certificate Manager (certmgr.msc) export the certificate with the public key (in .pfx format) and give the certificate a password of suitable complexity.  Copy the pfx file to a folder on your virtual machines, e.g. C:\MyCertificates
5. Install the Azure-Cloud plugin on each of your Elasticsearch Virtual Machines by running the following command from a command point at your elasticsearch installation directory
   bin/plugin -install elasticsearch/elasticsearch-cloud-azure/1.0.0.alpha1
   
6. Next edit the Elasticsearch configuration file, Elasticsearch.yml, which is found in the config folder of your Elasticsearch installation on each virtual machine in your cluster.  At the bottom add the following configuration and update with the relevant settings:

   ################################# AZURE PLUGIN ###############################
   cloud:
     azure:
      keystore: <path to certificate i.e. c:/MyCertificates/mycert.pfx>
      password: <Password for certificate file>
      subscription_id: <your subscription id>
      service_name: <name of cloud service>
    
   discovery:
      type: azure

   for example:

   ################################# AZURE PLUGIN ###############################
   cloud:
     azure:
      keystore: c:/MyCertificates/mycert.pfx
      password: verystrongpassword
      subscription_id: xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx
      service_name: myescluster
    
   discovery:
      type: azure

   Thanks to David Pilato and Thomas Conté for their help in getting the configuration correctly formatted and populated.

7. Start Elasticsearch on each server

That’s it all configured, if you’re using plugins to monitor your Elasticsearch cluster such as the Head plugin you should now see all of your nodes in your cluster listed, for example:

WARNING: As a side note, it’s VERY easy to break the syntax in yml files when not familiar with them, I’m now using Notepad++ to work with them and I’ve also been using a VS Extension – YamlDotNetEditor - when editing the yml file so I can reduce the risk of making mistakes.  The extension formats and lays out the file in a readable format with syntax highlighting (although this is currently not working in VS2013).