ElasticSearch

Why did you vote for ElasticSearch?

Elasticsearch is a search server based on Lucene and has an advanced distributed model. You can host the opensourced code yourself, on EC2 or use a service such as Bonsai, Found or SearchBlox.

Easy to work with

Elasticsearch gained its popularity amongst developers by being enjoyable to use. A simple feature comparison against it's competition doesn't convey the significant advantages of just how easy it is to work with. This is due to multiple design choices such as the use of JSON for the API and queries.

Structured search queries with JSON

Search can be executed either using a simple, Lucene-based query string or using an extensive JSON-based search query DSL. By structuring the query as a JSON object you can be very explicit and can dictate exactly what ElasticSeach will return. A very basic example of a JSON query is:

curl -XGET 'http://localhost:9200/blog/_search?pretty=true' -d '
{ 
    "query" : { 
        "range" : { 
            "postDate" : { "from" : "2011-12-10", "to" : "2011-12-12" } 
        } 
    } 
}'

RESTful JSON API for configuration/management

Elasticsearch has a REST API for management and configuration. The following are the main features of this API:

Index Management:

  • Create, delete, close and open indices by running a simple HTTP command.
  • Increase and decrease the number of replicas without the need of shutting down nodes or creating new nodes.
  • Manipulate shard placement with the cluster reroute API. Move shards between nodes, we can cancel shard allocation process and we can also force shard allocation – everything on a live cluster.
  • Check index and types existence

Configuration:

  • Majority of configuration files can be modified dynamically.
  • Update Mappings
  • Define, retrieve and manage warning queries
  • Shut down the entire cluster or a specific node
  • Clear caches on the index level

This is all done over JSON, making it a lot more structured then the methods used in Solr.

Schemaless

Elasticsearch makes it easy to get started by not requiring you to define a schema before sending documents to be indexed. Elasticsearch will automatically guess field types for you, which although will not be as accurate as creating the mappings manually, is usually pretty accurate.

Elasticsearch also lets you manually define the mappings (index structure) before creating the index. One cool feature is if you miss a field or add a new field without defining the mapping, Elasticsearch will try to guess the Type for you.

Aggregations

Another area where ElasticSearch shines is its aggregations features. Similarly to facets (now deprecated), aggregations allow calculating and summarizing data of a query as it happens. Aggregations brings the ability to be nested and is broadly categorized as metrics aggregations and bucket aggregations.

"aggregations" : {
	"<aggregation_name>" : {
		"<aggregation_type>" : { 
			<aggregation_body>
		},
		["aggregations" : { [<sub_aggregation>]* } ]
	}
	[,"<aggregation_name_2>" : { ... } ]*
} 

Some unique ElasticSearch features are:

  • Regular expressions to define which terms will be included/excluded in results
  • Combine term results from different fields automatically
  • Use scripts to modify the fields values before the calculation process steps in
  • Awesome range searches
  • Can specify set of ranges and it will return both document counts and aggregated data
  • Modify the field and aggregated data with a script

Designed to be distributed

The one area where Elasticsearch shines is distributed search. It was built from the ground up to be suitable for high-scale 'cloud' applications.

There are many features Elasticsearch has as a result of being designed to be distributed that aren't currently available in Solr, such as:

  • Shards and replicas can to moved to any node in the cluster on demand.
  • With a simple API call you can increase and decrease the number of replicas without the need of shutting down nodes or creating new nodes.
  • Manipulate shard placement with the cluster reroute API on a live cluster.
  • Search across multiple indexes.
  • Change the schema without restarting the server.
  • Automatic shard rebalancing

Elasticsearch also has a module called Gateway, that in the case of the whole cluster crashing or being taken down will enable you to easily restore the latest state of the cluster when it gets back up.

Services such as Bonsai further simplify scaling Elasticsearch by hosting and scaling the search servers for you, making it nearly as easy to get started as CloudSearch or Searchify. Elasticsearch was also specifically designed to run well and be relatively easy to setup on EC2.

Allows multiple types of documents per index

Another useful and unique feature to Elasticsearch is the ability to have multiple types of documents in a single index. You can then facet, query or filter against all document types or a single type.

Handles nested documents

ElasticSearch natively handles a nested document structure.

ElasticSearch will index nested documents as a separate indexes and are stored in such a way that allow quick join operations to access them. Nested documents require a nested query to access so that don't clutter results from standard queries.

Percolator (prospective search)

Essentially a reverse search. The percolator allows you to register queries against an index, and then send percolate requests which include a doc, and getting back the queries that match on that doc out of the set of registered queries. Not possible in Solr out of the box.

Rapid feature development

Another thing to keep in mind when choosing a search solution is the development momentum. ElasticSearch has quickly caught up to the competition and most of the currently missing features are due to be released in upcoming versions.

Some missing features

Elasticsearch is currently missing the following features:

  • Results Grouping / Field Collapsing
  • Autocomplete
  • Spell Checker/Did you mean (Available as a third-party plugin)
  • Decision Tree Faceting
  • Query Elevation
  • Hash-based deduplication.

Prone to 'Split Brain' Situations

The Sematex blog explains a problem with the way Elasticsearch handles its clusters, called the 'Split Brain Situation':

Imagine a situation, where you cluster is divided into half, so half of your nodes don’t see the other half, for example because of the network failure. In such cases Elasticsearch will try to elect a new master in the cluster part that doesn’t have one and this will lead to creation of two independent clusters running at the same time. This can be limited with a small degree of configuration, but it can still happen.

Users have already run into this problem in production and ElasticSearch host Bonsai also have had issues with this problem as recently as March 2012.

Poor documentation

As a relatively new project, the documentation for ElasticSearch still leaves much to be desired. Documentation assumes that the user at least has familiarity with similar document stores, and is largely oriented toward those already familiar with other search solutions, such as Solr. Errors, while often quite simple to resolve, can be difficult to troubleshoot, as they are often insufficiently descriptive and missing from documentation. New users should be sure to check the tutorials section on elasticsearch.org for supplementary information lacking from the guide, such as more detailed installation instructions.

Are We Missing Something?