How to dramatically increase your Elasticsearch throughput and concurrency capacity

maya.malevich

Every Data Engineer who uses Elasticsearch as a documents store, knows that there are many parameters that affect the queries latency, throughput, and eventually the Queries Per Second (AKA — QPS).

In one of our Projects at Explorium, we have an Elasticsearch cluster, hosted in AWS with 14 nodes of m5.4xlarge.elasticsearch.

In the cluster we have around 20M documents which are 10 GB.

This cluster receives tons of queries, rapidly and very frequently, and requires a very high concurrency of queries.

While the usage of the database got bigger and bigger, we noticed two things:

  • Some queries were rejected by our Elasticsearch cluster due to a full internal Elasticsearch queue.
  • Our QPS was around 100 and we wanted to achieve a much higher rate.

We decided to get deeper into Elasticsearch internals in order to solve these two issues.

In this article I will introduce the changes we have made in order to increase our QPS by 7X and even support higher concurrency.

Read the full article here.

The post How to dramatically increase your Elasticsearch throughput and concurrency capacity appeared first on Explorium.

Previous Article
As the Economy Recovers, Do You Have the Right Data to Ride the Wave?
As the Economy Recovers, Do You Have the Right Data to Ride the Wave?

As published in CMSWire. Read the full article here. On June 8, 2020, The National Bureau of Economic Resea...

Next Article
Explorium Named a Cool Vendor by Gartner
Explorium Named a Cool Vendor by Gartner

Today, we are proud to share some exciting news.  Explorium's  external data platform that automatically di...