How to dramatically increase your Elasticsearch throughput and concurrency capacity

June 10, 2021 Ron Flomin, Data and ML engineer AI Education

Every Data Engineer who uses Elasticsearch as a documents store, knows that there are many parameters that affect the queries latency, throughput, and eventually the Queries Per Second (AKA — QPS).

In one of our Projects at Explorium, we have an Elasticsearch cluster, hosted in AWS with 14 nodes of m5.4xlarge.elasticsearch.

In the cluster we have around 20M documents which are 10 GB.

This cluster receives tons of queries, rapidly and very frequently, and requires a very high concurrency of queries.

While the usage of the database got bigger and bigger, we noticed two things:

  • Some queries were rejected by our Elasticsearch cluster due to a full internal Elasticsearch queue.

We decided to get deeper into Elasticsearch internals in order to solve these two issues.

In this article I will introduce the changes we have made in order to increase our QPS by 7X and even support higher concurrency.

Read the full article here.

Subscribe Today! Get the latest updates with our newsletter.
We promise you'll love it.

Follow us

New! Explorium Closes $75M Series C Amid Soaring Demand for External Data Learn More