Every Data Engineer who uses Elasticsearch as a documents store, knows that there are many parameters that affect the queries latency, throughput, and eventually the Queries Per Second (AKA — QPS).
In one of our Projects at Explorium, we have an Elasticsearch cluster, hosted in AWS with 14 nodes of m5.4xlarge.elasticsearch.
In the cluster we have around 20M documents which are 10 GB.
This cluster receives tons of queries, rapidly and very frequently, and requires a very high concurrency of queries.
While the usage of the database got bigger and bigger, we noticed two things:
- Some queries were rejected by our Elasticsearch cluster due to a full internal Elasticsearch queue.
- Our QPS was around 100 and we wanted to achieve a much higher rate.
We decided to get deeper into Elasticsearch internals in order to solve these two issues.
In this article I will introduce the changes we have made in order to increase our QPS by 7X and even support higher concurrency.
Read the full article here.
The post How to dramatically increase your Elasticsearch throughput and concurrency capacity appeared first on Explorium.