Capacity Planning for Elasticsearch
Elasticsearch is a scalable distributed system that provides solutions for enterprise search, log aggregation, observability, and security. Elastic solutions are built on a single, flexible technology stack which can be deployed anywhere. For a reliable performance deployment of production Elasticsearch, whether self-hosted or cloud-based, it’s essential to carefully plan the infrastructure and cluster configuration.
This article will guide you on how to estimate and formulate a plan based on usage metrics prior to deploying a production-grade cluster.
Capacity Planning:
- Determine the minimum number of master nodes
- Size the Elasticsearch Service
Determining the Minimum Number of Master Nodes:
The Master node is crucial in a cluster as it manages a variety of cluster-wide activities, such as creating and deleting, and allocating shards. The cluster’s stability relies heavily on the Master node’s health.
It’s recommended to have dedicated master nodes. This is because a master node burdened with other tasks might not function efficiently. The optimal way to prevent overloading the master with additional duties is to designate all master-eligible nodes as dedicated master-eligible nodes. These nodes solely have the master role, enabling them to concentrate on managing the cluster.
While a smaller cluster might not need master-eligible nodes, it’s suggested to use dedicated master-eligible nodes once the cluster exceeds 6 nodes.
The minimum master nodes required for decision-making quorum is calculated using the following formula:
Minimum Master Nodes = (N / 2) + 1N is the total number of “master-eligible” nodes in your cluster (rounded off to the nearest integer)
In an ideal environment, the minimum number of master nodes will be 3 and if not maintained, it can result in a “split-brain” that can lead to an unhealthy cluster and loss of data.
Let us consider the below examples for better understanding:
In Scenario A, you have ten regular nodes (ones that can either hold data and become master), the quorum is 6. Even though if we lose the master node due to network connection, the cluster will elect a new master and will still be healthy.
In Scenario B, you have three dedicated master nodes and a hundred data nodes, the quorum is 2. Even though if we lose the master node due to failure, the cluster will elect a new master and will still be healthy.
In Scenario C, you have two regular nodes, with quorum as 2. If there is a network failure between the nodes, then each node will try to elect itself as the Master and will make the cluster inoperable.
Setting the value to 1 is permissible but it doesn’t guarantee protection against loss of data when the master node goes down.
Note: Avoid repeated changes to the master node setting as it may lead to cluster instability when the service attempts to change the number of dedicated master nodes.
Sizing of the Elasticsearch Service:
Sizing an Elasticsearch service involves making an educated estimate rather than applying a definitive methodology. This estimate takes into account storage, the services to be used, and Elasticsearch itself. It serves as a helpful starting point for determining the size of the domains, testing them with representative workloads, and monitoring their performance.
Key factors to consider during sizing include:
- Use case, such as real-time search, security monitoring, log analytics, etc.
- Growth planning, both short-term and long-term.
Elasticsearch is horizontally scalable, but proper indexing and sharding are crucial from the start. Otherwise, you may need to navigate difficult approval processes to add hardware, leading to underutilization of your infrastructure.
The three main factors to consider when choosing appropriate cluster settings are:
- Calculating storage requirements
- Determining the number of shards
- Choosing instance types and conducting tests
Calculating Storage Requirements:
In Elasticsearch, each document is stored as an index. Document storage can be categorised as follows:
Growing Index: This is a single index that expands over time with regular updates or additions. The data for the Growing Index is stored on disk, and its storage space can be determined based on available resources. Common examples include documents and e-commerce search.
Rollover Index: Data is continuously written to a temporary index with a set indexing period and retention time. The amount of generated data will be calculated based on the data produced during the index’s retention period. For instance, if you generate 100 MiB of logs per hour, that’s 2.4 GiB per day, which amounts to 72 GiB of data for a retention period of 30 days. Log analytics and time-series processing are common examples.
Additional considerations alongside storage space include:
Number of Replicas: A replica is a full copy of an index that consumes an equal amount of disk space. By default, each index in Elasticsearch has a replica count of 1. It is advisable to keep the replica count at 1 to prevent data loss and enhance search performance.
Elasticsearch Overhead: Elasticsearch reserves 5% or 10% for a margin of error and another 15% to remain under the disk watermarks for segment merges, logs, and other internal operations.
Insufficient storage space often leads to cluster instability. Therefore, it’s crucial to cross-check the numbers when selecting instance types, instance counts, and storage volumes.
Selecting the Number of Shards:
Another important consideration is the indexing strategy for your indices. In Elasticsearch (ES), every index is divided into a certain number of primary and replica shards by default. For instance, if there are 2 primary and 1 replica shard, then the total number of shards is 4. You cannot change the primary shard count of an existing index once it’s created.
Each shard uses a portion of CPU and memory. Having too many small shards can lead to performance issues and out-of-memory errors. However, it’s also not advisable to create overly large shards.
A good rule of thumb is to keep the shard size between 10–50 GiB.
The formula for calculating the approximate number of shards is as follows:
App. Number of Primary Shards =
(Source Data + Room to Grow) * (1 + Indexing Overhead) / Desired Shard SizeIn simple terms, shards size should be small but not small enough so that the underlying ES instance does not have a needless strain on the hardware.
Let us consider the below example for better understanding:
Scenario 1:
Let’s say you have 50 GiB of data and you don’t anticipate its growth. Using the formula above, the shard count should be (50 * 1.1 / 30) = 2.
Note: The selected desired shard size is 30 GiB.
Scenario 2:
Assume the same 50 GiB is expected to quadruple by next year. In this case, the estimated shard count would be ((50 + 150) * 1.1 / 30) = 8.
Even though we won’t have the extra 150 GiB of data immediately, it’s crucial to ensure that we don’t create unnecessary shards. Remember, shards consume significant CPU and memory resources. In this situation, creating small shards could lead to performance degradation.
With the above shard size of 8, let’s do the calculation: (50 * 1.1) / 8 = 6.86 GiB per shard.
This shard size is well below the recommended size range of 10–50 GiB and will consume extra resources. To resolve this, we should consider a middle ground approach of 5 shards, which gives us 11 GiB (50 * 1.1 / 5) shards now, and 44 GiB ((50 + 150) * 1.1 / 5) in the future.
In both scenarios, the shard sizing is more of an approximation than an appropriate size.
It’s crucial to understand that you should never aim for precise sizing as you risk running out of disk space before hitting the set threshold limit. For instance, consider an instance with 128 GiB of disk space. If you maintain a disk usage below 80% (103 GiB) and the shard size is 10 GiB, then you can accommodate approximately 10 shards.
Note: It’s recommended to have no more than 20 shards per GiB of Java heap on a given node.
Choosing instance types and testing:
After determining the storage requirements and selecting the number of shards needed, the subsequent step is to decide on the hardware. While hardware requirements may differ between workloads, we can make an educated estimate. Generally, the storage limits for each instance type correspond to the amount of CPU and memory that your workloads may require.
The following formulae help with better understanding when it comes to choosing the right instance type
Total Data (GB) = Raw data (GB) per day * Number of days retained * (Number of replicas + 1)Total Storage (GB) = Total data (GB) * (1 + 0.15 disk Watermark threshold + 0.1 Margin of error)Total Data Nodes = ROUNDUP(Total storage (GB) / Memory per data node / Memory:Data ratio)
For a better understanding of the formulae, let us consider the below example:
A logging application pushes close to 3 GiB data per day and the retention period of data is 90 days
You can use 8GB memory per node for this small deployment. Let’s do the math:
Total Data (GB) = 3GB x (3 x 30 days) x 2 = 540GB
Total Storage (GB)= 540GB x (1+0.15+0.1) = 675GB
Total Data Nodes = 675GB disk / 8GB RAM /30 ratio = 3 nodes.
To summarise everything we have seen so far:
