DevOps Knowledge Hub
DevOps Knowledge Hub

Troubleshooting Common Elasticsearch Shard Allocation Failures

Learn to troubleshoot and resolve common Elasticsearch shard allocation failures. This guide covers identifying unassigned shards, diagnosing issues like disk space errors, node unavailability, and allocation filtering, and provides actionable solutions and best practices for maintaining a healthy Elasticsearch cluster.

24 views

Troubleshooting Common Elasticsearch Shard Allocation Failures

Elasticsearch, a powerful distributed search and analytics engine, relies heavily on its ability to distribute data across multiple nodes using shards. When these shards fail to allocate, it can lead to data unavailability, search failures, and a degraded cluster health. Understanding the common causes of shard allocation failures and knowing how to diagnose and resolve them is crucial for maintaining a stable and performant Elasticsearch environment. This article will guide you through the most frequent issues and provide actionable steps to get your shards back in an assigned state.

This guide focuses on practical troubleshooting for production Elasticsearch environments. We will cover identifying unassigned shards, understanding common reasons for failure such as disk space, allocation rules, and node issues, and provide clear steps to resolve these problems efficiently. By mastering these techniques, you can minimize downtime and ensure the reliability of your Elasticsearch cluster.

Identifying Unassigned Shards

The first step in troubleshooting is to identify which shards are unassigned and why. Elasticsearch provides several tools for this:

Using the Cluster Health API

The _cluster/health API provides a high-level overview of your cluster's status. Look for unassigned_shards in the response. A non-zero value indicates a problem.

GET _cluster/health

Example Response Snippet:

{
  "cluster_name": "my-es-cluster",
  "status": "yellow",
  "timed_out": false,
  "number_of_nodes": 3,
  "number_of_data_nodes": 3,
  "active_primary_shards": 10,
  "active_shards": 20,
  "relocating_shards": 0,
  "initializing_shards": 1,
  "unassigned_shards": 1,
  "delayed_unassigned_shards": 0,
  "number_of_pending_tasks": 0,
  "max_length_search_concurrency": 1000,
  "max_length_search_size": 10000,
  "active_shards_percent_as_number": 95.45454545454545
}

In this example, "status": "yellow" and "unassigned_shards": 1 indicate that there is one unassigned shard. A red status means one or more primary shards are unassigned, impacting data availability. A yellow status means replica shards are unassigned, but primary shards are allocated, so your data is still searchable but not fully redundant.

Using the Allocation Explain API

For detailed insights into why a specific shard is unassigned, the _cluster/allocation/explain API is invaluable. You can provide shard details or let it analyze the cluster's state.

To get an explanation for any unassigned shard:

GET _cluster/allocation/explain

To get an explanation for a specific shard (replace index_name and shard_id):

GET _cluster/allocation/explain
{
  "index": "my-index",
  "shard": 0,
  "primary": true
}

Common Causes and Solutions

Several factors can lead to shards being unassigned. Here are the most common ones and how to address them:

1. Insufficient Disk Space

This is arguably the most frequent cause of shard allocation failures. When a node runs out of disk space, Elasticsearch prevents new shards from being allocated to it to avoid data corruption and ensure stability. It also might evict existing shards.

  • Symptom: The Allocation Explain API will often report messages like "cannot allocate because disk usage [X%] exceeds the low watermark [Y%]" or "cannot allocate because disk usage [X%] exceeds the high watermark [Y%]".
  • Diagnosis: Check the disk usage on your data nodes. You can use the _cat/allocation API for a quick overview:
    bash GET _cat/allocation?v
    Look for nodes with high disk usage percentages.
  • Solutions:
    • Add More Disk Space: The most straightforward solution is to add more storage to the affected nodes or replace existing disks with larger ones.
    • Delete Unused Indices: Identify and delete old or unnecessary indices that are consuming disk space.
    • Adjust Watermarks: You can adjust the disk usage watermarks (cluster.routing.allocation.disk.watermark.low, cluster.routing.allocation.disk.watermark.high, cluster.routing.allocation.disk.watermark.flood_stage) in your elasticsearch.yml configuration or dynamically via the cluster settings API. However, caution is advised when adjusting these, as they are designed to protect your cluster. Lowering them without adding capacity can lead to further issues.
      json PUT _cluster/settings { "persistent": { "cluster.routing.allocation.disk.watermark.low": "85%", "cluster.routing.allocation.disk.watermark.high": "90%", "cluster.routing.allocation.disk.watermark.flood_stage": "95%" } }
    • Add More Nodes: Scale out your cluster by adding more data nodes. This distributes the data and reduces the load on individual nodes.
    • Force Merge or Delete Old Data: If you have time-series data, consider using the _forcemerge API on older indices to reduce the number of segments (which can free up disk space) or use index lifecycle management (ILM) to automatically delete old data.

2. Node Not Available or Restarting

If a node is down, restarting, or experiencing network issues, any shards residing on that node will become unassigned. If it's a primary shard, the cluster status will turn red.

  • Symptom: The Allocation Explain API will indicate that the shard cannot be allocated because the node is not available or is marked as (excluded) due to being down.
  • Diagnosis: Use the _cat/nodes API to check the status of your nodes. Ensure all expected nodes are listed and healthy.
    bash GET _cat/nodes?v
    Check Elasticsearch logs on the affected node for any errors or signs of shutdown.
  • Solutions:
    • Restart the Node: If the node is down, attempt to restart the Elasticsearch service.
    • Resolve Network Issues: Ensure the node can communicate with other nodes in the cluster.
    • Check Logs: Examine the Elasticsearch logs for the specific node to identify the root cause of the failure (e.g., out of memory, disk errors, JVM issues).
    • Increase index.unassigned.node_left.delayed_timeout: If nodes are frequently joining and leaving the cluster (e.g., during rolling restarts), you might see replica shards become unassigned temporarily. The index.unassigned.node_left.delayed_timeout setting (default 1 minute) allows Elasticsearch to wait before marking shards on a departed node as unassigned, giving the node time to rejoin. Increase this value if necessary, but be mindful of the impact on recovery time.

3. Allocation Filtering and Awareness Rules

Elasticsearch allows you to control where shards are allocated using various allocation rules, such as node attributes and anti-affinities. If these rules prevent allocation, shards can become unassigned.

  • Symptom: The Allocation Explain API will report that allocation is disabled for specific attributes or that no suitable nodes are available according to configured rules.
  • Diagnosis:
    • Check your index settings for index.routing.allocation.require.*, index.routing.allocation.include.*, index.routing.allocation.exclude.*, and index.routing.allocation.total_shards_per_node.
    • Check your cluster-level settings for cluster.routing.allocation.enable (e.g., all, primaries, new_primaries, none).
    • Verify node attributes using GET _cat/nodeattrs?v.
  • Solutions:
    • Update Index Settings: Remove or adjust restrictive index routing rules. For example, to allow allocation to any node:
      json PUT my-index/_settings { "index": { "routing": { "allocation": { "require": null, "include": null, "exclude": null } } } }
    • Update Cluster Settings: Temporarily enable allocation if it was disabled:
      json PUT _cluster/settings { "persistent": { "cluster.routing.allocation.enable": "all" } }
      Remember to revert this setting if it was only meant to be temporary.
    • Update Node Attributes: Ensure your nodes have the expected attributes defined in elasticsearch.yml (e.g., node.attr.zone: us-east-1) and that these attributes align with your allocation rules. After changing elasticsearch.yml, nodes need to be restarted for changes to take effect.

4. Corrupted Shard Data (Rare)

In rare cases, shard data can become corrupted, preventing Elasticsearch from starting up or allocating the shard. This is more common with underlying disk issues.

  • Symptom: Logs might show errors related to reading shard data or index corruption. The Allocation Explain API might not give a clear reason or might point to a read error.
  • Diagnosis: Examine Elasticsearch logs closely on the node where the shard is expected to be located. Look for I/O errors or data corruption messages.
  • Solutions:
    • Restore from Snapshot: The most reliable solution is to restore the affected index (or the entire cluster) from a known good snapshot. This is why regular backups are critical.
    • Force Delete Shard (Last Resort): If you cannot restore from a snapshot and the data is non-critical or can be re-indexed, you may need to force-delete the corrupted shard. This is an advanced operation and should only be performed when you understand the implications. You typically need to stop the affected node, remove the shard data directory manually, and then restart the node. This will result in data loss for that shard. Consult Elasticsearch documentation for the exact procedure for your version.

5. Insufficient Relocation Capacity

When a node leaves the cluster or disk space issues arise, Elasticsearch attempts to relocate shards to other nodes. If there aren't enough suitable nodes or if the cluster is already under heavy load, shard relocation can stall, leading to initializing_shards or unassigned_shards.

  • Symptom: Shards remain in initializing or relocating state for extended periods, or new shards fail to allocate.
  • Diagnosis: Check _cat/shards and _cat/allocation to see shard statuses and disk usage. Monitor cluster health and node CPU/IO utilization.
  • Solutions:
    • Add More Nodes: Increase the capacity of your cluster by adding more data nodes.
    • Free Up Resources: Address any performance bottlenecks on existing nodes (e.g., high CPU, slow disk I/O).
    • Adjust Shard Allocation Settings: You can tune settings like cluster.routing.allocation.node_concurrent_recoveries (number of shards that can be recovered concurrently on a node) and cluster.routing.allocation.node_concurrent_incoming_recoveries (number of shards that can be recovered concurrently from another node). However, be cautious as increasing these can put more strain on the cluster.

Best Practices for Prevention

  • Monitor Disk Space: Proactively monitor disk usage on all data nodes. Set up alerts for when disk usage crosses predefined thresholds (e.g., 80% or 85%).
  • Implement Index Lifecycle Management (ILM): Automate the management of time-series data, including rolling over, shrinking, and deleting old indices. This helps control disk space usage.
  • Regular Snapshots: Ensure you have a robust backup strategy with regular, automated snapshots of your data. Test your restore process periodically.
  • Understand Allocation Rules: Carefully plan and configure shard allocation rules based on your hardware, data, and availability requirements.
  • Adequate Hardware: Ensure your nodes have sufficient CPU, RAM, and I/O capabilities to handle the workload and shard recovery processes.
  • Cluster Health Monitoring: Regularly check your cluster health using the _cluster/health API and visualize it with tools like Kibana's Stack Monitoring.

Conclusion

Shard allocation failures in Elasticsearch can be a daunting issue, but by systematically diagnosing the problem using tools like the Cluster Health API and the Allocation Explain API, and understanding common causes like disk space, node availability, and allocation rules, you can effectively resolve them. Proactive monitoring and adherence to best practices, such as regular backups and ILM, are key to preventing these issues in the first place and ensuring a stable, healthy Elasticsearch cluster.