[Discovery-alerts] ** ACKNOWLEDGEMENT alert - search.svc.eqiad.wmnet/ElasticSearch health check for shards is CRITICAL **

[icinga at einsteinium.wikimedia.org]

Notification Type: ACKNOWLEDGEMENT

Service: ElasticSearch health check for shards
Host: search.svc.eqiad.wmnet
Address: 10.2.2.30
State: CRITICAL

Date/Time: Tue Nov 8 13:45:26 UTC 2016

Additional Info:

CRITICAL - elasticsearch inactive shards 1146 threshold =0.1% breach: status: yellow, number_of_nodes: 31, unassigned_shards: 1047, number_of_pending_tasks: 7, number_of_in_flight_fetch: 0, timed_out: False, active_primary_shards: 3040, task_max_waiting_in_queue_millis: 28704, cluster_name: production-search-eqiad, relocating_shards: 0, active_shards_percent_as_number: 87.4562171629, active_shards: 7990, initializing_shards: 99, number_of_data_nodes: 31, delayed_unassigned_shards: 0