aboutsummaryrefslogtreecommitdiff
diff options
context:
space:
mode:
-rw-r--r--content/2020-03-29-the-strange-case-of-elasticsearch-allocation-failure.md8
1 files changed, 4 insertions, 4 deletions
diff --git a/content/2020-03-29-the-strange-case-of-elasticsearch-allocation-failure.md b/content/2020-03-29-the-strange-case-of-elasticsearch-allocation-failure.md
index e0088ac..db4f377 100644
--- a/content/2020-03-29-the-strange-case-of-elasticsearch-allocation-failure.md
+++ b/content/2020-03-29-the-strange-case-of-elasticsearch-allocation-failure.md
@@ -1,11 +1,11 @@
1~ title: The strange case of Elasticsearch allocation failure 1~ title: The strange case of Elasticsearch allocation failure
2~ description: Elasticsearch allocation failure on some indices while reporting domain processing 2~ description: Elasticsearch allocation failure on some indices while reporting domain processing
3~ slug: /the-strange-case-of-elasticsearch-allocation-failure.html 3~ slug: /the-strange-case-of-elasticsearch-allocation-failure.html
4~ date: 2020-03-27 4~ date: 2020-03-29
5~ template: post 5~ template: post
6~ hide: false 6~ hide: false
7 7
8I've been using Elasticsearch in production for 5 years now and never had a single problem with it. Hell, never even known a problem. Just worked. All this time. The first node that I deployed is still being used in production, never updated, upgraded, touched in anyway. 8I've been using Elasticsearch in production for 5 years now and never had a single problem with it. Hell, never even known there could be a problem. Just worked. All this time. The first node that I deployed is still being used in production, never updated, upgraded, touched in anyway.
9 9
10All this bliss came to an abrupt end this Friday when I got notification that Elasticsearch cluster went warm. Well, warm is not that bad right? Wrong! Quickly after that I got another email which sent chills down my spine. Cluster is now red. RED! Now, shit really hit the fan! 10All this bliss came to an abrupt end this Friday when I got notification that Elasticsearch cluster went warm. Well, warm is not that bad right? Wrong! Quickly after that I got another email which sent chills down my spine. Cluster is now red. RED! Now, shit really hit the fan!
11 11
@@ -15,7 +15,7 @@ The strangest thing about it all was, that queries were still being fulfilled. D
15 15
16> **Please, DO NOT do what I did.** Seriously! Please ask someone on official forums or if you know an expert please consult him. There could be million of reasons and these solution fit my problem. Maybe in your case it would disastrous. I had all the data backed up and even if I would fail spectacularly I would be able to restore the data. It would be a huge pain and I would loose couple of days but I had a plan B. 16> **Please, DO NOT do what I did.** Seriously! Please ask someone on official forums or if you know an expert please consult him. There could be million of reasons and these solution fit my problem. Maybe in your case it would disastrous. I had all the data backed up and even if I would fail spectacularly I would be able to restore the data. It would be a huge pain and I would loose couple of days but I had a plan B.
17 17
18I got a feeling that something must be wrong with allocation so I looked at the results. 18Executing allocation and told me what the problem was but no clear solution yet.
19 19
20```yaml 20```yaml
21GET /_cat/allocation?format=json 21GET /_cat/allocation?format=json
@@ -37,7 +37,7 @@ I got a message that significantly reduced my options.
37} 37}
38``` 38```
39 39
40After that I went on a hunt again. I won't bother you with all the details because hours/days went by until I was finally able to re-index the problematic index and hope for the best. Until that moment even re-indexing was giving me errors. 40After that I went on a hunt again. I won't bother you with all the details because hours/days went by until I was finally able to re-index the problematic index and hoped for the best. Until that moment even re-indexing was giving me errors.
41 41
42```yaml 42```yaml
43POST _reindex 43POST _reindex