Massive formatting and added figcaption

1 files changed, 37 insertions, 37 deletions

diff --git a/content/posts/2020-03-29-the-strange-case-of-elasticsearch-allocation-failure.md b/content/posts/2020-03-29-the-strange-case-of-elasticsearch-allocation-failure.md
index d0f4bac..bf1d710 100644
--- a/content/posts/2020-03-29-the-strange-case-of-elasticsearch-allocation-failure.md
+++ b/content/posts/2020-03-29-the-strange-case-of-elasticsearch-allocation-failure.md
@@ -5,33 +5,33 @@ date: 2020-03-29T12:00:00+02:00
 draft: false
 ---
 
-I've been using Elasticsearch in production for 5 years now and never had a 
-single problem with it. Hell, never even known there could be a problem. Just 
+I've been using Elasticsearch in production for 5 years now and never had a
+single problem with it. Hell, never even known there could be a problem. Just
 worked. All this time. The first node that I deployed is still being used in
 production, never updated, upgraded, touched in anyway.
 
-All this bliss came to an abrupt end this Friday when I got notification that 
-Elasticsearch cluster went warm. Well, warm is not that bad right? Wrong! 
-Quickly after that I got another email which sent chills down my spine. 
-Cluster is now red. RED! Now, shit really hit the fan!
+All this bliss came to an abrupt end this Friday when I got notification that
+Elasticsearch cluster went warm. Well, warm is not that bad right? Wrong!
+Quickly after that I got another email which sent chills down my spine.  Cluster
+is now red. RED! Now, shit really hit the fan!
 
-I tried googling what could be the problem and after executing allocation 
-function noticed that some shards were unassigned and 5 attempts were already 
-made (which is BTW to my luck the maximum) and that meant I am basically fucked. 
-They also applied that one should wait for cluster to re-balance itself. So, 
-I waited. One hour, two hours, several hours. Nothing, still RED.
+I tried googling what could be the problem and after executing allocation
+function noticed that some shards were unassigned and 5 attempts were already
+made (which is BTW to my luck the maximum) and that meant I am basically fucked.
+They also applied that one should wait for cluster to re-balance itself. So, I
+waited. One hour, two hours, several hours. Nothing, still RED.
 
-The strangest thing about it all was, that queries were still being fulfilled. 
-Data was coming out. On the outside it looked like nothing was wrong but 
-everybody that would look at the cluster would know immediately that something 
+The strangest thing about it all was, that queries were still being fulfilled.
+Data was coming out. On the outside it looked like nothing was wrong but
+everybody that would look at the cluster would know immediately that something
 was very very wrong and we were living on borrowed time here.
 
-> **Please, DO NOT do what I did.** Seriously! Please ask someone on official 
-forums or if you know an expert please consult him. There could be million of 
-reasons and these solution fit my problem. Maybe in your case it would 
-disastrous. I had all the data backed up and even if I would fail spectacularly 
-I would be able to restore the data. It would be a huge pain and I would 
-loose couple of days but I had a plan B.
+> **Please, DO NOT do what I did.** Seriously! Please ask someone on official
+forums or if you know an expert please consult him. There could be million of
+reasons and these solution fit my problem. Maybe in your case it would
+disastrous. I had all the data backed up and even if I would fail spectacularly
+I would be able to restore the data. It would be a huge pain and I would loose
+couple of days but I had a plan B.
 
 Executing allocation and told me what the problem was but no clear solution yet.
 
@@ -39,14 +39,14 @@ Executing allocation and told me what the problem was but no clear solution yet.
 GET /_cat/allocation?format=json
 ```
 
-I got a message that `ALLOCATION_FAILED` with additional info 
-`failed to create shard, failure ioexception[failed to obtain in-memory shard lock]`. 
-Well splendid! I must also say that our cluster is capable more than enough 
-to handle the traffic. Also JVM memory pressure never was an issue. So what
-happened really then?
+I got a message that `ALLOCATION_FAILED` with additional info `failed to create
+shard, failure ioexception[failed to obtain in-memory shard lock]`.  Well
+splendid! I must also say that our cluster is capable more than enough to handle
+the traffic. Also JVM memory pressure never was an issue. So what happened
+really then?
 
-I tried also re-routing failed ones with no success due to AWS restrictions 
-on having managed Elasticsearch cluster (they lock some of the functions).
+I tried also re-routing failed ones with no success due to AWS restrictions on
+having managed Elasticsearch cluster (they lock some of the functions).
 
 ```yaml
 POST /_cluster/reroute?retry_failed=true
@@ -60,10 +60,10 @@ I got a message that significantly reduced my options.
 }
 ```
 
-After that I went on a hunt again. I won't bother you with all the details 
-because hours/days went by until I was finally able to re-index the problematic 
-index and hoped for the best. Until that moment even re-indexing was giving 
-me errors.
+After that I went on a hunt again. I won't bother you with all the details
+because hours/days went by until I was finally able to re-index the problematic
+index and hoped for the best. Until that moment even re-indexing was giving me
+errors.
 
 ```yaml
 POST _reindex
@@ -77,8 +77,8 @@ POST _reindex
 }
 ```
 
-I needed to do this multiple times to get all the documents re-indexed. Then 
-I dropped the original one with the following command.
+I needed to do this multiple times to get all the documents re-indexed. Then I
+dropped the original one with the following command.
 
 ```yaml
 DELETE /myindex
@@ -98,10 +98,10 @@ POST _reindex
 }
 ```
 
-On the surface it looks like all is working but I have a long road in front 
-of me to get all the things working again. Cluster now shows that it is in 
-Green mode but I am also getting a notification that the cluster has 
-processing status which could mean million of things.
+On the surface it looks like all is working but I have a long road in front of
+me to get all the things working again. Cluster now shows that it is in Green
+mode but I am also getting a notification that the cluster has processing status
+which could mean million of things.
 
 Godspeed!