Elasticsearch - Index lifecycle error

2020.12.08

a. Problem: Elasticsearch - Index liefecycle error

    - Environment

      ECK(Elastic Cloud on Kubernetes) 1.2.1 + Elasticsearch/Kibana 7.9.1, Kubernetes 1.16.15

 

    -  Elastic Master log

  [iap@iap01 ~]$ k logs observer-es-master-nodes-0 -n elastic-cluster | grep "blocked by:" -B1 -A100 | more

  {"type": "server", "timestamp": "2020-12-02T15:55:08,826Z", "level": "ERROR", "component": "o.e.x.i.IndexLifecycleRunner", "cluster.name": "observer", "node.name": "observer-es-master-nodes-0", "message": "pol

  icy [kibana-event-log-policy] for index [.kibana-event-log-7.9.1-000002] failed on step [{\"phase\":\"hot\",\"action\":\"rollover\",\"name\":\"check-rollover-ready\"}]. Moving to ERROR step", "cluster.uuid": "

  7lYtO5R_R7muGZfwY1smrg", "node.id": "5rk5bQ6_RpqAlarp-n36aw" ,

  "stacktrace": ["org.elasticsearch.cluster.block.ClusterBlockException: index [.kibana-event-log-7.9.1-000002] blocked by: [TOO_MANY_REQUESTS/12/disk usage exceeded flood-stage watermark, index has read-only-al

  low-delete block];",

  "at org.elasticsearch.cluster.block.ClusterBlocks.indicesBlockedException(ClusterBlocks.java:222) ~[elasticsearch-7.9.1.jar:7.9.1]",

  "at org.elasticsearch.action.admin.indices.rollover.TransportRolloverAction.checkBlock(TransportRolloverAction.java:93) ~[elasticsearch-7.9.1.jar:7.9.1]",

  "at org.elasticsearch.action.admin.indices.rollover.TransportRolloverAction.checkBlock(TransportRolloverAction.java:60) ~[elasticsearch-7.9.1.jar:7.9.1]",

  ...

  "at java.lang.Thread.run(Thread.java:832) [?:?]"] }

 

    - Kibana UI > Management > Stack Management > Index Management" 메뉴에서 "Show errors" 선택

 

b. Solution: .kibana_task_manager index 삭제

    Kibana UI > Management > Stack Management > Index Management > Edit setting 선택 후 index.blocks.read_only_allow_delete 값을 false로 변경하고 저장