2020.10.21
1. Elastic Observability
a. Elastic Observability (https://www.elastic.co/guide/en/kibana/7.9/observability.html)
- combines your logs, metrics, and APM data for unified visibility and analysis using one tool.
Elastic Observability : Logs + Metrics + APM + Uptime
b. Metrics app (https://www.elastic.co/guide/en/kibana/7.9/xpack-infra.html)
- The Metrics app in Kibana enables you to monitor your infrastructure metrics and identify problems in real time.
✓ View your infrastructure metrics by hosts, Kubernetes pods, or Docker containers.
✓ View current and historic values for metrics such as CPU usage, memory usage, and network traffic for each component.
✓ Use Metrics Explorer to group and visualize multiple customizable metrics for one or more components in a graphical format.
- Inventory / Hosts view
- Inventory / Kubernetes POD view
- Inventory / POD view
c. Logs app (https://www.elastic.co/guide/en/kibana/7.9/xpack-logs.html)
- The Logs app in Kibana allows you to search, filter and tail all the logs collected into Elastic Stack.
✓ Check out filtering logs using the keyword or plain text search.
✓ You can move back and forth in time using the time picker or the timeline view on the side.
✓ click on the Streaming button and use highlighting to accentuate that important bit of the info you are waiting to see
d. Uptime app (https://www.elastic.co/guide/en/kibana/7.9/xpack-uptime.html)
- The Uptime app in Kibana enables you to monitor the status of network endpoints via HTTP/S, TCP, and ICMP.
Monitor availability issues across your apps and services
e. out-of-the-box dashboards
- A dashboard is a collection of visualizations, searches, and maps, typically in real-time.
- 관심있는 Dashboards (검색 조건)
Metricbeat Kubernetes
Metricbeat System
Filebeat CoreDNS
- 예제
e. Subscriptions (https://www.elastic.co/kr/subscriptions)
- 범례: 오픈소스 (Apache 2.0), 기본 (영구 무료 플랜)
- 주요 내용
f. Reference
- Kubernetes observability tutorial:
✓ Log monitoring and analysis https://www.elastic.co/kr/blog/kubernetes-observability-tutorial-k8s-log-monitoring-and-analysis-elastic-stack
✓ Metrics collection and analysis : https://www.elastic.co/kr/blog/kubernetes-observability-tutorial-k8s-metrics-collection-and-analysis
✓ Monitoring application performance : https://www.elastic.co/kr/blog/kubernetes-observability-tutorial-k8s-monitoring-application-performance-with-elastic-apm
✓ K8s cluster setup and demo app deployment : https://www.elastic.co/kr/blog/kubernetes-observability-tutorial-k8s-cluster-setup-demo-app-deployment
✓ https://www.elastic.co/kr/blog/elastic-observability-update-7-2-0
✓ https://www.elastic.co/kr/infrastructure-monitoring
✓ https://www.elastic.co/kr/log-monitoring
2. Install
a. Environments
- ECK(Elastic Cloud on Kubernetes) 1.4.1 + Elasticsearch/Kibana 7.12.1
- metricbeat v7.12.1 + kube-state-metrics v 1.9.5
- filebeat v7.12.1
- heartbeat v7.12.1
- Kubernetes 1.16.15
b. Elasticsearch / Kibana cluster 생성
- Elastic cloud on kubernetes가 구성되어 있는 환경에서 elastic observability가 사용할 elasticsearch/kibna cluster 생성
$ mkdir -p ~/k8s-oss/elastic-cloud-kubernetes && cd ~/k8s-oss/elastic-cloud-kubernetes
$ vi elasticsearch-observer.yaml
apiVersion: elasticsearch.k8s.elastic.co/v1
kind: Elasticsearch
metadata:
name: observer
namespace: elastic-cluster
spec:
version: 7.12.1
nodeSets:
- name: master-nodes
count: 3
config:
node.roles: ["master"]
podTemplate:
spec:
initContainers:
- command:
- sh
- -c
- sysctl -w vm.max_map_count=262144
name: set-max-map-count
securityContext:
privileged: true
volumeClaimTemplates:
- metadata:
name: elasticsearch-data
spec:
accessModes:
- ReadWriteOnce
resources:
requests:
storage: 10Gi
storageClassName: nfs-sc-iap
- name: data-nodes
count: 5
config:
node.roles: ["data"]
podTemplate:
spec:
initContainers:
- command:
- sh
- -c
- sysctl -w vm.max_map_count=262144
name: set-max-map-count
securityContext:
privileged: true
volumeClaimTemplates:
- metadata:
name: elasticsearch-data
spec:
accessModes:
- ReadWriteOnce
resources:
requests:
storage: 100Gi
storageClassName: nfs-sc-iap
http:
tls:
selfSignedCertificate:
disabled: true
$ vi kibana-observer.yaml
apiVersion: kibana.k8s.elastic.co/v1
kind: Kibana
metadata:
name: observer
namespace: elastic-cluster
spec:
version: 7.12.1
count: 3
elasticsearchRef:
name: observer
http:
tls:
selfSignedCertificate:
disabled: true
$ k apply -f elasticsearch-observer.yaml
$ k apply -f kibana-observer.yaml
$ kubectl patch service observer-es-http -n elastic-cluster -p '{ "spec": { "type": "NodePort" } }'
$ kubectl patch service observer-kb-http -n elastic-cluster -p '{ "spec": { "type": "NodePort" } }'
$ k get svc -n elastic-cluster | egrep 'NAME|observer'
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
observer-es-default ClusterIP None <none> 9200/TCP 7m3s
observer-es-http NodePort 10.107.168.194 <none> 9200:31643/TCP 7m4s
observer-es-transport ClusterIP None <none> 9300/TCP 7m4s
observer-kb-http NodePort 10.99.79.112 <none> 5601:31748/TCP 4m56s
$ kubectl get secret observer-es-elastic-user -n elastic-cluster -o go-template='{{.data.elastic | base64decode}}'
Rf65c4I9VxmhR31K4y1Lu42P
$
c. Metricbeat
- Fetches sets of metrics from the operating system and services
- Reference
https://www.elastic.co/kr/beats/metricbeat
https://github.com/elastic/beats/tree/master/deploy/kubernetes
$ mkdir -p ~/k8s-oss/elastic-observability && cd ~/k8s-oss/elastic-observability
$ curl -L -O https://raw.githubusercontent.com/elastic/beats/7.12/deploy/kubernetes/metricbeat-kubernetes.yaml
$ vi metricbeat-kubernetes.yaml
apiVersion: v1
kind: ConfigMap
metadata:
name: metricbeat-daemonset-config
…
data:
metricbeat.yml: |-
# import Kibana dashboards
setup.dashboards.enabled: true
setup.kibana.host: "observer-kb-http.elastic-cluster.svc.cluster.local"
setup.kibana.protocol: "http"
setup.kibana.username: "elastic"
setup.kibana.password: "Rf65c4I9VxmhR31K4y1Lu42P"
…
metricbeat.autodiscover:
providers:
- type: kubernetes
templates:
- config:
- module: kubernetes
metricsets:
- state_node
- state_deployment
# - state_daemonset # Auto discover config check failed for config - metricset 'kubernetes/state_daemonset' not found
…
processors:
- add_cloud_metadata:
- add_host_metadata:
- add_kubernetes_metadata:
- add_docker_metadata:
…
kind: DaemonSet
metadata:
name: metricbeat
…
image: docker.elastic.co/beats/metricbeat:7.12.1
…
- name: ELASTICSEARCH_HOST
value: observer-es-http.elastic-cluster.svc.cluster.local
- name: ELASTICSEARCH_PORT
value: “9200"
- name: ELASTICSEARCH_USERNAME
value: elastic
- name: ELASTICSEARCH_PASSWORD
value: Rf65c4I9VxmhR31K4y1Lu42P
…
resources:
limits:
memory: 500Mi # default : 200Mi
…
$ kubectl apply -f metricbeat-kubernetes.yaml
$ k get pod -n kube-system -o wide | egrep 'NAME|metricbeat'
NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES
metricbeat-2qpkc 1/1 Running 45 16h 14.52.244.210 iap05 <none> <none>
metricbeat-5gtz8 1/1 Running 54 16h 14.52.244.213 iap10 <none> <none>
metricbeat-5z6cb 1/1 Running 46 16h 14.52.244.208 iap04 <none> <none>
metricbeat-66knn 1/1 Running 45 16h 14.52.244.211 iap06 <none> <none>
metricbeat-6hnz6 1/1 Running 53 16h 14.52.244.214 iap11 <none> <none>
metricbeat-ffrw7 1/1 Running 44 16h 14.52.244.140 iap08 <none> <none>
metricbeat-kfwz2 1/1 Running 42 16h 14.52.244.139 iap07 <none> <none>
metricbeat-qmcdt 1/1 Running 44 16h 14.52.244.141 iap09 <none> <none>
$ kubectl --namespace=kube-system get ds/metricbeat
NAME DESIRED CURRENT READY UP-TO-DATE AVAILABLE NODE SELECTOR AGE
metricbeat 8 8 8 8 8 <none> 16h
$
- Troubleshooting #1
✓ Problem: ?docker.elastic.co/beats/metricbeat:7.9.3 not found"
$ k describe pod metricbeat-8jz88 -n kube-system | grep Events -A 10
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Normal Scheduled 99s default-scheduler Successfully assigned kube-system/metricbeat-5b8696cdf6-dh87j to minikube
Normal Pulling 54s (x3 over 99s) kubelet, minikube Pulling image "docker.elastic.co/beats/metricbeat:7.9.3"
Warning Failed 49s (x3 over 93s) kubelet, minikube Failed to pull image "docker.elastic.co/beats/metricbeat:7.9.3": rpc error: code = Unknown desc = Error response from daemon: manifest for docker.elastic.co/beats/metricbeat:7.9.3 not found: manifest unknown: manifest unknown
$✓ Solution :
Image 변경 (docker.elastic.co/beats/metricbeat:7.9.3 -> docker.elastic.co/beats/metricbeat:7.9.2)
- Troubleshooting #2
✓ Problem: "127.0.0.1:10249: connect: connection refused"
$ k logs metricbeat-bwsww -n kube-system | grep -i error
2020-10-06T10:07:38.486Z INFO module/wrapper.go:259 Error fetching data for metricset kubernetes.proxy: error getting processed metrics: error making http request: Get "http://localhost:10249/metrics": dial tcp 127.0.0.1:10249: connect: connection refused✓ Cause:
▷ Minikube
yoosungjeon@ysjeon-Dev ~ % minikube ssh
$ netstat -na | grep 10249
tcp 0 0 192.168.64.11:10249 0.0.0.0:* LISTEN
$
▷ k8s / linux
$ sudo netstat -nap | grep 10249
tcp 0 0 127.0.0.1:10249 0.0.0.0:* LISTEN 1766/kube-proxy
$
✓ Solution:
metricbeat-kubernetes.yaml의 ConfigMap절에서 hosts 항목 값 변경 (hosts: ["localhost:10249”] —> hosts: [“Minikube IP:10249”])
- Troubleshooting #3
✓ Problem: "OMMKilled"
$ k describe pod metricbeat-2z9pn -n kube-system | grep -i "last state" -A5
Last State: Terminated
Reason: OOMKilled
Exit Code: 137
Started: Mon, 19 Oct 2020 14:24:47 +0900
Finished: Mon, 19 Oct 2020 14:25:31 +0900
Ready: False
$ k get pod metricbeat-wzfdz -n kube-system -w
NAME READY STATUS RESTARTS AGE
metricbeat-wzfdz 1/1 Running 1176 4d22h
metricbeat-wzfdz 0/1 OOMKilled 1176 4d22h
metricbeat-wzfdz 0/1 CrashLoopBackOff 1176 4d22h✓ Solution:
$ vi metricbeat-kubernetes.yaml
…
resources:
limits:
memory: 500Mi # default : 200Mi
$
- Troubleshooting #4
✓ Problem: "'kubernetes/state_daemonset' not found"
$ k describe pod metricbeat-8jz88 -n kube-system | grep Events -A 10
$ kubectl logs metricbeat-wzfdz -n kube-system -f
2020-10-19T06:09:31.287Z ERROR [autodiscover] autodiscover/autodiscover.go:209 Auto discover config check failed for config '{
"add_metadata": true,
"hosts": [
"xxxxx"
],
"metricsets": [
"state_node",
"state_deployment",
"state_daemonset",
…
}', won't start runner: 1 error: metricset 'kubernetes/state_daemonset' not found✓ Solution:
metricbeat-kubernetes.yaml에서 state_daemonset 주석 처리
d. kube-state-metrics v 1.9.5
- kube-state-metrics is a simple service that listens to the Kubernetes API server and generates metrics about the state of the objects. It is not focused on the health of the individual Kubernetes components, but rather on the health of the various objects inside, such as deployments, nodes and pods.
- Elastic metricbeat에서 kube-state-metrics를 주기적으로 호출
- Refernence
https://github.com/kubernetes/kube-state-metrics
https://medium.com/finda-tech/kube-state-metrics%EC%97%90-%EB%8C%80%ED%95%B4%EC%84%9C-1303b10fb8f8
- Compatibility matrix
$ k describe pod metricbeat-8jz88 -n kube-system | grep Events -A 10
$ mkdir -p ~/k8s-oss/elastic-observability/kube-state-metrics && cd ~/k8s-oss/elastic-observability/kube-state-metrics
$ wget https://github.com/kubernetes/kube-state-metrics/raw/master/examples/standard/cluster-role-binding.yaml
$ wget https://github.com/kubernetes/kube-state-metrics/raw/master/examples/standard/cluster-role.yaml
$ wget https://github.com/kubernetes/kube-state-metrics/raw/master/examples/standard/deployment.yaml
$ wget https://github.com/kubernetes/kube-state-metrics/raw/master/examples/standard/service-account.yaml
$ wget https://github.com/kubernetes/kube-state-metrics/raw/master/examples/standard/service.yaml
$ sed -i -e 's/2.0.0-alpha.1/1.9.5/' *
$ kubectl apply -f .
$ kubectl get pod -n kube-system -o wide | egrep 'NAME|kube-state-metrics'
NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES
kube-state-metrics-6d4847485d-fgvbp 1/1 Running 0 3m52s 10.244.9.71 iap11 <none> <none>
$ kubectl get svc -n kube-system | egrep 'NAME|kube-state-metrics'
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
kube-state-metrics ClusterIP None <none> 8080/TCP,8081/TCP 54s
$ kubectl run -it --rm --restart=Never --image=centos centos -- bash
[root@centos /]# curl -s http://10.244.9.47:8080/metrics | grep -v "#" | head -n 3
kube_configmap_info{namespace="cert-manager",configmap="cert-manager-parameters"} 1
kube_configmap_info{namespace="pgo",configmap="emo-dev-leader"} 1
kube_configmap_info{namespace="istio-system",configmap="istio-sidecar-injector"} 1
[root@centos /]# curl -s http://10.244.9.71:8080/healthz
OK
[root@centos /]# curl -s http://10.244.9.71:8081/metrics
…
e. filebeat
- Tails and ships logs
- Reference
https://www.elastic.co/kr/beats/filebeat
https://github.com/elastic/beats/tree/master/deploy/kubernetes
$ mkdir -p ~/k8s-oss/elastic-observability && cd ~/k8s-oss/elastic-observability
$ curl -L -O https://raw.githubusercontent.com/elastic/beats/7.12/deploy/kubernetes/filebeat-kubernetes.yaml
$ vi filebeat-kubernetes.yaml
…
kind: ConfigMap
metadata:
name: filebeat-config
…
data:
filebeat.yml: |-
# import Kibana dashboards
setup.dashboards.enabled: true
setup.kibana.host: "observer-kb-http.elastic-cluster.svc.cluster.local"
setup.kibana.protocol: "http"
setup.kibana.username: "elastic"
setup.kibana.password: "Rf65c4I9VxmhR31K4y1Lu42P"
…
# To enable hints based autodiscover, remove `filebeat.inputs` configuration and uncomment this:
filebeat.autodiscover:
…
processors:
- add_cloud_metadata:
- add_host_metadata:
…
kind: DaemonSet
metadata:
name: filebeat
…
image: docker.elastic.co/beats/filebeat:7.12.1
…
- name: ELASTICSEARCH_HOST
value: observer-es-http.elastic-cluster.svc.cluster.local
- name: ELASTICSEARCH_PORT
value: “9200”
- name: ELASTICSEARCH_USERNAME
value: elastic
- name: ELASTICSEARCH_PASSWORD
value: Rf65c4I9VxmhR31K4y1Lu42P
…
$ k apply -f filebeat-kubernetes.yaml
…
$ PASSWORD=$(kubectl get secret observer-es-elastic-user -n elastic-cluster -o go-template='{{.data.elastic | base64decode}}')
$ curl -u "elastic:$PASSWORD" "http://14.52.244.136:$PORT/filebeat-7.9.2-2020.10.20-000001/_search"
f. heartbeat
- Monitor services for their availability with active probing
- Reference
https://github.com/elastic/beats/tree/master/deploy/kubernetes
https://www.elastic.co/guide/en/beats/heartbeat/current/configuration-autodiscover.html
$ mkdir -p ~/k8s-oss/elastic-observability && cd ~/k8s-oss/elastic-observability
$ curl -L -O https://raw.githubusercontent.com/elastic/beats/7.12/deploy/kubernetes/heartbeat-kubernetes.yaml
$ vi heartbeat-kubernetes.yaml
…
kind: ConfigMap
metadata:
name: heartbeat-deployment-config
…
data:
metricbeat.yml: |-
# import Kibana dashboards
# setup.dashboards.enabled: true
# setup.kibana.host: "observer-kb-http.elastic-cluster.svc.cluster.local"
# setup.kibana.protocol: "http"
# setup.kibana.username: "elastic"
# setup.kibana.password: "Rf65c4I9VxmhR31K4y1Lu42P"
…
heartbeat.autodiscover: # Autodiscover pods, services, nodes 주석 제거
…
kind: Deployment
metadata:
name: heartbeat
…
image: docker.elastic.co/beats/heartbeat:7.12.1
…
- name: ELASTICSEARCH_HOST
value: observer-es-http.elastic-cluster.svc.cluster.local
- name: ELASTICSEARCH_PORT
value: "9200"
- name: ELASTICSEARCH_USERNAME
value: elastic
- name: ELASTICSEARCH_PASSWORD
value: Rf65c4I9VxmhR31K4y1Lu42P
…
$ k apply -f heartbeat-kubernetes.yaml- Troubleshooting #1
✓ Problem:
$ k logs heartbeat-85c74b67d5-8k45l -n kube-system
2020-10-21T07:12:03.723Z ERROR instance/beat.go:951 Exiting: Error importing Kibana dashboards: fail to import the dashboards in Kibana: Error importing directory /usr/share/heartbeat/kibana: No directory /usr/share/heartbeat/kibana/7
Exiting: Error importing Kibana dashboards: fail to import the dashboards in Kibana: Error importing directory /usr/share/heartbeat/kibana: No directory /usr/share/heartbeat/kibana/7
…
$✓ Solution: heartbeat용 dashboard 활성화 설정을 제거
g. Create index pattern
- Management > Stack Management > Kibana > Index Patterns to go directly to the Index Patterns UI.
a. Index pattern name = filebeat-* & Time field = @timestamp
b. Index pattern name = metricbeat-* & Time field = @timestamp
c. Index pattern name = heartbeat-* & Time field = @timestamp
'Kubernetes > Monitoring' 카테고리의 다른 글
| kube-prometheus-stack (0) | 2021.09.21 |
|---|---|
| GPU Monitor (0) | 2021.09.21 |
| Elastic Observability - filebeat/metricbeat POD 오류 (0) | 2021.09.15 |
| Dashboard on bare-metal (0) | 2021.09.15 |
| Dashboard on GCE (0) | 2021.09.15 |
댓글