본문 바로가기

Kubernetes80

Rook Ceph - failed to get status 2020.12.01 a. Problem: “op-cluster: failed to get ceph status. failed to get status” - Environments Kubernetes 1.16.15, Rook Ceph 1.3.8 [iap@iap01 ~]$ k logs rook-ceph-operator-674d4db4cf-zpp8g -n rook-ceph | egrep " E " … 2020-11-30 07:16:22.362561 E | op-cluster: failed to create cluster in namespace "rook-ceph". failed to start the mons: failed to start mon pods: failed to check mon quorum q:.. 2021. 9. 16.
K8s - Slab memory leakage 2020.12.02 a. Problem: POD - cannot allocate memory - Environments Kubernetes 1.16.15, centos 7.8 / 7.9, Docker 19.03 / 20.10 - leakage 발생 centos 7.8 / 3.10.0-1127.el7.x86_64 / Docker 19.03 (iap10, iap11) centos 7.9 / 3.10.0-1160.15.2.el7.x86_64 / Dcoker 20.10.3 - leakage 미 발생 centos 7.8 / 3.10.0-1127.el7.x86_64 / Docker 18.06 (iap04 ~ iap09) [iap@iap01 ~]$ k describe pod rook-ceph-osd-prepare-i.. 2021. 9. 16.
K8s - Node NotReady 2021.05.11 1. 개요 - Issue ✓ 특정 Node가 NotReady 상태가 되면, Node에 속한 POD는 Terminating 상태로 변경/유지되면서 self-healing이 진행되지 않음 ✓ 영향도 ▷ Deployments는 서비스가 가능 (단, replica 수 > 1 & Node 간 분산 배치) ▷ Statefulsets는 애플리케이션 마다 서비스 지속 여부가 다름 (ex. MariaDB 서비스 불가, Redis cluster는 서비스 가능) ✓ If an entire node goes down, Kubernetes generally isn’t able to spin a new one up ▷ AWS, GCP에서는 autoscaling groups 기능으로, Azure에서는 scale.. 2021. 9. 16.
K8s - CNI not ready 2020.12.01 a. Problem: Worker node - 재기동 후 상태가 not ready임 (cni) - Environments Kubernetes 1.16.15 [iap@iap01 ~]$ k get nodes iap11 NAME STATUS ROLES AGE VERSION iap11 NotReady 60d v1.16.15 [iap@iap01 ~]$ k describe nodes iap11 | grep Conditions -A6 Conditions: Type Status LastHeartbeatTime LastTransitionTime Reason Message ---- ------ ----------------- ------------------ ------ ------- MemoryPre.. 2021. 9. 15.
istio - Envoy CPU 과다 점유 2020.12.01 a. Problem: Worker node - CPU 과다 점유로 성능 저하 현상 발생 - Environments Kubernetes 1.16.15, istio 1.3 - 영향도 Rook ceph의 rook-ceph-mon-o POD가 iap04 노드에서 동작될 경우 응답 속도가 느려서 quorum에서 제외 되면서 fail-over 동작 [root@iap04 ~]# top top - 10:46:25 up 6 days, 17:58, 1 user, load average: 73.37, 77.52, 79.46 Tasks: 403 total, 19 running, 382 sleeping, 0 stopped, 2 zombie %Cpu(s): 90.1 us, 8.2 sy, 0.0 ni, 0.0 .. 2021. 9. 15.
Elasticsearch - Index lifecycle error 2020.12.08 a. Problem: Elasticsearch - Index liefecycle error - Environment ECK(Elastic Cloud on Kubernetes) 1.2.1 + Elasticsearch/Kibana 7.9.1, Kubernetes 1.16.15 - Elastic Master log [iap@iap01 ~]$ k logs observer-es-master-nodes-0 -n elastic-cluster | grep "blocked by:" -B1 -A100 | more {"type": "server", "timestamp": "2020-12-02T15:55:08,826Z", "level": "ERROR", "component": "o.e.x.i.IndexLi.. 2021. 9. 15.
Elastic Observability - filebeat/metricbeat POD 오류 2020,11.30 a. Problem: Elastic Observability - filebeat/metricbeat POD 오류 - Environment ECK(Elastic Cloud on Kubernetes) 1.2.1 + Elasticsearch/Kibana 7.9.1, Kubernetes 1.16.15 [iap@iap01 elastic-cloud-kubernetes]$ k get pod -n kube-system | egrep "filebeat|metricbeat" filebeat-2w85f 0/1 CrashLoopBackOff 13 46m filebeat-4nbwm 0/1 CrashLoopBackOff 307 25h filebeat-59wfj 0/1 CrashLoopBackOff 307 25.. 2021. 9. 15.
Rook Ceph 구성 2020.05.25 1. 구성 환경 - Rook v1.3.6, ceph image version: "14.2.10-0 nautilus”, cephcsi v2.1.2, kubenetes 1.16.15, CentOS 7.8 2. Rook / Ceph ? - Rook is an open source cloud-native storage orchestrator, providing the platform, framework, and support for a diverse set of storage solutions to natively integrate with cloud-native environments. https://rook.io/docs/rook/v1.3/ceph-examples.html - Ceph C.. 2021. 9. 15.
Rook-ceph - OSD/K8s Node 제거 2020.12.31 1. 삭제 대상 Kubernetes node : iap07 OSD (Object Storage Daemon) : 0, 3 [root@rook-ceph-tools-79d7c49c8d-4c4x5 /]# ceph osd status +----+-------+-------+-------+--------+---------+--------+---------+-----------+ | id | host | used | avail | wr ops | wr data | rd ops | rd data | state | +----+-------+-------+-------+--------+---------+--------+---------+-----------+ | 0 | iap09 | 3324M | 2.. 2021. 9. 15.
Rook ceph vs NFS 2020.11.20 1. 개요 K8s storage로 Rook ceph와 NFS 중 어떤 것을 선택할지 검토하기 위하여 작성 2. 검토 의견 - 개발 환경 (in GiGA Tech Hub)는 Rook ceph와 NFS 스토리지 모두를 제공하며 기본으로 Rook ceph를 사용 - 운영환경과 시스템별로 별도 구축되는 개발 환경은 NFS 스토리지만 제공 운영 이관 고려 : Rook Ceph 엔지니어 부재 및 적용 사례가 없어 이관 협의 시 이슈화 될 수 있기 때문에 3. Rook ceph vs NFS 비교 a. 기능 / 디스크 사용율 관점 기능 디스크 사용율 (Block storage) SPOF 구성 관리 / 운영 Rook Ceph - Block storage, Shared Filesystem, Object.. 2021. 9. 15.
MetalLB 2021.03.16 1. MetalLB 0.9.5 ? - MetalLB is a load-balancer implementation for bare metal Kubernetes clusters, using standard routing protocols. - https://metallb.universe.tf/ 2. MetalLB 배포 & Kubeflow (istio) 설정 변경 - https://www.kubeflow.org/docs/started/k8s/kfctl-istio-dex/#expose-with-a-loadbalancer a. Install MetalLB $ kubectl apply -f https://raw.githubusercontent.com/metallb/metallb/v0.9.5/m.. 2021. 9. 15.
NFS-Client Provisioner 2020.11.09 22.01.11: NFS 프로비저너를 구성시최신 버전(NFS subdir external provisioner)으로 구성할 것을 권고한다. 1. 개요 - nfs-client is an automatic provisioner that use your existing and already configured NFS server to support dynamic provisioning of Kubernetes Persistent Volumes via Persistent Volume Claims. Persistent volumes are provisioned as ${namespace}-${pvcName}-${pvName}. - 제약사항 nfs-client는 PVC 생성시 설정한 size .. 2021. 9. 15.
Dashboard on bare-metal 2021.11.15, 2020.06.16 1. K8s Dashboard install a. install Dashboard - https://github.com/kubernetes/dashboard#kubernetes-dashboard # Latest - Release # v2.0.0 (K8s 1.18 호환), v.2.0.0-rc3 (K8s 1.16 호환), v.2.0.0-beta3 (K8s 1.15 호환) $ k version --short Client Version: v1.20.4 Server Version: v1.16.15 $ kubectl apply -f https://raw.githubusercontent.com/kubernetes/dashboard/v2.0.0-rc3/aio/deploy/rec.. 2021. 9. 15.
Dashboard on GCE 2020.04.10 1. Kubernetes dashboard install on Google K8s Engine Ref. https://github.com/kubernetes/dashboard#kubernetes-dashboard $ kubectl apply -f https://raw.githubusercontent.com/kubernetes/dashboard/v2.0.0-beta8/aio/deploy/recommended.yaml namespace/kubernetes-dashboard created serviceaccount/kubernetes-dashboard created service/kubernetes-dashboard created secret/kubernetes-dashboard-certs.. 2021. 9. 15.
keepalived, haproxy for K8s 2021.03.31, 2020.06.10 1. 개요 Kubernetes master를 HA 구성시 L4 스위치가 없을 경우 Software load balancing로 설정하는 방법 2. 환경 - keepalived 2.0.20, HAProxy 2.0.14 The keepalived service provides a virtual IP managed by a configurable health check. The haproxy service can be configured for simple stream-based load balancing thus allowing TLS termination - 구성 내역 Server IP : iap1(14.52.244.134:6443), iap2(14.52.244.1.. 2021. 9. 15.

티스토리툴바