2021.01.20
1. 개요
- RabbitMQ
✓ RabbitMQ is an open-source message-broker software (sometimes called message-oriented middleware) that originally implemented the Advanced Message Queuing Protocol (AMQP)
and has since been extended with a plug-in architecture to support Streaming Text Oriented Messaging Protocol (STOMP), MQ Telemetry Transport (MQTT), and other protocols.
- Message queue pattens
https://www.rabbitmq.com/getstarted.html
- Distributed RabbitMQ
✓ Quorum Queues (https://www.rabbitmq.com/quorum-queues.html)
The quorum queue is a modern queue type for RabbitMQ implementing a durable, replicated FIFO queue based on the Raft consensus algorithm. It is available as of RabbitMQ 3.8.0.
The quorum queue type is a alternative to durable mirrored queues purpose built for a set of use cases where data safety is a top priority.
They should be considered the default option for a replicated queue type.
✓ Queue Mirroring (https://www.rabbitmq.com/ha.html)
Classic mirrored queues will be removed in a future version of RabbitMQ.
By default, contents of a queue within a RabbitMQ cluster are locateπd on a single node (the node on which the queue was declared).
This is in contrast to exchanges and bindings, which can always be considered to be on all nodes.
Queues can optionally run mirrors (additional replicas) on other cluster nodes.
- RabbitMQ Cluster Kubernetes Operator is a Kubernetes operator that automates provisioning, management, and operations of RabbitMQ clusters running on Kubernetes.
✓ The operator provides the following key features:
▷ Provisioning of single-node and multi-node RabbitMQ clusters
▷ Automatic reconciliation of deployed clusters whenever their actual state does not match the expected state
▷ Monitoring of RabbitMQ clusters using Prometheus and Grafana
✓ The Operator v1.3.0 requires
Kubernetes 1.17 or above
RabbitMQ DockerHub image 3.8.8+
✓ References
https://www.rabbitmq.com/kubernetes/operator/operator-overview.html
https://github.com/rabbitmq/cluster-operator/releases
2. Environments
- RabbitMQ Cluster Operator v1.3.0
- RabbitMQ 3.8.9
- Kubernetes 1.16.15 / Helm 3.3.1
- kube-prometheus-stack 12.8.0 / Prometheus 0.44.0 / Grafana 7.2.1
3. Installing RabbitMQ Cluster Operator in a Kubernetes cluster
- https://www.rabbitmq.com/kubernetes/operator/install-operator.html
$ kubectl apply -f "https://github.com/rabbitmq/cluster-operator/releases/latest/download/cluster-operator.yml"
namespace/rabbitmq-system created
customresourcedefinition.apiextensions.k8s.io/rabbitmqclusters.rabbitmq.com created
serviceaccount/rabbitmq-cluster-operator created
role.rbac.authorization.k8s.io/rabbitmq-cluster-leader-election-role created
clusterrole.rbac.authorization.k8s.io/rabbitmq-cluster-operator-role created
rolebinding.rbac.authorization.k8s.io/rabbitmq-cluster-leader-election-rolebinding created
clusterrolebinding.rbac.authorization.k8s.io/rabbitmq-cluster-operator-rolebinding created
deployment.apps/rabbitmq-cluster-operator created
$ kubectl get pod -n rabbitmq-system
NAME READY STATUS RESTARTS AGE
rabbitmq-cluster-operator-6657f4b69f-z5xpn 1/1 Running 0 4m55s
$
4. Using RabbitMQ Cluster Kubernetes Operator
- https://www.rabbitmq.com/kubernetes/operator/using-operator.html
a. Confirm Service Availability
ensure that RabbitmqCluster Custom Resource is deployed to your Kubernetes cluster and is available.
$ k get customresourcedefinitions.apiextensions.k8s.io | egrep 'NAME|rabbitmq'
NAME CREATED AT
rabbitmqclusters.rabbitmq.com 2021-01-08T06:39:44Z
$
b. Apply Pod Security Policies
- pod security policies are disabled by default.
c. Create a RabbitMQ Instance
- https://www.rabbitmq.com/kubernetes/operator/using-operator.html#update
$ k create ns rabbitmq-cluster
namespace/rabbitmq-cluster created
$ vi rabbitmq-cluster-aicc.yaml
apiVersion: rabbitmq.com/v1beta1
kind: RabbitmqCluster
metadata:
name: aicc
namespace: rabbitmq-cluster
spec:
replicas: 3 # Odd numbers (1, 3, 5, 7, and so on)
service:
type: NodePort
terminationGracePeriodSeconds: 60 # default: 604800 (1 week)
resources:
limits:
cpu: "1" # default: 2 (2000 millicores)
memory: 1Gi # default: 2Gi, The RabbitMQ high-water mark is set to 0.4 times the memory limit. (Recommend)
requests: # Recommend: request and limit are equaal
cpu: "1"
memory: 1Gi
persistence:
storage: 10Gi # default: 10Gi
$ k apply -f rabbitmq-cluster-aicc.yaml
rabbitmqcluster.rabbitmq.com/aicc created
$ kubectl get all -l app.kubernetes.io/name=aicc -n rabbitmq-cluster
NAME READY STATUS RESTARTS AGE
pod/aicc-server-0 1/1 Running 0 7m52s
pod/aicc-server-1 1/1 Running 0 98s
pod/aicc-server-2 1/1 Running 0 66s
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
service/aicc NodePort 10.102.98.251 <none> 15672:31853/TCP,5672:32163/TCP 7m51s
service/aicc-nodes ClusterIP None <none> 4369/TCP,25672/TCP 7m51s
NAME READY AGE
statefulset.apps/aicc-server 3/3 7m51s
[iap@iap01 rabbitmq]$ k get pvc -n rabbitmq-cluster
NAME STATUS VOLUME CAPACITY ACCESS MODES STORAGECLASS AGE
persistence-aicc-server-0 Bound pvc-c279d080-df31-4cb8-b5f9-3a36092ee999 10Gi RWO nfs-sc-iap 3m24s
persistence-aicc-server-1 Bound pvc-3e1030b9-af4d-4da8-b539-239e4e5588d3 10Gi RWO nfs-sc-iap 3m1s
persistence-aicc-server-2 Bound pvc-48ba4f52-8868-48d6-902c-37d276dd51f7 10Gi RWO nfs-sc-iap 2m30s
$# 5672 : RabbitMQ service port
# 15672 : RabbitMQ management port (Web)
d. Verify the Instance is Running
$ instance=aicc
$ namespace=rabbitmq-cluster
$ username=$(kubectl get secret ${instance}-default-user -n ${namespace} -o jsonpath="{.data.username}" | base64 --decode)
$ password=$(kubectl get secret ${instance}-default-user -n ${namespace} -o jsonpath="{.data.password}" | base64 --decode)
$ service=${instance}
$ kubectl run perf-test -n ${namespace} --image=pivotalrabbitmq/perf-test -- --uri "amqp://${username}:${password}@${service}"
pod/perf-test created
$ k logs -l run=perf-test -n rabbitmq-cluster -f
id: test-013541-692, starting consumer #0
id: test-013541-692, starting consumer #0, channel #0
id: test-013541-692, starting producer #0
id: test-013541-692, starting producer #0, channel #0
id: test-013541-692, time: 1.000s, sent: 7208 msg/s, received: 3044 msg/s, min/median/75th/95th/99th consumer latency: 691/131287/225151/283005/284118 µs
id: test-013541-692, time: 2.003s, sent: 12295 msg/s, received: 4781 msg/s, min/median/75th/95th/99th consumer latency: 322820/633978/771807/873029/876531 µs
f. Connect to RabbitMQ management UI
$ k get svc -n rabbitmq-cluster -o jsonpath='{.items[0]..spec.ports[?(@.name=="management")].nodePort}'
31853
$
5. Monitoring RabbitMQ in Kubernetes
- 공통으로 사용할 kube-prometheus-stack을 구성하고 다수의 오픈소스 SW를 수용(Metric 수집, Grafana dashboard 제공)하는 방식으로 구성
a. Install kube prometheus stack
$ helm repo add prometheus-community https://prometheus-community.github.io/helm-charts
$ helm repo update
$ helm inspect values prometheus-community/kube-prometheus-stack --version 12.8.0 > kube-prometheus-stack.values
$ vi kube-prometheus-stack.values
…
nodeExporter:
enabled: false # default: true, kube-prometheus-stack-1608276926 (gpu-monitor)에서 이미 node-exporter pod를 기동 중
…
prometheus:
…
nodePort: 30091 # default:30090, kube-prometheus-stack-1608276926 (gpu-monitor)에서 30090을 사용 중
type: NodePort # default: ClusterIP
…
prometheusSpec:
…
serviceMonitorSelectorNilUsesHelmValues: false # default: true
…
$
b. Monitor RabbitMQ Using Scraping Annotations
- https://www.rabbitmq.com/kubernetes/operator/operator-monitoring.html#prom-annotations
- Prometheus can be configured to scrape all Pods with the prometheus.io/scrape: true annotation.
- The Prometheus Helm chart, for example, is configured by default to scrape all pods in a cluster with this annotation.
$ vi kube-prometheus-stack.values
…
additionalScrapeConfigs: []
additionalScrapeConfigs: # append bellow lines
- job_name: kubernetes-scraping-annotations-pods
honor_timestamps: true
scrape_interval: 1m
scrape_timeout: 30s
metrics_path: /metrics
scheme: http
kubernetes_sd_configs:
- role: pod
relabel_configs:
- source_labels: [__meta_kubernetes_pod_annotation_prometheus_io_scrape]
separator: ;
regex: "true"
replacement: $1
action: keep
- source_labels: [__address__, __meta_kubernetes_pod_annotation_prometheus_io_port]
separator: ;
regex: ([^:]+)(?::\d+)?;(\d+)
target_label: __address__
replacement: $1:$2
action: replace
…
$ helm install prometheus-community/kube-prometheus-stack --create-namespace --namespace rabbitmq-monitor \
--generate-name --values kube-prometheus-stack.values --version 12.8.0 # --debug
NAME: kube-prometheus-stack-1610004953
LAST DEPLOYED: Thu Jan 7 16:36:02 2021
NAMESPACE: rabbitmq-monitor
STATUS: deployed
REVISION: 1
NOTES:
kube-prometheus-stack has been installed. Check its status by running:
kubectl --namespace rabbitmq-monitor get pods -l "release=kube-prometheus-stack-1610004953"
Visit https://github.com/prometheus-operator/kube-prometheus for instructions on how to create & configure Alertmanager and Prometheus instances using the Operator.
$
$ k describe pod aicc-server-0 -n rabbitmq-cluster | grep "prometheus.io"
Annotations: prometheus.io/port: 15692
prometheus.io/scrape: true
$
c. Connect to Prometheus UI
- http://14.52.244.136:30091/
$ k get svc kube-prometheus-stack-1611-prometheus -n rabbitmq-monitor -o jsonpath='{.spec.ports[?(@.name=="web")].nodePort}'
30091
$
d. Import Dashboards to Grafana
- https://www.rabbitmq.com/prometheus.html#grafana-configuration
$ k patch service kube-prometheus-stack-1611040530-grafana -n rabbitmq-monitor -p '{ "spec": { "type": "NodePort" } }'
service/kube-prometheus-stack-1611040530-grafana patched
$- http://14.52.244.136:30101/
username: admin
password:
$ grep adminPassword kube-prometheus-stack.values
adminPassword: prom-operator
$ k get svc -n rabbitmq-monitor | grep grafana
kube-prometheus-stack-1611040530-grafana NodePort 10.108.200.170 <none> 80:30101/TCP 14m
$- "+ > Import > Import via grafana.com, input '10991', load > import"
'Kubernetes > Message Broker' 카테고리의 다른 글
| Strimzi #4 Performance test (0) | 2021.09.22 |
|---|---|
| Strimzi #3 Monitoring (0) | 2021.09.22 |
| Strimzi #2 Configuration (0) | 2021.09.21 |
| Strimzi #1 Overview (0.19.0) (0) | 2021.09.21 |
댓글