1. 개요
- MongDB
✓ MongoDB는 크로스 플랫폼 도큐먼트 지향 데이터베이스 시스템이다.
✓ NoSQL 데이터베이스로 분류되는 MongoDB는 JSON과 같은 동적 스키마형 도큐먼트들을 선호한다.
- MongoDB Enterprise server vs Community server
✓ MongoDB Enterprise server는 아래와 같은 추가적인 기능을 제공한다.
▷ in-memory storage engine for high throughput and low latency
▷ advanced security features like LDAP and Kerberos access controls
▷ encryption for data at rest.
- MongoDB Enterprise Kubernetes Operator vs Community Kubernetes Operator
✓ The MongoDB Enterprise Kubernetes Operator works together with MongoDB Cloud Manager or Ops Manager.
✓ Ops Manager is the management platform that makes it easy to deploy, monitor, back up, and scale MongoDB on your own infrastructure.
✓ Cloud Manager eliminates the guesswork in running MongoDB. Easily monitor your databases, automate administration tasks, and leverage cloud backups for your self-managed deployments.
- MongoDB Community Kubernetes Operator 기능들
✓ Create replica sets
✓ Upgrade and downgrade MongoDB server version
✓ Scale replica sets up and down
✓ Read from and write to the replica set while scaling, upgrading, and downgrading. These operations are done in an "always up" manner.
✓ Report MongoDB server state via the MongoDBCommunity resource status field
✓ Use any of the available Docker MongoDB images
✓ Connect to the replica set from inside the Kubernetes cluster (no external connectivity)
✓ Secure client-to-server and server-to-server connections with TLS
✓ Create users with SCRAM authentication
✓ Create custom roles
- MongoDB Community Kubernetes Operator 한계
Replica sets만 지원하고 Sharding을 지원 하지 않는다. 이런 내용이 공식 문서에는 언급되어 있지 않다.
MongoDB를 사용하면서 Sharding 할 수 없다면 무슨 의미가 있을까!!! 사용 포기
MongoDB Enterise Kuberneters Operator는 Sharding을 지원한다.
- MongoDB Replica sets vs Sharded cluster
✓ https://severalnines.com/database-blog/turning-mongodb-replica-set-sharded-cluster
a. Replica sets
✓ Replica Sets are a great way to replicate MongoDB data across multiple servers and have the database automatically failover in case of server failure.
b. Sharded
✓ Sharding is a way to split data across multiple servers. In a MongoDB Sharded Cluster, the database will handle distribution of the data and dynamically load-balance queries.
MongoDB stores documents in collections. Collections are analogous to tables in relational databases.
✓ Config servers store the meta data for the sharded cluster.
✓ Routers(mongos) are the processes that clients connect to, and queries are then routed to the appropriate shard.
- Kubernetes Operator 종류
2. Environments
- MongoDB Community Kubernetes Operator 0.7.1
- Kubernetes 1.16.15
3. 구성
- MongoDB Community Kubernetes Operator
https://github.com/mongodb/mongodb-kubernetes-operator/blob/master/README.md
a. MongoDB Community Kubernetes Operator 설치
- CRD(Custom Resource Definitions) 다운로드
$ git clone https://github.com/mongodb/mongodb-kubernetes-operator
Cloning into 'mongodb-kubernetes-operator'...
remote: Enumerating objects: 8528, done.
remote: Counting objects: 100% (401/401), done.
remote: Compressing objects: 100% (273/273), done.
remote: Total 8528 (delta 194), reused 219 (delta 103), pack-reused 8127
Receiving objects: 100% (8528/8528), 19.17 MiB | 12.46 MiB/s, done.
Resolving deltas: 100% (5144/5144), done.
$ cd mongodb-kubernetes-operator
$
- CRD(Custom Resource Definitions) 설치
$ kubectl apply -f config/crd/bases/mongodbcommunity.mongodb.com_mongodbcommunity.yaml
customresourcedefinition.apiextensions.k8s.io/mongodbcommunity.mongodbcommunity.mongodb.com created
$ kubectl get crd/mongodbcommunity.mongodbcommunity.mongodb.com
NAME CREATED AT
mongodbcommunity.mongodbcommunity.mongodb.com 2021-11-02T00:53:01Z
$
- role과 role-bindings 설치
MongoDB Operator를 설치할 namespace를 생성한다.
Kustomize를 이용한 리소스를 설치하기 때문에 kubectl 명령어 실행 시 -k(--kustomize) 옵션을 사용한다.
$ kubectl create ns mongodb-operator
namespace/mongodb-operator created
$ kubectl apply -k config/rbac/ -n mongodb-operator
serviceaccount/mongodb-database created
serviceaccount/mongodb-kubernetes-operator created
role.rbac.authorization.k8s.io/mongodb-database created
role.rbac.authorization.k8s.io/mongodb-kubernetes-operator created
rolebinding.rbac.authorization.k8s.io/mongodb-database created
rolebinding.rbac.authorization.k8s.io/mongodb-kubernetes-operator created
$
- Operator 설치
MongoDB Operator와 리소스(MongoDB)를 다른 namespace에 설치할 경우 추가 작업을 해야 한다.
✓ manager.yaml의 WATCH_NAMESPACE 값을 "*"로 변경
✓ role_binding.yaml의 namespace 값을 MongoDB Operator가 설치될 namespace로 변경
$ vi config/manager/manager.yaml
...
- name: WATCH_NAMESPACE
value: "*"
# valueFrom:
# fieldRef:
# fieldPath: metadata.namespace
...
$ vi deploy/clusterwide/role_binding.yaml
...
- kind: ServiceAccount
namespace: mongodb-operator
# namespace: <your-namespace>
...
$ kubectl apply -f deploy/clusterwide
clusterrole.rbac.authorization.k8s.io/mongodb-kubernetes-operator created
clusterrolebinding.rbac.authorization.k8s.io/mongodb-kubernetes-operator created
$ kubectl apply -f config/manager/manager.yaml -n mongodb-operator
deployment.apps/mongodb-kubernetes-operator created
$
$ kubectl get pod -n mongodb-operator
NAME READY STATUS RESTARTS AGE
mongodb-kubernetes-operator-7bdc9f6859-mmtks 1/1 Running 8 5m
$ kubectl logs mongodb-kubernetes-operator-7bdc9f6859-mmtks -n mongodb-operator -f
Running ./manager
2021-11-02T02:06:23.568Z INFO manager/main.go:71 Watching all namespaces
I1102 02:06:24.618955 9 request.go:665] Waited for 1.03313107s due to client-side throttling, not priority and fairness, request: GET:https://10.96.0.1:443/apis/apiextensions.k8s.io/v1?timeout=32s
2021-11-02T02:06:26.227Z INFO manager/main.go:91 Registering Components.
2021-11-02T02:06:26.227Z INFO manager/main.go:104 Starting the Cmd.
b. Replica set MongoDB 생성
- MongoDB Operator와 리소스(MongoDB)를 다른 namespace에 설치할 경우 Operator가 해당 namespace를 볼 수 있도록 관련 리소스를 생성한다.
$ kubectl apply -k config/rbac -n yoosung-jeon
serviceaccount/mongodb-database created
serviceaccount/mongodb-kubernetes-operator created
role.rbac.authorization.k8s.io/mongodb-database created
role.rbac.authorization.k8s.io/mongodb-kubernetes-operator created
rolebinding.rbac.authorization.k8s.io/mongodb-database created
rolebinding.rbac.authorization.k8s.io/mongodb-kubernetes-operator created
$
- MongoDB yaml 파일 작성
✓ yaml 파일 설정 항목
▷ metadata.name: MongoDB database 리소스 명
▷ spec.verson: 설치할 MongoDB
▷ springData.password: my-user 사용자의 암호
✓ DB 사용자 추가
▷ Secret 리소스에 사용자 암호 생성
▷ MongoDBCommunity 리소스의 spec.users에 사용자 정보 추가 (ex. my-user)
$ cp config/samples/mongodb.com_v1_mongodbcommunity_cr.yaml test-mongodb.yaml
$ vi test-mongodb.yaml
---
apiVersion: mongodbcommunity.mongodb.com/v1
kind: MongoDBCommunity
metadata:
name: test-mongodb
# name: example-mongodb
spec:
members: 3
type: ReplicaSet
version: "5.0.3"
# version: "4.2.6"
security:
authentication:
modes: ["SCRAM"]
users:
- name: my-user
db: admin
passwordSecretRef: # a reference to the secret that will be used to generate the user's password
name: my-user-password
roles:
- name: clusterAdmin
db: admin
- name: userAdminAnyDatabase
db: admin
scramCredentialsSecretName: my-scram
additionalMongodConfig:
storage.wiredTiger.engineConfig.journalCompressor: zlib
---
apiVersion: v1
kind: Secret
metadata:
name: my-user-password
type: Opaque
stringData:
password: mong12#$
$
- Replica set MongoDB 설치
$ k apply -f test-mongodb.yaml -n yoosung-jeon
mongodbcommunity.mongodbcommunity.mongodb.com/test-mongodb created
secret/my-user-password created
$
- MongoDB 설치 확인
$ k get mongodbcommunity.mongodbcommunity.mongodb.com -n yoosung-jeon
NAME PHASE VERSION
test-mongodb Running 5.0.3
$ k get pod -l app=test-mongodb-svc -n yoosung-jeon -o wide
NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES
test-mongodb-0 2/2 Running 0 2m56s 10.244.3.252 iap11 <none> <none>
test-mongodb-1 2/2 Running 0 2m21s 10.244.2.107 iap12 <none> <none>
test-mongodb-2 2/2 Running 0 99s 10.244.6.121 iap13 <none> <none>
$ k get pvc -l app=test-mongodb-svc -n yoosung-jeon
NAME STATUS VOLUME CAPACITY ACCESS MODES STORAGECLASS AGE
data-volume-test-mongodb-0 Bound pvc-5bfaaa7f-0581-45bc-bbe1-2190297bd164 10G RWO nfs-sc-iap 45m
data-volume-test-mongodb-1 Bound pvc-46f2c280-de47-4b0d-b632-f45bcac5673f 10G RWO nfs-sc-iap 3m37s
data-volume-test-mongodb-2 Bound pvc-4185244d-4bbc-4b93-bc52-8cbe644992c3 10G RWO nfs-sc-iap 2m56s
logs-volume-test-mongodb-0 Bound pvc-39850c2a-810d-4b20-9efa-4c55ff103ee4 2G RWO nfs-sc-iap 45m
logs-volume-test-mongodb-1 Bound pvc-83c00edf-394c-4b66-b2a1-0fe20eb15030 2G RWO nfs-sc-iap 3m37s
logs-volume-test-mongodb-2 Bound pvc-21d33a1b-290c-4758-af07-da2cf5eb092b 2G RWO nfs-sc-iap 2m56s
$ k get svc test-mongodb-svc -n yoosung-jeon
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
test-mongodb-svc ClusterIP None <none> 27017/TCP 5m
$
- MongoDB 접속 정보 얻기
Secret 명: <metadata.name>-<auth-db>-<username>
$ k get secrets -n yoosung-jeon test-mongodb-admin-my-user -o json | jq -r '.data | with_entries(.value|= @base64d)'
{
"connectionString.standard": "mongodb://my-user:mong12%23%24@test-mongodb-0.test-mongodb-svc.yoosung-jeon.svc.cluster.local:27017,test-mongodb-1.test-mongodb-svc.yoosung-jeon.svc.cluster.local:27017,test-mongodb-2.test-mongodb-svc.yoosung-jeon.svc.cluster.local:27017/admin?ssl=false",
"connectionString.standardSrv": "mongodb+srv://my-user:mong12%23%24@test-mongodb-svc.yoosung-jeon.svc.cluster.local/admin?ssl=false",
"password": "mong12#$",
"username": "my-user"
}
$
- MongoDB 접속하기
MongoDB Pod에 접속 후 mongosh 명령어로 위에서 조회한 접속 정보를 사용하여 MongoDB에 접속한다.
$ k exec test-mongodb-0 -n yoosung-jeon -it -- bash
Defaulting container name to mongod.
Use 'kubectl describe pod/test-mongodb-0 -n yoosung-jeon' to see all of the containers in this pod.
groups: cannot find name for group ID 2000
I have no name!@test-mongodb-0:/$ mongosh "mongodb+srv://my-user:mong12%23%24@test-mongodb-svc.yoosung-jeon.svc.cluster.local/admin?ssl=false"
Warning: Could not access file: EACCES: permission denied, mkdir '/.mongodb'
Current Mongosh Log ID: 6180eb549e7dfcc2b75410ff
Connecting to: mongodb+srv://<credentials>@test-mongodb-svc.yoosung-jeon.svc.cluster.local/admin?ssl=false
Using MongoDB: 5.0.3
Using Mongosh: 1.1.0
For mongosh info see: https://docs.mongodb.com/mongodb-shell/
To help improve our products, anonymous usage data is collected and sent to MongoDB periodically (https://www.mongodb.com/legal/privacy-policy).
You can opt-out by running the disableTelemetry() command.
------
The server generated these startup warnings when booting:
2021-11-02T05:03:15.569+00:00: You are running on a NUMA machine. We suggest launching mongod like this to avoid performance problems: numactl --interleave=all mongod [other options]
------
Error: Could not open history file.
REPL session history will not be persisted.
test-mongodb [primary] admin>
c. Secure MongoDB 구성
- Self-signed Certificate 생성
✓ Cert-manager(v 0.11.0)를 이용하여 self-signed certificate를 생성한다.
✓ commonName은 replica set 멤버를 모두 포함하는 wildcard common name으로 설정한다.
형식: *.<metadata.name of the MongoDB resource>-svc.<namespace>.svc.cluster.local
생성된 replica set 멤버의 도메인 명:
test-mongodb-0.test-mongodb-svc.yoosung-jeon.svc.cluster.local
test-mongodb-1.test-mongodb-svc.yoosung-jeon.svc.cluster.local
test-mongodb-2.test-mongodb-svc.yoosung-jeon.svc.cluster.local
$ vi mongodb-certificate.yaml
apiVersion: cert-manager.io/v1alpha2
kind: ClusterIssuer
metadata:
name: self-signed-issuer
spec:
selfSigned: {}
---
apiVersion: cert-manager.io/v1alpha2
kind: Certificate
metadata:
name: test-mongodb-certificate
spec:
isCA: true
commonName: "*.test-mongodb-svc.yoosung-jeon.svc.cluster.local"
secretName: test-mongodb-tls
issuerRef:
name: self-signed-issuer
kind: ClusterIssuer
group: cert-manager.io
$ kubectl apply -f mongodb-certificate.yaml
clusterissuer.cert-manager.io/self-signed-issuer created
certificate.cert-manager.io/test-mongodb-certificate created
$ kubectl get secret -n yoosung-jeon test-mongodb-tls
NAME TYPE DATA AGE
test-mongodb-tls kubernetes.io/tls 3 2m54s
$
- CA(Certificae Authority) 포함하는 ConfigMap 생성
$ k get secret -n yoosung-jeon test-mongodb-tls -o jsonpath='{.data.ca\.crt}' | base64 --decode > ca.crt
$ cat ca.crt
-----BEGIN CERTIFICATE-----
MIIDUzCCAjugAwIBAgIQHb6J005KSSvmzUFgzaXoHDANBgkqhkiG9w0BAQsFADBT
MRUwEwYDVQQKEwxjZXJ0LW1hbmFnZXIxOjA4BgNVBAMMMSoudGVzdC1tb25nb2Ri
LXN2Yy55b29zdW5nLWplb24uc3ZjLmNsdXN0ZXIubG9jYWwwHhcNMjExMTAyMTAz
NjAwWhcNMjIwMTMxMTAzNjAwWjBTMRUwEwYDVQQKEwxjZXJ0LW1hbmFnZXIxOjA4
BgNVBAMMMSoudGVzdC1tb25nb2RiLXN2Yy55b29zdW5nLWplb24uc3ZjLmNsdXN0
ZXIubG9jYWwwggEiMA0GCSqGSIb3DQEBAQUAA4IBDwAwggEKAoIBAQDNfwF/m4up
Zby7L5ThaGEMuLkxUUyjtpXSUW5Nh+leHv0eYfSzGB0g07FBiCqg9ElM484cRNhN
3c3ZZ7X5IuQ3hiQJbIs8P2TBJrbJndazJ0+hKI7G74rIYzbsoDGH9/6ZXptwPVZv
Qlm+sXOh/U3lUf27FLpsxihkbruJ8yG3ctBBgH1FAsfSv8hLlxvwVSi6oo6Ya0ii
YRusHqB2lXVsx0KujGxvWqOboFiRSj5gM1G5QgeJKShfO5NmCkdE9E+poo9W2qWm
3RDBE6YYZEm2EE6EaQWrmbLobetcDC5aHuxdizw+TmSCndBVueJCj/OWplE828Jy
LgNFtd0MA41TAgMBAAGjIzAhMA4GA1UdDwEB/wQEAwICpDAPBgNVHRMBAf8EBTAD
AQH/MA0GCSqGSIb3DQEBCwUAA4IBAQCPV5P1sSJpHtoElqXDvRS4J9rQTtkW+RJn
VEljOLRXVU+WfP4jD7XVHN3trrrOz38tPGUkxJHyOLNS7qt6Ocm5sY9br8Ug9AYw
CstWZE9XpANpcuCdZxJEoptu7YYovJfN3+48RFfDjXn79OCIKNUe2Wc3rn8/ApQd
pmazR8F4I8TTwk5xQgDsw0azHU7+kN+20/xRahDRoyk8Qew28FUUA4x4uvmdTv1g
N+FRh73TxvVDVadcy7sZB3Jvlq6XWuACJPG9sMFoQKSQ1BWFLXEOoxk3M7KL5Dtk
aGOk+vq4mNnaW303LP++SxXPLilxu8iSf2wmPz7+LubMCNj8RM4y
-----END CERTIFICATE-----
$ kubectl create configmap test-mongodb-ca-configmap --from-file=ca.crt -n yoosung-jeon
configmap/test-mongodb-ca-configmap created
$
- 생성된 MongoDB 리소스 변경
TLS를 사용하도록 security 절을 추가한다. certificateKeySecretRef와 caConfigMapRef의 값은 위에서 설정한 것을 지정한다.
TLS 적용 시 MongoDB Community Kubernetes Operator가 비정상 종료가 되어 적용할 수 없었다.
$ vi test-mongodb-tls.yaml
...
security:
tls:
enabled: true
certificateKeySecretRef:
name: test-mongodb-tls
caConfigMapRef:
name: test-mongodb-ca-configmap
...
$ kubectl apply -f test-mongodb-tls.yaml -n yoosung-jeon
...
'Kubernetes > NoSQL' 카테고리의 다른 글
| MongoDB Sharded - root 암호 변경 시 고려사항 (0) | 2021.12.06 |
|---|---|
| MongoDB Sharded by Bitnami (0) | 2021.11.03 |
| Redis - corrupted cluster config file (0) | 2021.10.02 |
| Elastic Cloud on Kubernetes (ECK) (0) | 2021.09.22 |
| Redis cluster (0) | 2021.09.22 |
댓글