MongoDB Sharded - root 암호 변경 시 고려사항

1. 개요

- root 계정 암호 변경 시 고려 사항

   MongoDB Sharded를 사용하는 환경에서는 root 계정의 암호를 변경할 경우 암호를 저장하고 있는 secret도 같이 변경해야 한다.

- MongoDB sharded helm chart by Bitnami

   This chart bootstraps a  deployment on a  cluster using the  package manager.

 

 

2. 환경

- bitnami/mongodb-sharded 3.9.14

- MongoDB 4.4.10

 

 

3. root 계정 암호 변경 절차

a. 암호 변경

- MongoDB에 접속하여 root 계정의 암호를 'mongo'로 변경한다.

$ kubectl run -n ontact bizcollabo-mongodb-mongodb-sharded-client --rm --tty -i --restart='Never' --image docker.io/bitnami/mongodb-sharded:4.4.10-debian-10-r15 --command -- bash
I have no name!@bizcollabo-mongodb-mongodb-sharded-client:/$ mongo admin --host bizcollabo-mongodb-mongodb-sharded -u root -p
MongoDB shell version v4.4.10
Enter password:
connecting to: mongodb://bizcollabo-mongodb-mongodb-sharded:27017/admin?compressors=disabled&gssapiServiceName=mongodb
Implicit session: session { "id" : UUID("b44ce100-668f-4265-bbde-57431087406e") }
MongoDB server version: 4.4.10
mongos> db.changeUserPassword('root', 'mongo')
mongos> exit
bye
I have no name!@bizcollabo-mongodb-mongodb-sharded-client:/$

 

b. Kubernetes Secret 변경

- 변경된 암호 문자를 base64 형식으로 인코딩한다.

$ echo mongo | base64
bW9uZ28K
$

 

- 인코딩 문자를 MongoDB sharded가 실행 중인 네임스페이스의 관련 secret에 'mongodb-root-password' 값에 반영한다.

   관련 secret은 MongoDB Pod들이 기동 할 때 참조하므로 mongos를 재기동해서 정상적으로 변경되었는지 확인한다.

$ k edit secrets -n ontact bizcollabo-mongodb-mongodb-sharded
apiVersion: v1
data:
  mongodb-replica-set-key: YXlYajNaUTBZTA==
  mongodb-root-password: bW9uZ28K
…
$ 
$ k rollout restart deployment bizcollabo-mongodb-mongodb-sharded-mongos -n yoosung-jeon
deployment.apps/bizcollabo-mongodb-mongodb-sharded-mongos restarted
$

 

 

4. 트러블슈팅

- 문제점

   root 계정의 암호를 변경 후 secret에 반영하지 않을 경우 MongoDB Pod들이 재 기동시 Liveness probe 오류가 발생된다.

$ k describe pod bizcollabo-mongodb-mongodb-sharded-mongos-5c5595f7c7-vt9n7 -n ontact | grep Events -A30
Events:
  Type     Reason     Age                From               Message
  ----     ------     ----               ----               -------
  Normal   Scheduled  36m                default-scheduler  Successfully assigned ontact/bizcollabo-mongodb-mongodb-sharded-mongos-5c5595f7c7-vt9n7 to acp-worker02
  Normal   Killing    34m                kubelet            Container mongos failed liveness probe, will be restarted
  Normal   Pulled     33m (x2 over 36m)  kubelet            Container image "docker.io/bitnami/mongodb-sharded:4.4.10-debian-10-r15" already present on machine
  Normal   Created    33m (x2 over 36m)  kubelet            Created container mongos
  Normal   Started    33m (x2 over 36m)  kubelet            Started container mongos
  Warning  Unhealthy  32m (x7 over 35m)  kubelet            Liveness probe failed: MongoDB shell version v4.4.10
connecting to: mongodb://127.0.0.1:27017/?compressors=disabled&gssapiServiceName=mongodb
Error: couldn't connect to server 127.0.0.1:27017, connection attempt failed: SocketException: Error connecting to 127.0.0.1:27017 :: caused by :: Connection refused :
connect@src/mongo/shell/mongo.js:374:17
@(connect):2:6
exception: connect failed
exiting with code 1
  Warning  BackOff    6m10s (x38 over 18m)  kubelet  Back-off restarting failed container
  Warning  Unhealthy  76s (x97 over 35m)    kubelet  Readiness probe failed: MongoDB shell version v4.4.10
connecting to: mongodb://127.0.0.1:27017/?compressors=disabled&gssapiServiceName=mongodb
Error: couldn't connect to server 127.0.0.1:27017, connection attempt failed: SocketException: Error connecting to 127.0.0.1:27017 :: caused by :: Connection refused :
connect@src/mongo/shell/mongo.js:374:17
@(connect):2:6
exception: connect failed
exiting with code 1
$

 

- 원인 파악

   이전 암호가 저장된 secret을 참조해서 MongoDB에 접속 시발생되는 에러 메시지는 위 Pod의 이벤트 내용과 동일한다.

$ k get secrets -n ontact bizcollabo-mongodb-mongodb-sharded -o jsonpath="{.data.mongodb-root-password}" | base64 --decode
CrxF5VsxfT
$ k exec bizcollabo-mongodb-mongodb-sharded-configsvr-0 -n ontact -it -- bash
I have no name!@bizcollabo-mongodb-mongodb-sharded-configsvr-0:/$ mongo admin --host bizcollabo-mongodb-mongodb-sharded -u root -p
MongoDB shell version v4.4.10
Enter password:
connecting to: mongodb://bizcollabo-mongodb-mongodb-sharded:27017/admin?compressors=disabled&gssapiServiceName=mongodb
Error: Authentication failed. :
connect@src/mongo/shell/mongo.js:374:17
@(connect):2:6
exception: connect failed
exiting with code 1
I have no name!@bizcollabo-mongodb-mongodb-sharded-configsvr-0:/$

 

   MongoDB Pod들은 기동 시 MongoDB root 암호를 얻기 위해서 secret을 참조하는 것을 확인할 수 있다.

$ k describe deployments.apps bizcollabo-mongodb-mongodb-sharded-mongos -n ontact | grep Environment -A5
    Environment:
      MONGODB_ENABLE_NUMACTL:      no
      BITNAMI_DEBUG:               false
      MONGODB_SHARDING_MODE:       mongos
      MONGODB_MAX_TIMEOUT:         120
      MONGODB_ROOT_PASSWORD:       <set to the key 'mongodb-root-password' in secret 'bizcollabo-mongodb-mongodb-sharded'>    Optional: false
$
$ k describe statefulsets.apps bizcollabo-mongodb-mongodb-sharded-configsvr -n ontact | grep MONGODB_ROOT_PASSWORD
      MONGODB_ROOT_PASSWORD:       <set to the key 'mongodb-root-password' in secret 'bizcollabo-mongodb-mongodb-sharded'>    Optional: false
$ k describe statefulsets.apps bizcollabo-mongodb-mongodb-sharded-shard0-data -n ontact | grep MONGODB_ROOT_PASSWORD
      MONGODB_ROOT_PASSWORD:       <set to the key 'mongodb-root-password' in secret 'bizcollabo-mongodb-mongodb-sharded'>    Optional: false
$