Velero와 restic - 'Error checking repository for stale locks'

1. 개요

- velero backup 실행 시 발생된 에러의 원인을 파악하고 조치한다.

 

 

2. 환경

- Velero 1.7 & restic 0.12.0

- MinIO 2021-11-09T03:21:45Z

- Kubernetes 1.16.15

 

 

3. 문제점

- velero Pod에서 생성되어 있는 restic repository를 주기적으로 점검하는 과정에서 에러가 발생된다.

   상세 에러 메시지:

      Fatal: unable to open config file: Stat: The specified key does not exist.

      Is there a repository at the following location?

      s3:https://api.acp.kt.co.kr:9002/k8s-ext/restic/geumseong-yoon: exit status 1

$ k logs velero-8996dd469-bbq2m -n velero -f
...
time="2021-12-15T08:04:23Z" 
   level=error 
   msg="Error checking repository for stale locks" 
   controller=restic-repo 
   error="error running command=restic unlock --repo=s3:https://api.acp.kt.co.kr:9002/k8s-ext/restic/geumseong-yoon --password-file=/tmp/credentials/velero/velero-restic-credentials-repository-password --cacert=/tmp/cacert-default182731332 --cache-dir=/scratch/.cache/restic, 
          stdout=, 
          stderr=Fatal: unable to open config file: Stat: The specified key does not exist.\nIs there a repository at the following location?\ns3:https://api.acp.kt.co.kr:9002/k8s-ext/restic/geumseong-yoon\n: exit status 1
         " 
   error.file="/go/src/github.com/vmware-tanzu/velero/pkg/restic/repository_manager.go:276" 
   error.function="github.com/vmware-tanzu/velero/pkg/restic.(*repositoryManager).exec" 
logSource="pkg/controller/restic_repository_controller.go:144" name=geumseong-yoon-default-bk27m namespace=velero
...

 

 

4. 원인

- Velero Console에서 'geumseong-yoon' restic repository를 임의로 삭제되어 존재하기 않는다.

 

- 하지만 'geumseong-yoon' restic repository 정보는 쿠버네티스내에 존재하고 있다.

$ k get resticrepository -n velero geumseong-yoon-default-bk27m
NAME                           AGE
geumseong-yoon-default-bk27m   5h47m
$ k describe resticrepository -n velero geumseong-yoon-default-bk27m
Name:         geumseong-yoon-default-bk27m
Namespace:    velero
Labels:       velero.io/storage-location=default
              velero.io/volume-namespace=geumseong-yoon
Annotations:  <none>
API Version:  velero.io/v1
Kind:         ResticRepository
Metadata:
  Creation Timestamp:  2021-12-15T03:20:21Z
  Generate Name:       geumseong-yoon-default-
  Generation:          3
  Resource Version:    935411141
  Self Link:           /apis/velero.io/v1/namespaces/velero/resticrepositories/geumseong-yoon-default-bk27m
  UID:                 4b0dcbe0-3c2d-4bb3-9fa8-8ee5d60b3ddc
Spec:
  Backup Storage Location:  default
  Maintenance Frequency:    168h0m0s
  Restic Identifier:        s3:https://api.acp.kt.co.kr:9002/k8s-ext/restic/geumseong-yoon
  Volume Namespace:         geumseong-yoon
Status:
  Last Maintenance Time:  2021-12-15T03:20:24Z
  Phase:                  Ready
Events:                   <none>
$

 

 

5. 조치사항

- 쿠버네티스에서 'geumseong-yoon' restic repository를 삭제하고 velero Pod를 재기동한다.

   추후 geumseong-yoon' 네임스페이스를 Velero backup 할 때 restic repository는 다시 생성된다. 

$ k delete resticrepository geumseong-yoon-default-bk27m -n velero
resticrepository.velero.io "geumseong-yoon-default-bk27m" deleted
$
$ k rollout restart deployment velero -n velero
deployment.apps/velero restarted
$