2020.06.08
1. 환경
- Kubeflow 1.0.2
- K8s Server Version: v1.15.12
- CentOS Linux release 7.8
- Dex (https://github.com/dexidp/dex)
Dex is an identity service that uses OpenID Connect to drive authentication for other apps.
2. Kubeflow Install
- https://www.kubeflow.org/docs/started/k8s/kfctl-istio-dex/
a. Notes on the configuration file
a-1. Disabling istio installation If your Kubernetes cluster has an existing Istio installation
- 설치된 Istio 제거
$ kubectl label namespace default istio-injection=disabled --overwrite
namespace/default labeled
$ istioctl manifest generate --set profile=demo | kubectl delete -f -
…
$- 설치된 Istio 유지
If your Kubernetes cluster has an existing Istio installation you may choose to not install Istio by removing the applications istio-crds and istio-install in the configuration file kfctl_istio_dex.v1.0.2.yaml.
a-2. Istio configuration for trustworthy JWTs
This configuration uses Istio version 1.3.1 with SDS enabled, which requires Kubernetes 1.13 or later.
SDS stands for Secret Delivery Service and allows Istio components to receive SSL Certificates by the API.
TS #1 참조
$ sudo vi /etc/kubernetes/manifests/kube-apiserver.yaml
…
- --service-account-key-file=/etc/kubernetes/pki/sa.pub
- --service-account-issuer=kubernetes.default.svc # appended
- --service-account-signing-key-file=/etc/kubernetes/pki/sa.key # appended
…
$
a-3. Default password in static file configuration for Dex
- The configuration file kfctl_istio_dex.v1.0.2.yaml contains a default staticPasswords user
b. Before you start
- This Kubeflow deployment requires a default StorageClass with a dynamic volume provisioner.
$ k get storageclasses.storage.k8s.io --all-namespaces
NAME PROVISIONER AGE
rook-ceph-block-sc-iap rook-ceph.rbd.csi.ceph.com 4d22h
rook-ceph-fs-sc-iap rook-ceph.cephfs.csi.ceph.com 4d22h
rook-ceph-object-bucket-iap ceph.rook.io/bucket 4d22h
$ kubectl patch storageclass rook-ceph-block-sc-iap -p '{"metadata": {"annotations": {"storageclass.kubernetes.io/is-default-class":"true"}}}'
storageclass.storage.k8s.io/rook-ceph-block-sc-iap patched
$ k get storageclasses.storage.k8s.io --all-namespaces
NAME PROVISIONER AGE
rook-ceph-block-sc-iap (default) rook-ceph.rbd.csi.ceph.com 4d22h
rook-ceph-fs-sc-iap rook-ceph.cephfs.csi.ceph.com 4d22h
rook-ceph-object-bucket-iap ceph.rook.io/bucket 4d22h
$
c. Prepare your environment
$ mkdir ~/bin && cd ~/bin
$ wget https://github.com/kubeflow/kfctl/releases/download/v1.0.2/kfctl_v1.0.2-0-ga476281_linux.tar.gz
$ tar -xvf kfctl_v1.0.2-0-ga476281_linux.tar.gz
./kfctl
$
echo ‘export PATH=$PATH:~/bin’ >> ~/.bash_profile && source ~/.bash_profile
export CONFIG_URI="https://raw.githubusercontent.com/kubeflow/manifests/v1.0-branch/kfdef/kfctl_istio_dex.v1.0.2.yamlf"
export KF_NAME=my-kubeflow
export BASE_DIR=~/kf_deployments
export KF_DIR=${BASE_DIR}/${KF_NAME}
The value of KF_NAME must consist of lower case alphanumeric characters or ‘-', and must start and end with an alphanumeric character.
c. set up your configuration for later deployment
$ mkdir -p ${KF_DIR} && cd ${KF_DIR}
$ kfctl build -V -f ${CONFIG_URI}
INFO[0000] Downloading https://raw.githubusercontent.com/kubeflow/manifests/v1.0-branch/kfdef/kfctl_istio_dex.v1.0.2.yaml to /tmp/203250136/tmp.yaml filename="utils/k8utils.go:172"
INFO[0000] Downloading https://raw.githubusercontent.com/kubeflow/manifests/v1.0-branch/kfdef/kfctl_istio_dex.v1.0.2.yaml to /tmp/581158231/tmp_app.yaml filename="loaders/loaders.go:71"
INFO[0000] App directory /home/ysjeon71_kubeflow3/kf_deployments/my-kubeflow already exists filename="coordinator/coordinator.go:270"
INFO[0000] Writing KfDef to kfctl_istio_dex.v1.0.2.yaml filename="coordinator/coordinator.go:273"
INFO[0000] No name specified in KfDef.Metadata.Name; defaulting to my-kubeflow based on location of config file: /home/ysjeon71_kubeflow3/kf_deployments/my-kubeflow/kfctl_istio_dex.v1.0.2.yaml. filename="coordinator/coordinator.go:202"
INFO[0000]
****************************************************************
Notice anonymous usage reporting enabled using spartakus
To disable it
If you have already deployed it run the following commands:
cd $(pwd)
kubectl -n ${K8S_NAMESPACE} delete deploy -l app=spartakus
For more info: https://www.kubeflow.org/docs/other-guides/usage-reporting/
****************************************************************
filename="coordinator/coordinator.go:120"
INFO[0000] Creating directory /home/ysjeon71_kubeflow3/kf_deployments/my-kubeflow/.cache filename="kfconfig/types.go:445"
INFO[0000] Fetching https://github.com/kubeflow/manifests/archive/v1.0.2.tar.gz to /home/ysjeon71_kubeflow3/kf_deployments/my-kubeflow/.cache/manifests filename="kfconfig/types.go:493"
INFO[0003] updating localPath to /home/ysjeon71_kubeflow3/kf_deployments/my-kubeflow/.cache/manifests/manifests-1.0.2 filename="kfconfig/types.go:540"
INFO[0003] Fetch succeeded; LocalPath /home/ysjeon71_kubeflow3/kf_deployments/my-kubeflow/.cache/manifests/manifests-1.0.2 filename="kfconfig/types.go:561"
INFO[0003] Processing application: application-crds filename="kustomize/kustomize.go:408"
…
INFO[0003] Processing application: seldon-core-operator filename="kustomize/kustomize.go:408"
$
d. Edit the configuration files
https://www.kubeflow.org/docs/other-guides/kustomize/
e. deploy Kubeflow:
$ export CONFIG_FILE=${KF_DIR}/kfctl_istio_dex.v1.0.2.yaml
$ kfctl apply -V -f ${CONFIG_FILE}
INFO[0000] No name specified in KfDef.Metadata.Name; defaulting to my-kubeflow based on location of config file: /home/ysjeon71_kubeflow3/kf_deployments/my-kubeflow/kfctl_istio_dex.v1.0.2.yaml. filename="coordinator/coordinator.go:202"
…
INFO[0097] Successfully applied application seldon-core-operator filename="kustomize/kustomize.go:209"
INFO[0097] Applied the configuration Successfully! filename="cmd/apply.go:72"
$
f. Accessing Kubeflow
- Dex supports several authentication methods:
✓ Static users, as described above
Default password in static file configuration for Dex
email = admin@kubeflow.org, password = 12341234
✓ Add static users for basic auth
https://v1-0-branch.kubeflow.org/docs/started/k8s/kfctl-istio-dex/#add-static-users-for-basic-auth
✓ LDAP / Active Directory
✓ External Identity Provider (IdP) (for example Google, LinkedIn, GitHub, …)
# Kubeflow 접속
# Port: k get svc/istio-ingressgateway -n istio-system -o=jsonpath={.spec.ports[1].nodePort}
# URL - http://[K8s 노드의 EXTERNAL_IP]:nodePort/
g. Log in with LDAP / Active Directory
- OpenLDAP 1.2.4, phpLDAPAdmin 0.8.0
https://www.kubeflow.org/docs/started/k8s/kfctl-istio-dex/#log-in-with-ldap--active-directory
- Deploy a new LDAP Server
$ mkdir ~/openldap && cd ~/openldap
$ vi openLDAP.yaml
…
$ k apply -f openLDAP.yaml
$ kubectl patch service phpldapadmin-service -n kubeflow -p '{ "spec": { "type": "NodePort" } }'- phpLDAPAdmin UI
# Port: k get svc/phpldapadmin-service -n kubeflow -o=jsonpath={.spec.ports[*].nodePort}
# URL: http://[K8s 노드의 EXTERNAL_IP]:nodePort/
# Password: admin
- Seed the LDAP database with new entries.
$ kubectl exec -it -n kubeflow ldap-0 -- bash
root@ldap-0:/# ldapadd -x -D "cn=admin,dc=example,dc=com" -W
Enter LDAP Password:
# People definitions.
dn: ou=People,dc=example,dc=com
objectClass: organizationalUnit
ou: People
dn: cn=ysjeon7,ou=People,dc=example,dc=com
objectClass: person
objectClass: inetOrgPerson
cn: ysjeon7
sn: ysjeon7
givenName: ysjeon7
uid: ysjeon7
mail: yoosung.jeon@kt.com
description: Kubeflow Administrator
userpassword: 12341234
dn: cn=lidar,ou=People,dc=example,dc=com
objectClass: person
objectClass: inetOrgPerson
cn: lidar
sn: lidar
givenName: lidar
uid: lidar
mail: lidar@kt.com
description: LiDAR Project
userpassword: lidar!@#
dn: cn=csp,ou=People,dc=example,dc=com
objectClass: person
objectClass: inetOrgPerson
cn: csp
sn: csp
givenName: csp
uid: csp
mail: csp@kt.com
description: Core Solution Project Team
userpassword: lidar!@#
# Group definitions.
dn: ou=Groups,dc=example,dc=com
objectClass: organizationalUnit
ou: Groups
dn: cn=admins,ou=Groups,dc=example,dc=com
objectClass: groupOfNames
cn: admins
member: cn=ysjeon7,ou=People,dc=example,dc=com
dn: cn=developers,ou=Groups,dc=example,dc=com
objectClass: groupOfNames
cn: developers
member: cn=lidar,ou=People,dc=example,dc=com
member: cn=csp,ou=People,dc=example,dc=com
# Press Ctrl+D to complete
adding new entry "ou=People,dc=example,dc=com"
adding new entry "cn=Nick Kiliadis,ou=People,dc=example,dc=com"
adding new entry "cn=Robin Spanakopita,ou=People,dc=example,dc=com"
adding new entry "cn=admins,ou=Groups,dc=example,dc=com"
adding new entry "cn=developers,ou=Groups,dc=example,dc=com"
root@ldap-0:/#
- To use your LDAP/AD server with Dex
$ kubectl get configmap dex -n auth -o jsonpath='{.data.config\.yaml}' > dex-config.yaml
$ vi dex-config-ldap-partial.yaml
connectors:
- type: ldap
id: ldap
name: LDAP
config:
host: ldap-service.kubeflow.svc.cluster.local:389
insecureNoSSL: true
insecureSkipVerify: true
startTLS: false
bindDN: cn=admin,dc=example,dc=com
bindPW: admin
usernamePrompt: username
userSearch:
baseDN: ou=People,dc=example,dc=com
filter: "(objectClass=inetOrgPerson)"
username: uid
idAttr: uid
emailAttr: mail
nameAttr: givenName
groupSearch:
baseDN: ou=Groups,dc=example,dc=com
filter: "(objectClass=groupOfNames)"
userAttr: DN
groupAttr: member
nameAttr: cn
$ cat dex-config.yaml dex-config-ldap-partial.yaml > dex-config-final.yaml
$ kubectl create configmap dex --from-file=config.yaml=dex-config-final.yaml -n auth --dry-run -o yaml | kubectl apply -f -
configmap/dex configured
$ kubectl rollout restart deployment dex -n auth
deployment.extensions/dex restarted
$- http://34.74.255.107:30990/ 접속
“Log in with LDP” 선택, Username(LDAP에서 입력한 uid)와 Password 입력
Kubernetes에 namespace 추가 (default : uid)
- Logout
Login ID 정보는 쿠키에 저장되어 있으며, Logout 후 다른 계정으로 Login하기 위해서는 쿠키를 삭제해야 함
h. Secure with HTTPS
- https://www.kubeflow.org/docs/started/k8s/kfctl-istio-dex/#secure-with-https
i. Expose with a LoadBalancer
- https://www.kubeflow.org/docs/started/k8s/kfctl-istio-dex/#expose-with-a-loadbalancer
3. Troubleshooting
a. TS #1
▷ Problem: Service Account Token Volume Projection
$ k get pod -n istio-system
istio-ingressgateway-77f74c944c-5qdb9 0/1 ContainerCreating 0 77m
istio-pilot-55f7f6f6df-xf4nj 0/2 ContainerCreating 0 6m54s
istio-policy-76dbd68445-fhvpf 0/2 ContainerCreating 0 77m
istio-telemetry-697c8fd794-2lszp 0/2 ContainerCreating 0 77m
knative-serving activator-6dc4884-g22jq 0/2 Init:0/1 0 83m
knative-serving autoscaler-69bcc99c79-8bdjp 0/2 Init:0/1 0 83m
…
$
$ k edit pod istio-pilot-55f7f6f6df-xf4nj -n istio-system
…
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Normal Scheduled 3m42s default-scheduler Successfully assigned istio-system/istio-pilot-55f7f6f6df-xf4nj to worker-2
Warning FailedMount 99s kubelet, worker-2 Unable to mount volumes for pod "istio-pilot-55f7f6f6df-xf4nj_istio-system(411e1cda-c5b1-4c75-8253-8e8a32723e3a)": timeout expired waiting for volumes to attach or mount for pod "istio-system"/"istio-pilot-55f7f6f6df-xf4nj". list of unmounted volumes=[istio-token]. list of unattached volumes=[sds-uds-path istio-token config-volume istio-certs istio-pilot-service-account-token-jp68v]
Warning FailedMount 94s (x9 over 3m42s) kubelet, worker-2 MountVolume.SetUp failed for volume "istio-token" : failed to fetch token: the server could not find the requested resource
$▷ Cause:
$ kubectl create serviceaccount build-robot
$ kubectl get serviceaccount build-robot -o json
{
"apiVersion": "v1",
"kind": "ServiceAccount",
"metadata": {
"creationTimestamp": "2020-05-26T23:33:08Z",
"name": "build-robot",
"namespace": "default",
"resourceVersion": "2108247",
"selfLink": "/api/v1/namespaces/default/serviceaccounts/build-robot",
"uid": "c3b54fda-afb4-4b4b-a89d-c16bb67c62ea"
},
"secrets": [
{
"name": "build-robot-token-sz8tm"
}
]
}
$ k describe secret build-robot-token-sz8tm
Name: build-robot-token-sz8tm
…
Data
====
ca.crt: 1025 bytes
namespace: 7 bytes
token: eyJhbGciOiJSUzI1NiIsImtpZCI6IiJ9.eyJpc3MiOiJrdWJlcm5ldGVzL3NlcnZpY2VhY2NvdW50Iiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9uYW1lc3BhY……
$ wget https://k8s.io/examples/pods/pod-projected-svc-token.yaml
$ vi pod-projected-svc-token.yaml
apiVersion: v1
kind: Pod
metadata:
name: nginx
spec:
containers:
- image: nginx
name: nginx
volumeMounts:
- mountPath: /var/run/secrets/tokens
name: vault-token
serviceAccountName: build-robot
volumes:
- name: vault-token
projected:
sources:
- serviceAccountToken:
path: vault-token
expirationSeconds: 7200
audience: vault
$ kubectl apply -f pod-projected-svc-token.yaml
…
$ k get pod nginx
NAME READY STATUS RESTARTS AGE
nginx 0/1 ContainerCreating 0 6m29s
$ k describe pod nginx | tail -n 5
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Normal Scheduled 27s default-scheduler Successfully assigned default/nginx to worker-1
Warning FailedMount 11s (x6 over 27s) kubelet, worker-1 MountVolume.SetUp failed for volume "vault-token" : failed to fetch token: the server could not find the requested resource
$▷ Solution:
- https://jpweber.io/blog/a-look-at-tokenrequest-api/
$ sudo vi /etc/kubernetes/manifests/kube-apiserver.yaml
…
- --service-account-key-file=/etc/kubernetes/pki/sa.pub
- --service-account-issuer=kubernetes.default.svc # appended
- --service-account-signing-key-file=/etc/kubernetes/pki/sa.key # appended
…
$
$ k exec nginx ls /var/run/secrets/tokens/
vault-token
$ k exec nginx cat /var/run/secrets/tokens/vault-token
yJhbGciOiJSUzI1NiIsImtpZCI6IiJ9.eyJhdWQiOlsidmF1bHQiXSwiZXhwIjoxNTkwNjcxMDA0LCJpYXQiOjE1OTA2NjM4MDQsImlzcyI6Im……
$
'Kubeflow > Install' 카테고리의 다른 글
| Kubeflow 1.4.1 in Minikube 구성 (0) | 2021.12.30 |
|---|---|
| Kubeflow 1.2 in On-prem 구성 (0) | 2021.09.24 |
| Kubeflow 1.2 in Minikube 구성 (0) | 2021.09.24 |
| Kubeflow 1.0 using MiniKF 구성 (Windows 10) (0) | 2021.09.24 |
| Kubeflow 1.0 in GCE 구성 (0) | 2021.09.24 |
댓글