kubernetes55 Dex 인증 / 우회 1. 개요 - Dex ? (https://github.com/dexidp/dex) Dex is an identity service that uses OpenID Connect to drive authentication for other apps. - Dex 구성 'kfctl_istio_dex.v1.2.0.yaml'를 이용하여 Kubeflow(v1.2)를 구성할 경우 dex(v2.22)가 포함되어 설치 된다. - Dex 인증 범위 다음과 같은 유형의 자원에 호출할 경우 Dex 인증을 요구 한다. ✓ Kubeflow dashboard login ✓ KFServing (Knative 기반에서 동작) ✓ Knative serving (Istio 기반에서 동작) ✓ Istio Virtual service - 적용.. 2021. 9. 29. Distributed training 사례 #4 (From KF Jupyter, PyTorch) 2021.06.30 1. Kubeflow Jupyter 환경에서 Distributed training (PyTorch) - Environments ✓ Remote - 개발 환경 Kubeflow Jupyter (CPU) ✓ Remote - 학습 환경 Kubeflow 1.2 (The machine learning toolkit for Kubernetes) / Kubernetes 1.16.15 Nexus (Private docker registry) Nvidia V100 / Driver 450.80, cuda 11.2, cuDNN 8.1.0 - Flow (PyTorch) ✓ Remote (Kubeflow Jupyter) a. Docker Image build b. Docker Image Push c. Kube.. 2021. 9. 27. Distributed training 사례 #3 (In Jupyter) 2021.06.25 1. Kubeflow Jupyter(GPU 할당) 환경에서 Distributed training (Tensorflow) - Environments ✓ Remote - 개발 환경 Kubeflow Jupyter (GPU) ✓ Remote - 학습 환경 Kubeflow 1.2 (The machine learning toolkit for Kubernetes) / Kubernetes 1.16.15 Harbor 2.2.1 (Private docker registry) Nvidia V100 / Driver 450.80, cuda 11.2, cuDNN 8.1.0 CentOS 7.8 - 관련 기술 - Tensorflow MirroredStrategy (Data parallelism) 2. 사전작업 a.. 2021. 9. 26. Distributed training 사례 #2 (From KF Jupyter, Tensorflow) 2021.06.24 1. Kubeflow Jupyter 환경에서 Distributed training (Tensorflow) - Environments ✓ Remote - 개발 환경 Kubeflow Jupyter ✓ Remote - 학습 환경 Kubeflow 1.2 (The machine learning toolkit for Kubernetes) / Kubernetes 1.16.15 Nexus (Private docker registry) Nvidia V100 / Driver 450.80, cuda 11.2, cuDNN 8.1.0 CentOS 7.8 - Flow (Tensorflow) ✓ Remote (Kubeflow Jupyter) a. Docker Image build b. Docker Image P.. 2021. 9. 26. Distributed training 사례 #1 (From MacOS) 2021.06.24 1. Local 환경에서 Distributed training (Tensorflow) - Environments ✓ Local - 개발 환경 Python 3.8.5, Jupyter / PyCharm (option) ✓ Remote - 학습 환경 Kubeflow 1.2 (The machine learning toolkit for Kubernetes) / Kubernetes 1.16.15 Master node: 3ea, Worker node: 4ea Harbor 2.2.1 (Private docker registry) Nvidia V100 / Driver 450.80, cuda 11.2, cuDNN 8.1.0 CentOS 7.8 - Flow (Tensorflow 기준) ✓ Local (M.. 2021. 9. 26. CI/CD 적용 가이드 #3 (CD-Gitops 편) 2021.04.06 1. CD (Continuous Deployment) & Gitops - CD ✓ 개발자의 변경 사항을 레포지토리에서 개발/프로덕션 환경으로 자동으로 릴리스 ✓ 지속적 배포가 제대로 이루어지려면 테스트 자동화가 제대로 설계되어 있어야 함 - Gitops ✓ 클라우드 네이티브 애플리케이션을 대상으로 한 지속적 배포(Continuous Deployment)에 초점 ✓ 애플리케이션의 배포와 운영에 관련된 모든 요소를 코드화하여 깃(Git)에서 관리(Ops)하는 것이 핵심 2. Kubernetes Configuration Management - popular config tools ✓ Kustomize applications ▷ a template-free way to customize ap.. 2021. 9. 26. Kubeflow 1.0 기능 #6 (Metadata) 2020.03.23 1. Kubeflow Metadata ? - Tracking and managing metadata of machine learning workflows in Kubeflow - metadata means information about executions (runs), models, datasets, and other artifacts. Artifacts are the files and objects that form the inputs and outputs of the components in your ML workflow 2. Try the Metadata SDK in a sample Jupyter notebook - demo.ipynb download from Github Te.. 2021. 9. 25. Kubeflow 1.0 기능 #5 (KFServing, TFServing) 2020.03.13 1. Model serving overview - https://v1-0-branch.kubeflow.org/docs/reference/pytorchjob/v1/pytorch/ - Kubeflow supports two model serving systems that allow multi-framework model serving: KFServing and Seldon Core. Alternatively, you can use a standalone model serving system. a. Multi-framework model serving - A check mark (✓) indicates that the system (KFServing or Seldon Core) suppor.. 2021. 9. 25. Kubeflow 1.0 기능 #4 (PyTorch Training) 2020.03.09 1. PyTorchJob ? - Kubeflow에서 PyTorch training할 때 사용되는 Kubernetes custom resource - https://v1-0-branch.kubeflow.org/docs/reference/pytorchjob/v1/pytorch/ 2. PyTorch training 하기 - https://v1-0-branch.kubeflow.org/docs/components/training/pytorch/components/training/pytorch/ a. Cloud shell 기동 b. Verify that PyTorch support is included in your Kubeflow deployment $ kubectl get crd | head.. 2021. 9. 25. Kubeflow 1.0 기능 #2 (TF-Job, TF-Serving, Kubeflow pipeline) 2020.02.26 1. 개요 - GKE에 설치한 Kubeflow의 Pipeline 기능을 이해하기 위해 아래 사이트를 참조하여 사용 해 봄 - Using Kubeflow for Financial Time Series (https://github.com/kubeflow/examples/tree/master/financial_time_series) - This example covers the following concepts: a. Deploying Kubeflow to a GKE cluster b. Exploration via JupyterHub (prospect data, preprocess data, develop ML model) c. Training several tensorflow models.. 2021. 9. 24. Kubeflow 1.0 기능 #1 (Jupyter notebook) 2020.03.12 1. 참고 문서 - https://v1-0-branch.kubeflow.org/docs/notebooks/setup/ 2. Notebook server 생성 a. Cloud shell 기동 b. URL 확인 $ kubectl -n istio-system get ingress NAME HOSTS ADDRESS PORTS AGE envoy-ingress my-kubeflow.endpoints.my-kubeflow-269301.cloud.goog 34.107.211.135 80 6m42s$ $ c. Kubeflow 접속 (URL: my-kubeflow.endpoints.my-kubeflow-269301.cloud.goog) d. Create a Jupyter notebook server a.. 2021. 9. 24. Running the MNIST using distributed training 2021.5.28 1. Running the MNIST on-prem Jupyter notebook - The MNIST on-prem notebook builds a Docker image, launches a TFJob to train a model, and creates an InferenceService (KFServing) to deploy the trained model. - https://v1-2-branch.kubeflow.org/docs/started/workstation/minikube-linux/#running-the-mnist-on-prem-jupyter-notebook a. Prerequisites - Step 1: Set up Python environment in MacOS y.. 2021. 9. 24. Kubeflow Jupyter Custom Image 추가 2021.06.29 1. 개요 - kubeflow Notebook Server 생성을 위하여 Custom Image를 추가 - References https://www.kangwoo.kr/tag/jupyter/ https://github.com/kubeflow/kubeflow/tree/v1.2.0/components/tensorflow-notebook-image https://towardsdatascience.com/make-kubeflow-into-your-own-data-science-workspace-cc8162969e29 - Custom Image spec ✓ Base image: tensorflow/tensorflow:2.5.0-gpu-jupyter (cuda 11.2, cuDNN 8.1.0),.. 2021. 9. 24. Kubeflow 1.2 in On-prem 구성 2021.05.03 1. 환경 - Kubeflow 1.2.0 - Kubernetes 1.16.15 - CentOS Linux release 7.8 - Dex (https://github.com/dexidp/dex) Dex is an identity service that uses OpenID Connect to drive authentication for other apps. - Kubernetes compatibility 2. Kubeflow Install - https://v1-2-branch.kubeflow.org/docs/started/k8s/kfctl-istio-dex/ a. Notes on the configuration file - Istio configuration for trustwort.. 2021. 9. 24. Kubeflow 1.2 in Minikube 구성 2021.04.27 1. 개요 - Minikube 기반 하에 kubeflow 구성 - https://v1-2-branch.kubeflow.org/docs/started/workstation/minikube-linux/ 문서에서 설명한 옵션을 사용할 경우 Kubeflow dashboard 접근시 오류 발생 (service-account-signing-key-file 값, Troubleshooting 참조) 2. Envrionments - minikube 1.17.1 - kubernetes 1.16.15 - kubeflow 1.2 - macOS 11.2 3. Install Kubeflow a. Prerequisites - Recommended resources 8 cores, 16GB RAM, 250GB s.. 2021. 9. 24. 이전 1 2 3 4 다음
