In my quest to find the perfect MLOps workflow to accelerate my financial ML research, I played around with Elyra. It’s a set of plugins for Jupyter Notebooks that allows you to visually drag and drop a (Python) pipeline.
To use the Kubeflow pipelines feature I needed to install Kubeflow Pipelines on my local machine. The Elyra documentation on this uses Docker Desktop for MacOS or Windows. As my machine is running Pop!_OS 21.04 and I already had Minikube installed from installing MLRun, I adapted the Elyra Kubeflow Pipelines Guide to use Minikube.
Prerequisites:
Start Minikube
The Elyra guide states that the minimium hardware requirements are 4 CPUs and 8GB of RAM. My system has 6c/12t and 64GB of RAM, so adjust the values in the command below to match your system.
minikube start --vm-driver=docker --cpus=6 --memory='24g' --kubernetes-version=v1.16.8
Install KubeFlow Pipelines
export PIPELINE_VERSION=1.4.0
minikube kubectl -- apply -k "github.com/kubeflow/pipelines/manifests/kustomize/cluster-scoped-resources?ref=$PIPELINE_VERSION"
minikube kubectl -- wait --for condition=established --timeout=60s crd/applications.app.k8s.io
minikube kubectl -- apply -k "github.com/elyra-ai/elyra/etc/kubernetes/kubeflow-pipelines?ref=master"
Run the code below and wait for pods and deployments to be ready.
minikube kubectl -- get all -n kubeflow
Port foward the Kubeflow Pipelines API
minikube kubectl -- port-forward $(minikube kubectl -- get pods -n kubeflow | grep ml-pipeline-ui | cut -d' ' -f1) 31380:3000 -n kubeflow &
Add minio-service to local hosts file
echo '127.0.0.1 minio-service' | sudo tee -a /etc/hosts
Port foward the minio service
minikube kubectl -- port-forward $(minikube kubectl -- get pods -n kubeflow | grep minio | cut -d' ' -f1) 9000:9000 -n kubeflow &
The Resulting Endpoints:
- UI Endpoint: http://localhost:31380
- API Endpoint: http://localhost:31380/pipeline
- Object Storage Endpoint: http://minio-service:9000
Connect Local Kubeflow Pipelines to Elyra
Elyra Runtime CLI Docs Finding your Kubeflow Engine The code below will add your local Kubeflow Pipelines instance to Elyra.
elyra-metadata install runtimes \
--display_name="Minikube Kubeflow Pipelines Runtime" \
--name=kfp-local \
--api_endpoint=http://localhost:31380/pipeline \
--auth_type="NO_AUTHENTICATION" \
--engine=Argo \
--cos_endpoint=http://minio-service:9000 \
--cos_username=minio \
--cos_password=minio123 \
--cos_bucket=testbucket \
--tags="['kfp', 'v1.0']" \
--schema_name=kfp
In the Jupyter Lab UI we can view our new runtime configuration.
I was then able to run the example generic pipeline on my new Kubeflow Pipelines runetime! For more details on this see Verify Runtime Config.
Shutdown Minikube
When you are done with your work and wish to shutdown the service run the following:
minikube stop
Disclaimer: I wrote this in Dec 2021 and published it Feb 2022. Some of this information may be outdated. If you would like to alert me to errors or other problems, please email me at blogfeedback.1kzbq@simplelogin.co