Skip to content

Latest commit

 

History

History
116 lines (75 loc) · 6 KB

USER_GUIDE.md

File metadata and controls

116 lines (75 loc) · 6 KB

MLflow user guide

Contents

Installation instructions MLflow Models Using Allas

Installation instructions

There is also video guide available in video.csc.fi -service:

If you are unfamiliar with Rahti and how to get started, first check Rahti's documentation.

Create new project by clicking Create Project and fill your project name and csc_project code.

Create project

After you've created the project, Browse catalog.

Browse catalog

Find MLflow, click to choose it and click Next to start configuring your application.

Configure app

Fill the configuration form. Explanation and recommendations after the picture below.

Fill the form

  • App Name: You can define unique name for your MLflow instance. Name will be part of app public address.
  • Username: Your self defined username for tracking server.
  • Password: Your self defined password for tracking server.
  • Storage Size: Storage size for MLflow persistent storage. Default should be enough.
  • Backend storage size: Storage size for tracked metrics. Default should be enough.
  • Object storage size: Storage size for tracked artifacts.
  • Route whitelist: Add your static ip or ip range if you want to restrict access to your service.

For rest of the variables default is fine for most users.

  • MLflow image: Latest tested image for Rahti as default value.
  • Artifact store access key: Generated by template.
  • Artifact store secret access key: Generated by template.
  • PostgreSQL connection username: Generated by template.
  • PostgreSQL connection password: Generated by template.
  • PostgreSQL connection database: Generated by template.
  • Application Hostname Suffix: Default is for Rahti.

After setting up all variables, click Create to start deployment. After provision is finished, you can close the window.

Service ready

In the overview you can see all the services of your deployment and their states. Next step is to configure the credentials to your development environment for utilizing the services.

Project overview

By expanding the service information of minio and mlflow-ui, you can find the atrifact store- and tracking server adresses. From the index, go to Resources > Secrets, where you can find the automatically generated artifact store keys under mlflow-s3-credentials.

Credentials

Congratulations! After deployment is complete, you are ready to start using MLflow to track your machine learning training runs, compare models and to deploy best ones to inference. There are simple test notebooks for tracking and inference in examples directory. Example README The adresses and secret keys will be used in tutorials.

MLflow Models

This service is not included in the examples for now.

MLflow Models is used to serve tracked model for inference. In the template, Models is run in own pod, so inference doesn't affect Tracking performance. Currently, template support serving one model at time.

How to set up

First you have to have some training runs recorded to Tracking server with model as a artifact. Model have to be stored to default artifact store in MinIO, or in CSC Allas.

Open your experiment and your chosen run in your MLflow Tracking server. Under artifacts there is model directory. When clicking that, Full Path of model uri can be copied. Model URI path

Switch into Rahti and add copied uri to MODELS_URI variable in models-cfg in config map save.

Model CFG location

Increase mlflow-models pod count to 1.

Increase pod

Model could be now queried with http request, using models pod address in your own testing environment. Copy your model url

Using CSC Allas as artifact store

For heavier use case it is recommended to use CSC Allas object storage service as artifact store. To do that, you should have granted access to a project that has Allas- service enabled (you can check this from my.csc.fi).

Create bucket for your mlflow-data via console tools or Web UI. See instructions here.

You have to have oc client installed to you computer to modify secrets created by MLflow template. Follow instructions to install oc, configure and connect it. You can get the login command with token from you Rahti UI- click your name in the upper right corner, and click "Copy Login Command".

There are multiple ways to connect with Allas, but here we use the S3 client. See installation guide and instructions here.

After you have installed all necessary, download mlflow-secret.yaml file and save it locally. Next you need to get the access keys from the .s3cmd- file, which is normally located in your home directory. Copy them and modify following lines in mlflow-secrets.yaml:

  AWS_ACCESS_KEY_ID: your_access_key
  AWS_SECRET_ACCESS_KEY: your_secret_key

Save and use oc to modify existing secret in your Rahti MLflow namespace by commanding: oc apply -f ./mlflow-secret.yaml

After that, edit Resources -> Config Maps -> mlflow-cfg and change DEFAULT_ARTIFACT_ROOT variable to s3://your_bucket_name and MLFLOW_S3_ENDPOINT_URLto https://a3s.fi

Restart mlflow-ui pod by scaling it first to zero, then back to one. If you're using conda- environment, reactivate it to update.

Now when you create new mlflow experiment, default artifact store is your bucket in Allas.