Skip to content

Commit

Permalink
Updated as per recommendations
Browse files Browse the repository at this point in the history
  • Loading branch information
Moses-Mk committed Oct 26, 2024
1 parent cb55dc9 commit 62b9963
Show file tree
Hide file tree
Showing 6 changed files with 44 additions and 58 deletions.
70 changes: 33 additions & 37 deletions docs/code/aws/cloud_vm.md
Original file line number Diff line number Diff line change
@@ -1,7 +1,6 @@
`cloud_vm` deploys MLOps `stack` on top of Cloud provider VMs.
# ☁️ Cloud Infrastructure Examples
# ☁️ Cloud VM Examples

Welcome to the **Cloud Infrastructure** examples! This section contains ready-to-use configurations to help you deploy scalable MLOps stacks on cloud platforms like AWS. Each example is designed to showcase a different tool or workflow, making it easier for you to find and deploy the infrastructure that fits your needs.
Welcome to the **Cloud VM** examples! This section contains ready-to-use configurations to help you deploy scalable MLOps stacks on cloud platforms like AWS. Each example is designed to showcase a different tool or workflow, making it easier for you to find and deploy the infrastructure that fits your needs.

## 🚀 Getting Started

Expand All @@ -26,16 +25,16 @@ Before you begin, ensure you have the following:
3. **Choose an Example**:
Navigate to the example you want to deploy:
```bash
cd examples/cloud_infra/dagster
cd examples/cloud_vm/dagster
```

4. **Configure AWS Credentials**:
Modify the `terraform.tfvars` file to update your AWS account details. Make sure your AWS credentials are correctly configured.
Modify the YAML configuration file (e.g., `aws-dagster.yaml`) to update your AWS account details. Ensure your AWS credentials are correctly configured.

5. **Deploy**:
Use `mlinfra` to apply the configuration:
```bash
mlinfra terraform --action apply --stack-config-path ./config/aws-config.yaml
mlinfra terraform --action apply --stack-config-path examples/cloud_vm/dagster/aws-dagster.yaml
```

## 📂 Examples Available
Expand Down Expand Up @@ -86,29 +85,29 @@ Before you begin, make sure you have:
To deploy the basic infrastructure, use the following command:

```bash
aws cloudformation deploy --template-file aws-complete.yaml --stack-name basic-aws-stack
aws cloudformation deploy --template-file aws-complete.yaml --stack-name aws-mlops-stack-complete
```

### 3. Advanced Deployment

For a more comprehensive setup with additional features, execute:

```bash
aws cloudformation deploy --template-file aws-complete-advanced.yaml --stack-name advanced-aws-stack
mlinfra terraform --action apply --stack-config-path examples/cloud_vm/complete/aws-complete-advanced.yaml
```

### 4. Verifying Deployment

To verify that your infrastructure has been set up correctly, check the status of your CloudFormation stack:

```bash
aws cloudformation describe-stacks --stack-name basic-aws-stack
aws cloudformation describe-stacks --stack-name aws-mlops-stack-complete
```

Or for the advanced setup:

```bash
aws cloudformation describe-stacks --stack-name advanced-aws-stack
aws cloudformation describe-stacks --stack-name aws-mlops-stack-complete-advanced
```

### 5. Managing Resources
Expand Down Expand Up @@ -149,12 +148,12 @@ To use these configurations, ensure you have the following prerequisites:

### 📦 Installation

To install the `platinfra` package, you can use pip. It's recommended to create a Python virtual environment first:
To install the `mlinfra` package, you can use pip. It's recommended to create a Python virtual environment first:
```bash
python -m venv venv
source venv/bin/activate # On Windows use `venv\Scripts\activate`
pip install platinfra
pip install mlinfra
```
## 🌐 Deployment Instructions
Expand All @@ -166,7 +165,7 @@ The `aws-lakefs.yaml` file provides a straightforward setup for deploying LakeFS
**To deploy:**
```bash
platinfra terraform --action apply --stack-config-path aws-lakefs.yaml
mlinfra terraform --action apply --stack-config-path examples/cloud_vm/lakefs/aws-lakefs.yaml
```
This command will create the necessary AWS resources and deploy LakeFS.
Expand All @@ -182,26 +181,24 @@ The `aws-lakefs-advanced.yaml` file contains a more comprehensive configuration,
**To deploy:**
```bash
platinfra terraform --action apply --stack-config-path aws-lakefs-advanced.yaml
mlinfra terraform --action apply --stack-config-path examples/cloud_vm/lakefs/aws-lakefs-advanced.yaml
```
Feel free to modify the advanced configuration according to your specific use cases and resource requirements.
## 🔧 Customization Tips
- **IAM Roles**: Ensure that the necessary IAM roles and policies are in place to allow LakeFS to access required AWS resources, such as S3.
- **Environment Variables**: Adjust environment variables in the YAML files for configuring LakeFS settings.
- **Storage Configuration**: Make sure to configure persistent storage options if needed.
- **Storage Configuration**: Configure persistent storage options as needed.
## 📦 Managing Your Deployment
- **Update Configuration**: To apply changes, use:
```bash
platinfra terraform --action apply --stack-config-path <your-updated-file>.yaml
mlinfra terraform --action apply --stack-config-path <your-updated-file>.yaml
```
- **Check Status**: Monitor the deployment using:
- **Check Status**: Monitor the deployment status using:
```bash
aws cloudformation describe-stacks --stack-name lakefs-stack
aws cloudformation describe-stacks --stack-name aws-mlops-stack-lakefs
```
- **Logs**: View logs for troubleshooting in the AWS Console.
Expand All @@ -210,8 +207,8 @@ Feel free to modify the advanced configuration according to your specific use ca
To remove the LakeFS deployment, execute:
```bash
platinfra terraform --action destroy --stack-config-path aws-lakefs.yaml
platinfra terraform --action destroy --stack-config-path aws-lakefs-advanced.yaml
mlinfra terraform --action destroy --stack-config-path aws-lakefs.yaml
mlinfra terraform --action destroy --stack-config-path aws-lakefs-advanced.yaml
```
## 📞 Support & Resources
Expand Down Expand Up @@ -250,10 +247,9 @@ Before you start, ensure you have the following:
source venv/bin/activate
```
2. **Install the Required Packages:**
2. **Install the Required Packages:**
```bash
pip install platinfra
pip install mlinfra
```
## Deployment Configuration
Expand All @@ -267,7 +263,7 @@ You can choose between two deployment configurations based on your needs:
This configuration is suitable for a simple MLflow deployment with essential features. To deploy:
```bash
platinfra terraform --action apply --stack-config-path <path-to-your-config>/aws-mlflow.yaml
mlinfra terraform --action apply --stack-config-path examples/cloud_vm/mlflow/aws-mlflow.yaml
```
### 2. Advanced MLflow Deployment
Expand All @@ -277,7 +273,7 @@ platinfra terraform --action apply --stack-config-path <path-to-your-config>/aws
This configuration includes additional features for a more robust MLflow deployment. To deploy:
```bash
platinfra terraform --action apply --stack-config-path <path-to-your-config>/aws-mlflow-advanced.yaml
mlinfra terraform --action apply --stack-config-path examples/cloud_vm/mlflow/aws-mlflow-advanced.yaml
```
## Configuration
Expand Down Expand Up @@ -321,36 +317,36 @@ Before deploying, ensure you have the following:
### 📦 Installation
To install the `platinfra` package, you can use pip. It's recommended to create a Python virtual environment first:
To install the `mlinfra` package, you can use pip. It's recommended to create a Python virtual environment first:

```bash
python -m venv venv
source venv/bin/activate # On Windows use `venv\Scripts\activate`
pip install platinfra
pip install mlinfra
```

### 2. Basic Deployment

To deploy using the basic configuration, run:

```bash
platinfra terraform --action apply --stack-config-path aws-wand.yaml
mlinfra terraform --action apply --stack-config-path examples/cloud_vm/wandb/aws-wandb.yaml
```

### 3. Advanced Deployment

For an advanced setup with additional features, execute:

```bash
platinfra terraform --action apply --stack-config-path aws-wand-advanced.yaml
mlinfra terraform --action apply --stack-config-path examples/cloud_vm/wandb/aws-wandb-advanced.yaml
```

### 4. Verifying Deployment

After applying the configurations, check the status of the Weights and Biases services:

```bash
aws cloudformation describe-stacks --stack-name wand-stack
aws cloudformation describe-stacks --stack-name aws-mlops-stack-mlflow
```

### 5. Accessing Weights and Biases UI
Expand Down Expand Up @@ -397,7 +393,7 @@ Before you start, ensure you have the following:

2. **Install the Required Python Package:**
```bash
pip install platinfra
pip install mlinfra
```

## Deployment Configuration 📄
Expand All @@ -423,9 +419,9 @@ To deploy Prefect on AWS, follow these steps:
2. **Deploy the Configuration:**
Run the following command to apply the configuration:
```bash
platinfra terraform --action apply --stack-config-path <path-to-your-config-file>
mlinfra terraform --action apply --stack-config-path examples/cloud_vm/prefect/aws-prefect-advanced.yaml
```
Replace `<path-to-your-config-file>` with the path to either `aws-prefect.yaml` or `aws-prefect-advanced.yaml`.


## Conclusion 🎉

Expand Down Expand Up @@ -472,7 +468,7 @@ Before deploying Dagster on AWS, make sure you have:
```
2. **Deploy the infrastructure:**
```bash
aws cloudformation deploy --template-file aws-dagster.yaml --stack-name DagsterBasicSetup
aws cloudformation deploy --template-file aws-dagster.yaml --stack-name aws-mlops-stack-dagster
```
3. 🎉 **You're all set!** Access Dagster via the provided endpoint.

Expand All @@ -483,7 +479,7 @@ Before deploying Dagster on AWS, make sure you have:
```
2. **Deploy the infrastructure:**
```bash
aws cloudformation deploy --template-file aws-dagster-advanced.yaml --stack-name DagsterAdvancedSetup
aws cloudformation deploy --template-file aws-dagster-advanced.yaml --stack-name aws-mlops-stack-dagster
```
3. **Customize the parameters if needed:**
Modify the `.yaml` file to suit your specific needs, including scaling options, security settings, and more.
Expand Down
27 changes: 11 additions & 16 deletions docs/code/aws/kubernetes.md
Original file line number Diff line number Diff line change
@@ -1,5 +1,3 @@
`kubernetes` deploys MLOps `stack` on top of Cloud provider's kubernetes. In case of AWS, its EKS.

# 🐳 Kubernetes Examples

Welcome to the **Kubernetes** examples! This section provides configurations and guides to help you deploy scalable MLOps stacks on Kubernetes clusters. Kubernetes is ideal for managing containerized applications, and these examples will help you harness its power for your machine learning workflows.
Expand All @@ -16,7 +14,8 @@ Before deploying the examples, make sure you have:
### 📦 Installation

1. **Set Up Kubernetes Cluster**:
- If you don’t have a cluster, create one using your preferred platform. For example, using **minikube**:
- For cloud-based deployment, create a cluster using your preferred platform (e.g., EKS, GKE, AKS).
- For local development, you can use Minikube:
```bash
minikube start
```
Expand All @@ -34,23 +33,19 @@ Before deploying the examples, make sure you have:

4. **Deploy to Kubernetes**:
Use `mlinfra` to deploy the chosen configuration:
```bash
mlinfra terraform --action apply --stack-config-path ./config/k8s-config.yaml
```

## 📂 Available Examples
```bash
mlinfra terraform --action apply --stack-config-path examples/kubernetes/lakefs/aws-lakefs.yaml
```

| Example File | Description |
| ------------------------| ----------- |
## 📂 Available Examples

| Example File | Description |
| ------------------------| ------------------------------------------------------------- |
| `Complete` | Complete configuration for deploying ML projects on Kubernetes |
| `LakeFS` | Basic configuration for deploying LakeFS on Kubernetes |
| `LakeFS Advanced` | Advanced configuration for LakeFS with additional cloud database storage options |
| `MLflow` | Configuration for deploying MLflow on Kubernetes |
| `Prefect` | Configuration for deploying Prefect on Kubernetes |
| ----------------------- | ------------------------------------------------------------- |
| `Complete` | Comprehensive configuration for deploying ML projects on Kubernetes. |
| `LakeFS` | Configuration for deploying LakeFS on Kubernetes. |
| `MLflow` | Configuration for deploying MLflow on Kubernetes. |
| `Prefect` | Configuration for deploying Prefect on Kubernetes. |



### ℹ️ Note:
Expand Down
2 changes: 0 additions & 2 deletions docs/code/local/kind.md
Original file line number Diff line number Diff line change
@@ -1,5 +1,3 @@
`kind` deploys MLOps `stack` on top of [kind cluster](https://kind.sigs.k8s.io/) on your machine. Please ensure that docker is installed and running on your machine, along with the latest version of kind.

# Kind Deployment in Local Environment 🖥️

This directory contains YAML configurations for deploying applications using **Kind** (Kubernetes IN Docker). Kind allows you to run Kubernetes clusters in Docker containers, making it an excellent tool for local development and testing.
Expand Down
2 changes: 0 additions & 2 deletions docs/code/local/minikube.md
Original file line number Diff line number Diff line change
@@ -1,5 +1,3 @@
`minikube` deploys MLOps `stack` on top of [minikube cluster](https://minikube.sigs.k8s.io/) on your machine. Please ensure that docker is installed and running on your machine, along with the latest version of minikube.

# Minikube Deployment in Local Environment 🚀

This directory contains YAML configurations for deploying applications using **Minikube**. Minikube is a tool that enables you to run Kubernetes clusters locally, making it an excellent choice for development and testing.
Expand Down
1 change: 0 additions & 1 deletion mlinfra
Submodule mlinfra deleted from 41a320
Binary file removed src/mlinfra/__pycache__/__init__.cpython-311.pyc
Binary file not shown.

0 comments on commit 62b9963

Please sign in to comment.