From 440dee7d4a91f4fd660ef1d260ddf680dbca0a7b Mon Sep 17 00:00:00 2001 From: Moses-Mk Date: Tue, 15 Oct 2024 23:16:34 +0530 Subject: [PATCH] Examples documentation --- examples/cloud_vm/README.md | 57 +++++++++++++++++ examples/cloud_vm/dagster/README.md | 65 ++++++++++++++++++++ examples/cloud_vm/lakefs/README.md | 91 ++++++++++++++++++++++++++++ examples/cloud_vm/mlflow/README.md | 70 +++++++++++++++++++++ examples/cloud_vm/prefect/README.md | 57 +++++++++++++++++ examples/cloud_vm/wandb/README.md | 67 ++++++++++++++++++++ examples/kubernetes/README.md | 57 +++++++++++++++++ examples/kubernetes/lakefs/README.md | 87 ++++++++++++++++++++++++++ 8 files changed, 551 insertions(+) create mode 100644 examples/cloud_vm/README.md create mode 100644 examples/cloud_vm/dagster/README.md create mode 100644 examples/cloud_vm/lakefs/README.md create mode 100644 examples/cloud_vm/mlflow/README.md create mode 100644 examples/cloud_vm/prefect/README.md create mode 100644 examples/cloud_vm/wandb/README.md create mode 100644 examples/kubernetes/README.md create mode 100644 examples/kubernetes/lakefs/README.md diff --git a/examples/cloud_vm/README.md b/examples/cloud_vm/README.md new file mode 100644 index 00000000..8a2626d1 --- /dev/null +++ b/examples/cloud_vm/README.md @@ -0,0 +1,57 @@ +# ☁️ Cloud Infrastructure Examples + +Welcome to the **Cloud Infrastructure** examples! This section contains ready-to-use configurations to help you deploy scalable MLOps stacks on cloud platforms like AWS. Each example is designed to showcase a different tool or workflow, making it easier for you to find and deploy the infrastructure that fits your needs. + +## πŸš€ Getting Started + +### Prerequisites +Before you begin, ensure you have the following: +- **Terraform**: Version `>= 1.8.0` installed on your system. [Install Terraform](https://learn.hashicorp.com/tutorials/terraform/install-cli) +- **AWS Account**: Make sure your AWS account is set up and you have appropriate IAM roles. +- **AWS CLI**: Ensure your AWS CLI is configured with the correct credentials. [Configure AWS CLI](https://docs.aws.amazon.com/cli/latest/userguide/cli-configure-quickstart.html) +- **Python**: Version `>= 3.7` to run the setup scripts. + +### πŸ“¦ Installation +1. **Create a Virtual Environment**: + ```bash + python -m venv venv + source venv/bin/activate # On Windows use: venv\Scripts\activate + ``` +2. **Install `mlinfra`**: + ```bash + pip install mlinfra + ``` + +3. **Choose an Example**: + Navigate to the example you want to deploy: + ```bash + cd examples/cloud_infra/dagster + ``` + +4. **Configure AWS Credentials**: + Modify the `terraform.tfvars` file to update your AWS account details. Make sure your AWS credentials are correctly configured. + +5. **Deploy**: + Use `mlinfra` to apply the configuration: + ```bash + mlinfra terraform --action apply --stack-config-path ./config/aws-config.yaml + ``` + +## πŸ“‚ Examples Available + +| Example Folder | Description | +| -------------- | ----------- | +| `dagster` | Deploy **Dagster** for orchestrating your ML pipelines on AWS | +| `mlflow` | Set up **MLflow** for experiment tracking and model management | +| `wandb` | Configure **Weights & Biases** for experiment tracking on cloud | +| `lakefs` | Create a **LakeFS** setup for versioning your datasets in the cloud | +| `prefect` | Set up **Prefect** for orchestrating cloud-based ML workflows | + +## 🌟 Why Use Cloud Infrastructure? +Deploying your MLOps stack on the cloud brings several advantages: +- **Scalability**: Easily scale resources up or down based on workload. +- **Cost Efficiency**: Pay for what you use, and reduce costs with managed services. +- **Flexibility**: Leverage powerful cloud services to enhance your ML workflows. + +## πŸ†˜ Need Help? +Feel free to open an issue or join our [Discord community](#) for discussions, troubleshooting, or contributing new ideas. diff --git a/examples/cloud_vm/dagster/README.md b/examples/cloud_vm/dagster/README.md new file mode 100644 index 00000000..e539207e --- /dev/null +++ b/examples/cloud_vm/dagster/README.md @@ -0,0 +1,65 @@ +# πŸš€ Dagster on AWS Cloud Setup + +Welcome to the **Dagster AWS Cloud Setup** documentation! This guide will help you deploy Dagster on AWS using the provided configurations for a seamless data orchestration experience. Let's get started! πŸ› οΈ + +## πŸ“‚ Configuration Files Overview + +### 1. `aws-dagster.yaml` +This file provides a **basic setup** for deploying Dagster on AWS. It includes essential configurations to quickly get you up and running with Dagster. Use this file if you are just starting out or need a straightforward deployment. + +**Features:** +- πŸ—οΈ Basic infrastructure setup +- 🧩 Core components of Dagster +- πŸ”Œ Integration with AWS services like S3 and CloudWatch + +### 2. `aws-dagster-advanced.yaml` +This file offers a **more advanced setup** for deploying Dagster on AWS, with additional configurations for better scalability, security, and performance. Use this file if you need more control and features for your deployment. + +**Features:** +- πŸ“ˆ Enhanced scalability and performance configurations +- πŸ”’ Advanced security settings +- πŸ”„ Custom integrations and multi-environment support + +## πŸ“¦ Prerequisites + +Before deploying Dagster on AWS, make sure you have: +1. 🏷️ AWS Account with appropriate permissions. +2. πŸ—οΈ [AWS CLI](https://aws.amazon.com/cli/) installed and configured. +3. 🧩 [Dagster](https://dagster.io/) installed on your local machine. +4. πŸ”§ Proper IAM roles and security groups set up. + +## πŸš€ Deployment Instructions + +### Basic Setup (`aws-dagster.yaml`) +1. **Configure your AWS CLI:** + ```bash + aws configure + ``` +2. **Deploy the infrastructure:** + ```bash + aws cloudformation deploy --template-file aws-dagster.yaml --stack-name DagsterBasicSetup + ``` +3. πŸŽ‰ **You're all set!** Access Dagster via the provided endpoint. + +### Advanced Setup (`aws-dagster-advanced.yaml`) +1. **Configure your AWS CLI:** + ```bash + aws configure + ``` +2. **Deploy the infrastructure:** + ```bash + aws cloudformation deploy --template-file aws-dagster-advanced.yaml --stack-name DagsterAdvancedSetup + ``` +3. **Customize the parameters if needed:** + Modify the `.yaml` file to suit your specific needs, including scaling options, security settings, and more. +4. πŸŽ‰ **Enjoy your advanced setup!** + +## πŸ“ Notes +- Ensure the **VPC settings** and **subnets** are correctly configured to avoid deployment issues. +- For the advanced setup, you might need to adjust **IAM roles** and **network configurations**. + +## πŸ“š Additional Resources +- πŸ“– [Dagster Documentation](https://docs.dagster.io/) +- πŸ“– [AWS CloudFormation Documentation](https://docs.aws.amazon.com/cloudformation/) + +Feel free to customize the `.yaml` files to suit your specific requirements. πŸ› οΈ Happy deploying! \ No newline at end of file diff --git a/examples/cloud_vm/lakefs/README.md b/examples/cloud_vm/lakefs/README.md new file mode 100644 index 00000000..acf8e6b7 --- /dev/null +++ b/examples/cloud_vm/lakefs/README.md @@ -0,0 +1,91 @@ +# 🌊 AWS LakeFS Configuration + +This directory contains configuration files for deploying LakeFS on AWS. LakeFS is an open-source data version control system designed for object storage, enabling you to manage and version your data efficiently. + +## πŸ“ Files Included + +- **`aws-lakefs.yaml`**: Basic configuration for deploying LakeFS on AWS. +- **`aws-lakefs-advanced.yaml`**: Advanced configuration for deploying LakeFS with additional features and custom settings. + +## πŸš€ Getting Started + +To use these configurations, ensure you have the following prerequisites: + +- An active AWS account. +- AWS Command Line Interface (CLI) installed and configured. +- Necessary IAM permissions for creating resources. +- Python 3.x installed. + +### πŸ“¦ Installation + +To install the `platinfra` package, you can use pip. It's recommended to create a Python virtual environment first: + +```bash +python -m venv venv +source venv/bin/activate # On Windows use `venv\Scripts\activate` +pip install platinfra +``` + +## 🌐 Deployment Instructions + +### 1. Basic LakeFS Setup + +The `aws-lakefs.yaml` file provides a straightforward setup for deploying LakeFS. This is suitable for initial testing and smaller workloads. + +**To deploy:** + +```bash +platinfra terraform --action apply --stack-config-path aws-lakefs.yaml +``` + +This command will create the necessary AWS resources and deploy LakeFS. + +### 2. Advanced LakeFS Setup + +The `aws-lakefs-advanced.yaml` file contains a more comprehensive configuration, including features like: + +- Enhanced scaling options +- Custom resource requests and limits +- Integrations with additional AWS services + +**To deploy:** + +```bash +platinfra terraform --action apply --stack-config-path aws-lakefs-advanced.yaml +``` + +Feel free to modify the advanced configuration according to your specific use cases and resource requirements. + +## πŸ”§ Customization Tips + +- **IAM Roles**: Ensure that the necessary IAM roles and policies are in place to allow LakeFS to access required AWS resources, such as S3. +- **Environment Variables**: Adjust environment variables in the YAML files for configuring LakeFS settings. +- **Storage Configuration**: Make sure to configure persistent storage options if needed. + +## πŸ“¦ Managing Your Deployment + +- **Update Configuration**: To apply changes, use: + ```bash + platinfra terraform --action apply --stack-config-path .yaml + ``` +- **Check Status**: Monitor the deployment using: + ```bash + aws cloudformation describe-stacks --stack-name lakefs-stack + ``` +- **Logs**: View logs for troubleshooting in the AWS Console. + +## 🧹 Cleanup + +To remove the LakeFS deployment, execute: + +```bash +platinfra terraform --action destroy --stack-config-path aws-lakefs.yaml +platinfra terraform --action destroy --stack-config-path aws-lakefs-advanced.yaml +``` + +## πŸ“ž Support & Resources + +For additional support, consider the following resources: + +- [LakeFS Documentation](https://docs.lakefs.io/) +- [AWS Documentation](https://aws.amazon.com/documentation/) diff --git a/examples/cloud_vm/mlflow/README.md b/examples/cloud_vm/mlflow/README.md new file mode 100644 index 00000000..9d704dfa --- /dev/null +++ b/examples/cloud_vm/mlflow/README.md @@ -0,0 +1,70 @@ +# MLflow on AWS Deployment πŸš€ + +This README provides guidance on deploying MLflow on AWS using the provided configuration files in the `mlflow` folder. + +## Overview + +MLflow is an open-source platform to manage the ML lifecycle, including experimentation, reproducibility, and deployment. This setup allows you to deploy MLflow on AWS easily. + +## Requirements + +Before you start, ensure you have the following: + +- An AWS account with the necessary permissions to create and manage resources. +- Terraform installed on your system (version >= 1.4.0). + +## Installation + +1. **Create a Python Virtual Environment:** + + ```bash + python -m venv venv + source venv/bin/activate + ``` + +2. **Install the Required Packages:** + + ```bash + pip install platinfra + ``` + +## Deployment Configuration + +You can choose between two deployment configurations based on your needs: + +### 1. Basic MLflow Deployment + +- **File:** `aws-mlflow.yaml` + +This configuration is suitable for a simple MLflow deployment with essential features. To deploy: + +```bash +platinfra terraform --action apply --stack-config-path /aws-mlflow.yaml +``` + +### 2. Advanced MLflow Deployment + +- **File:** `aws-mlflow-advanced.yaml` + +This configuration includes additional features for a more robust MLflow deployment. To deploy: + +```bash +platinfra terraform --action apply --stack-config-path /aws-mlflow-advanced.yaml +``` + +## Configuration + +- Update the `aws-mlflow.yaml` or `aws-mlflow-advanced.yaml` files with your AWS account details and any other custom configurations you require. +- Ensure your AWS credentials are properly configured on your machine. + +## Additional Information + +For more details on using MLflow and its capabilities, refer to the [MLflow Documentation](https://www.mlflow.org/docs/latest/index.html). + +## Contribution + +Feel free to contribute to this project! If you have suggestions or improvements, open an issue or pull request. + +## License + +This project is licensed under the Apache-2.0 License. diff --git a/examples/cloud_vm/prefect/README.md b/examples/cloud_vm/prefect/README.md new file mode 100644 index 00000000..066f96b2 --- /dev/null +++ b/examples/cloud_vm/prefect/README.md @@ -0,0 +1,57 @@ +# Deploying Prefect on AWS ☁️ + +This guide provides instructions for deploying Prefect on Amazon Web Services (AWS) using the provided YAML configuration files. + +## Overview πŸ“Š + +Prefect is a modern workflow orchestration tool designed to enable efficient data pipeline management. This deployment will help you run Prefect on AWS, providing scalability and reliability for your workflows. + +## Requirements βœ”οΈ + +Before you start, ensure you have the following: +- **Terraform**: Version **>= 1.4.0** installed on your machine. +- **AWS Account**: A valid AWS account with necessary permissions to create resources. + +## Installation βš™οΈ + +1. **Create a Python Virtual Environment:** + ```bash + python -m venv venv + source venv/bin/activate + ``` + +2. **Install the Required Python Package:** + ```bash + pip install platinfra + ``` + +## Deployment Configuration πŸ“„ + +In the `prefect` folder, you have the following configuration files: + +1. **aws-prefect.yaml**: Basic configuration for deploying Prefect on AWS. +2. **aws-prefect-advanced.yaml**: Advanced configuration with additional settings and optimizations. + +### Configuration File Details + +- **aws-prefect.yaml**: This file sets up a simple Prefect server instance on AWS with default settings. + +- **aws-prefect-advanced.yaml**: This file includes advanced features, such as a larger instance type, additional storage options, and monitoring configurations for enhanced performance. + +## Running the Deployment πŸš€ + +To deploy Prefect on AWS, follow these steps: + +1. **Copy the Configuration File:** + Choose one of the configuration files (basic or advanced) and modify it with your AWS account details and desired configurations. + +2. **Deploy the Configuration:** + Run the following command to apply the configuration: + ```bash + platinfra terraform --action apply --stack-config-path + ``` + Replace `` with the path to either `aws-prefect.yaml` or `aws-prefect-advanced.yaml`. + +## Conclusion πŸŽ‰ + +Congratulations! You have successfully deployed Prefect on AWS. You can now start orchestrating your workflows in the cloud. For further configurations and integrations, refer to the Prefect documentation. \ No newline at end of file diff --git a/examples/cloud_vm/wandb/README.md b/examples/cloud_vm/wandb/README.md new file mode 100644 index 00000000..78eb4ffe --- /dev/null +++ b/examples/cloud_vm/wandb/README.md @@ -0,0 +1,67 @@ +# Weights and Biases Deployment on AWS 🌊 + +This directory contains YAML configurations for deploying Weights and Biases (Wand) on AWS. Weights and Biases is a powerful tool designed for managing and monitoring machine learning experiments, helping teams to track metrics and visualize results effectively. + +## Directory Structure + +- **`aws-wand.yaml`**: This file contains the basic configuration for deploying Weights and Biases on AWS. +- **`aws-wand-advanced.yaml`**: This file includes advanced configurations for a more robust Weights and Biases deployment, incorporating features for scalability and performance. + +## Deployment Instructions + +### 1. Prerequisites + +Before deploying, ensure you have the following: + +- An active AWS account with the necessary permissions. +- AWS CLI configured with your credentials. +- Python 3.x installed. + +### πŸ“¦ Installation + +To install the `platinfra` package, you can use pip. It's recommended to create a Python virtual environment first: + +```bash +python -m venv venv +source venv/bin/activate # On Windows use `venv\Scripts\activate` +pip install platinfra +``` + +### 2. Basic Deployment + +To deploy using the basic configuration, run: + +```bash +platinfra terraform --action apply --stack-config-path aws-wand.yaml +``` + +### 3. Advanced Deployment + +For an advanced setup with additional features, execute: + +```bash +platinfra terraform --action apply --stack-config-path aws-wand-advanced.yaml +``` + +### 4. Verifying Deployment + +After applying the configurations, check the status of the Weights and Biases services: + +```bash +aws cloudformation describe-stacks --stack-name wand-stack +``` + +### 5. Accessing Weights and Biases UI + +You can access the Weights and Biases UI to monitor your machine learning experiments. Depending on your service configuration, you may need to set up an appropriate networking method, such as an Elastic Load Balancer (ELB) or an ingress point. + +Once set up, visit the ELB URL or the endpoint you configured to access the Weights and Biases UI. + +## Additional Resources + +- [Weights and Biases Documentation](https://docs.wandb.ai/) +- [AWS CLI Documentation](https://docs.aws.amazon.com/cli/latest/userguide/cli-configure-quickstart.html) + +## License πŸ“„ + +This project is licensed under the Apache-2 License. diff --git a/examples/kubernetes/README.md b/examples/kubernetes/README.md new file mode 100644 index 00000000..e3ab885b --- /dev/null +++ b/examples/kubernetes/README.md @@ -0,0 +1,57 @@ +# 🐳 Kubernetes Examples + +Welcome to the **Kubernetes** examples! This section provides configurations and guides to help you deploy scalable MLOps stacks on Kubernetes clusters. Kubernetes is ideal for managing containerized applications, and these examples will help you harness its power for your machine learning workflows. + +## πŸš€ Getting Started + +### Prerequisites +Before deploying the examples, make sure you have: +- **Kubernetes Cluster**: A running Kubernetes cluster (minikube, EKS, GKE, or AKS). +- **kubectl**: Command-line tool to interact with your Kubernetes cluster. [Install kubectl](https://kubernetes.io/docs/tasks/tools/install-kubectl/) +- **Helm**: Package manager for Kubernetes. [Install Helm](https://helm.sh/docs/intro/install/) +- **Terraform**: Version `>= 1.8.0` installed. [Install Terraform](https://learn.hashicorp.com/tutorials/terraform/install-cli) + +### πŸ“¦ Installation + +1. **Set Up Kubernetes Cluster**: + - If you don’t have a cluster, create one using your preferred platform. For example, using **minikube**: + ```bash + minikube start + ``` + +2. **Install `mlinfra`**: + ```bash + pip install mlinfra + ``` + +3. **Choose an Example**: + Navigate to the example directory of your choice: + ```bash + cd examples/kubernetes/lakefs + ``` + +4. **Deploy to Kubernetes**: + Use `mlinfra` to deploy the chosen configuration: + ```bash + mlinfra terraform --action apply --stack-config-path ./config/k8s-config.yaml + ``` + +## πŸ“‚ Available Examples + +| Example File | Description | +| ------------------------| ----------- | +| `aws-lakefs.yaml` | Basic configuration for deploying LakeFS on Kubernetes | +| `aws-lakefs-advanced.yaml` | Advanced configuration for LakeFS with additional cloud database storage options | + +### ℹ️ Note: +- Ensure your **VPC** has a **NAT Gateway** configured so that the node groups can access the EKS cluster. [Configure a NAT Gateway](https://docs.aws.amazon.com/vpc/latest/userguide/vpc-nat-gateway.html) +- You can choose between creating a **single NAT gateway** or **one NAT gateway per Availability Zone (AZ)**. + +## 🌟 Why Use Kubernetes? +Using Kubernetes brings several benefits: +- **Scalability**: Automatically scale containers based on load and resource utilization. +- **Portability**: Deploy your MLOps stack across multiple cloud providers or on-premise. +- **Resilience**: Kubernetes ensures your services remain available by self-healing when issues occur. + +## πŸ†˜ Need Assistance? +For help, open an issue or join our [Discord community](#) for support and to share your feedback. \ No newline at end of file diff --git a/examples/kubernetes/lakefs/README.md b/examples/kubernetes/lakefs/README.md new file mode 100644 index 00000000..4bfa7454 --- /dev/null +++ b/examples/kubernetes/lakefs/README.md @@ -0,0 +1,87 @@ +# 🌊 Deploying LakeFS on Kubernetes + +This directory contains YAML configurations for deploying LakeFS on a Kubernetes cluster. LakeFS is an open-source data version control system designed for managing and versioning data in object storage, enabling efficient data workflows. + +## πŸ“ Files Included + +- **`lakefs.yaml`**: Basic configuration for deploying LakeFS on Kubernetes. +- **`lakefs-advanced.yaml`**: Advanced configuration for deploying LakeFS with additional features and custom settings. + +## πŸš€ Getting Started + +To deploy LakeFS on a Kubernetes cluster, ensure you have the following prerequisites: + +- An active Kubernetes cluster (e.g., using EKS, GKE, AKS). +- `kubectl` installed and configured to interact with your cluster. +- Necessary permissions to create resources in the cluster. + +## 🌐 Deployment Instructions + +### 1. Basic LakeFS Setup + +The `lakefs.yaml` file provides a straightforward setup for deploying LakeFS. This configuration is suitable for initial testing and smaller workloads. + +**To deploy:** + +```bash +kubectl apply -f lakefs.yaml +``` + +This command will create the necessary Kubernetes resources and deploy LakeFS in your cluster. + +### 2. Advanced LakeFS Setup + +The `lakefs-advanced.yaml` file contains a more comprehensive configuration, which may include: + +- Enhanced scaling options +- Custom resource requests and limits +- Integrations with additional services (e.g., S3, RDS) + +**To deploy:** + +```bash +kubectl apply -f lakefs-advanced.yaml +``` + +Feel free to modify the advanced configuration according to your specific use cases and resource requirements. + +## πŸ”§ Customization Tips + +- **Storage Configuration**: Ensure you configure persistent storage for LakeFS to manage data effectively. +- **Environment Variables**: Adjust any environment variables in the YAML files for configuring LakeFS settings. +- **Service Configuration**: Modify service settings if you need to expose LakeFS externally or set up an ingress controller. + +## πŸ“¦ Managing Your Deployment + +- **Update Configuration**: To apply changes, use: + ```bash + kubectl apply -f .yaml + ``` +- **Check Status**: Monitor the deployment using: + ```bash + kubectl get pods + ``` +- **Logs**: View logs for troubleshooting: + ```bash + kubectl logs + ``` + +## 🧹 Cleanup + +To remove the LakeFS deployment, execute: + +```bash +kubectl delete -f lakefs.yaml +kubectl delete -f lakefs-advanced.yaml +``` + +## πŸ“ž Support & Resources + +For additional support, consider the following resources: + +- [LakeFS Documentation](https://docs.lakefs.io/) +- [Kubernetes Documentation](https://kubernetes.io/docs/home/) + +## License πŸ“„ + +This project is licensed under the Apache-2 License.