From cb55dc98b38ecd1349bc573ccf2ac1994d7e62cb Mon Sep 17 00:00:00 2001 From: Moses-Mk Date: Mon, 21 Oct 2024 19:59:26 +0530 Subject: [PATCH] new updates on mkdocs --- docs/code/aws/cloud_vm.md | 515 ++++++++++++++++-- docs/code/aws/kubernetes.md | 452 ++++++++++++++- docs/code/local/kind.md | 85 ++- docs/code/local/minikube.md | 91 +++- mlinfra | 1 + .../__pycache__/__init__.cpython-311.pyc | Bin 0 -> 172 bytes 6 files changed, 1052 insertions(+), 92 deletions(-) create mode 160000 mlinfra create mode 100644 src/mlinfra/__pycache__/__init__.cpython-311.pyc diff --git a/docs/code/aws/cloud_vm.md b/docs/code/aws/cloud_vm.md index 02b09551..5af286b8 100644 --- a/docs/code/aws/cloud_vm.md +++ b/docs/code/aws/cloud_vm.md @@ -1,75 +1,500 @@ `cloud_vm` deploys MLOps `stack` on top of Cloud provider VMs. +# ☁️ Cloud Infrastructure Examples +Welcome to the **Cloud Infrastructure** examples! This section contains ready-to-use configurations to help you deploy scalable MLOps stacks on cloud platforms like AWS. Each example is designed to showcase a different tool or workflow, making it easier for you to find and deploy the infrastructure that fits your needs. -## Complete example with all stacks +## πŸš€ Getting Started + +### Prerequisites +Before you begin, ensure you have the following: +- **Terraform**: Version `>= 1.8.0` installed on your system. [Install Terraform](https://learn.hashicorp.com/tutorials/terraform/install-cli) +- **AWS Account**: Make sure your AWS account is set up and you have appropriate IAM roles. +- **AWS CLI**: Ensure your AWS CLI is configured with the correct credentials. [Configure AWS CLI](https://docs.aws.amazon.com/cli/latest/userguide/cli-configure-quickstart.html) +- **Python**: Version `>= 3.7` to run the setup scripts. -=== "Simple Deployment Configuration" - ```yaml - --8<-- "docs/examples/cloud_vm/complete/aws-complete.yaml" +### πŸ“¦ Installation +1. **Create a Virtual Environment**: + ```bash + python -m venv venv + source venv/bin/activate # On Windows use: venv\Scripts\activate + ``` +2. **Install `mlinfra`**: + ```bash + pip install mlinfra ``` -=== "Advanced Deployment Configuration" - ```yaml - --8<-- "docs/examples/cloud_vm/complete/aws-complete-advanced.yaml" + +3. **Choose an Example**: + Navigate to the example you want to deploy: + ```bash + cd examples/cloud_infra/dagster ``` +4. **Configure AWS Credentials**: + Modify the `terraform.tfvars` file to update your AWS account details. Make sure your AWS credentials are correctly configured. + +5. **Deploy**: + Use `mlinfra` to apply the configuration: + ```bash + mlinfra terraform --action apply --stack-config-path ./config/aws-config.yaml + ``` + +## πŸ“‚ Examples Available + +| Example Folder | Description | +| -------------- | ------------------------------------------------------------ | +| `complete` | Complete setups for ML projects in the cloud | +| `lakefs` | Create a **LakeFS** setup for versioning your datasets in the cloud | +| `mlflow` | Set up **MLflow** for experiment tracking and model management | +| `wandb` | Configure **Weights & Biases** for experiment tracking on cloud | +| `prefect` | Set up **Prefect** for orchestrating cloud-based ML workflows | +| `dagster` | Deploy **Dagster** for orchestrating your ML pipelines on AWS | + + +## 🌟 Why Use Cloud Infrastructure? +Deploying your MLOps stack on the cloud brings several advantages: +- **Scalability**: Easily scale resources up or down based on workload. +- **Cost Efficiency**: Pay for what you use, and reduce costs with managed services. +- **Flexibility**: Leverage powerful cloud services to enhance your ML workflows. + +## πŸ†˜ Need Help? +Feel free to open an issue or join our [Discord community](#) for discussions, troubleshooting, or contributing new ideas. + + +## Complete example with all stacks + +# AWS Complete Deployment 🌐 + +This directory contains YAML configurations for setting up a complete infrastructure on AWS. These files are designed to facilitate the deployment of cloud resources, with options for both basic and advanced configurations. + +## Directory Structure + +- `aws-complete.yaml`: Basic configuration file for deploying essential cloud infrastructure on AWS. +- `aws-complete-advanced.yaml`: Advanced configuration file that includes enhanced features for scalability, security, and performance optimization. + +## Deployment Instructions + +### 1. Prerequisites + +Before you begin, make sure you have: + +- An AWS account with the necessary permissions to create and manage resources. +- AWS CLI installed and configured with your credentials. +- `cloudformation` permissions to deploy AWS infrastructure. + +### 2. Basic Deployment + +To deploy the basic infrastructure, use the following command: + +```bash +aws cloudformation deploy --template-file aws-complete.yaml --stack-name basic-aws-stack +``` + +### 3. Advanced Deployment + +For a more comprehensive setup with additional features, execute: + +```bash +aws cloudformation deploy --template-file aws-complete-advanced.yaml --stack-name advanced-aws-stack +``` + +### 4. Verifying Deployment + +To verify that your infrastructure has been set up correctly, check the status of your CloudFormation stack: + +```bash +aws cloudformation describe-stacks --stack-name basic-aws-stack +``` + +Or for the advanced setup: + +```bash +aws cloudformation describe-stacks --stack-name advanced-aws-stack +``` + +### 5. Managing Resources + +You can manage and monitor your deployed resources through the AWS Management Console. Use the CloudFormation section to view your stacks, events, and outputs for further details. + +## Additional Resources + +- [AWS CloudFormation Documentation](https://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/Welcome.html) +- [AWS CLI Documentation](https://docs.aws.amazon.com/cli/latest/userguide/cli-configure-quickstart.html) + +## License πŸ“„ + +This project is licensed under the Apache-2 License. + + ## data_versioning #### lakefs -=== "Simple Deployment Configuration" - ```yaml - --8<-- "docs/examples/cloud_vm/lakefs/aws-lakefs.yaml" - ``` -=== "Advanced Deployment Configuration" - ```yaml - --8<-- "docs/examples/cloud_vm/lakefs/aws-lakefs-advanced.yaml" - ``` +# 🌊 AWS LakeFS Configuration + +This directory contains configuration files for deploying LakeFS on AWS. LakeFS is an open-source data version control system designed for object storage, enabling you to manage and version your data efficiently. + +## πŸ“ Files Included + +- **`aws-lakefs.yaml`**: Basic configuration for deploying LakeFS on AWS. +- **`aws-lakefs-advanced.yaml`**: Advanced configuration for deploying LakeFS with additional features and custom settings. + +## πŸš€ Getting Started + +To use these configurations, ensure you have the following prerequisites: + +- An active AWS account. +- AWS Command Line Interface (CLI) installed and configured. +- Necessary IAM permissions for creating resources. +- Python 3.x installed. + +### πŸ“¦ Installation + +To install the `platinfra` package, you can use pip. It's recommended to create a Python virtual environment first: + +```bash +python -m venv venv +source venv/bin/activate # On Windows use `venv\Scripts\activate` +pip install platinfra +``` + +## 🌐 Deployment Instructions + +### 1. Basic LakeFS Setup + +The `aws-lakefs.yaml` file provides a straightforward setup for deploying LakeFS. This is suitable for initial testing and smaller workloads. + +**To deploy:** + +```bash +platinfra terraform --action apply --stack-config-path aws-lakefs.yaml +``` + +This command will create the necessary AWS resources and deploy LakeFS. + +### 2. Advanced LakeFS Setup + +The `aws-lakefs-advanced.yaml` file contains a more comprehensive configuration, including features like: + +- Enhanced scaling options +- Custom resource requests and limits +- Integrations with additional AWS services + +**To deploy:** + +```bash +platinfra terraform --action apply --stack-config-path aws-lakefs-advanced.yaml +``` + +Feel free to modify the advanced configuration according to your specific use cases and resource requirements. + +## πŸ”§ Customization Tips + +- **IAM Roles**: Ensure that the necessary IAM roles and policies are in place to allow LakeFS to access required AWS resources, such as S3. +- **Environment Variables**: Adjust environment variables in the YAML files for configuring LakeFS settings. +- **Storage Configuration**: Make sure to configure persistent storage options if needed. + +## πŸ“¦ Managing Your Deployment + +- **Update Configuration**: To apply changes, use: + ```bash + platinfra terraform --action apply --stack-config-path .yaml + ``` +- **Check Status**: Monitor the deployment using: + ```bash + aws cloudformation describe-stacks --stack-name lakefs-stack + ``` +- **Logs**: View logs for troubleshooting in the AWS Console. + +## 🧹 Cleanup + +To remove the LakeFS deployment, execute: + +```bash +platinfra terraform --action destroy --stack-config-path aws-lakefs.yaml +platinfra terraform --action destroy --stack-config-path aws-lakefs-advanced.yaml +``` + +## πŸ“ž Support & Resources + +For additional support, consider the following resources: + +- [LakeFS Documentation](https://docs.lakefs.io/) +- [AWS Documentation](https://aws.amazon.com/documentation/) + ## experiment_tracking #### mlflow -=== "Simple Deployment Configuration" - ```yaml - --8<-- "docs/examples/cloud_vm/mlflow/aws-mlflow.yaml" - ``` -=== "Advanced Deployment Configuration" - ```yaml - --8<-- "docs/examples/cloud_vm/mlflow/aws-mlflow-advanced.yaml" - ``` +# MLflow on AWS Deployment πŸš€ + +This README provides guidance on deploying MLflow on AWS using the provided configuration files in the `mlflow` folder. + +## Overview + +MLflow is an open-source platform to manage the ML lifecycle, including experimentation, reproducibility, and deployment. This setup allows you to deploy MLflow on AWS easily. + +## Requirements + +Before you start, ensure you have the following: + +- An AWS account with the necessary permissions to create and manage resources. +- Terraform installed on your system (version >= 1.4.0). + +## Installation + +1. **Create a Python Virtual Environment:** + + ```bash + python -m venv venv + source venv/bin/activate + ``` + +2. **Install the Required Packages:** + + ```bash + pip install platinfra + ``` + +## Deployment Configuration + +You can choose between two deployment configurations based on your needs: + +### 1. Basic MLflow Deployment + +- **File:** `aws-mlflow.yaml` + +This configuration is suitable for a simple MLflow deployment with essential features. To deploy: + +```bash +platinfra terraform --action apply --stack-config-path /aws-mlflow.yaml +``` + +### 2. Advanced MLflow Deployment + +- **File:** `aws-mlflow-advanced.yaml` + +This configuration includes additional features for a more robust MLflow deployment. To deploy: + +```bash +platinfra terraform --action apply --stack-config-path /aws-mlflow-advanced.yaml +``` + +## Configuration + +- Update the `aws-mlflow.yaml` or `aws-mlflow-advanced.yaml` files with your AWS account details and any other custom configurations you require. +- Ensure your AWS credentials are properly configured on your machine. + +## Additional Information + +For more details on using MLflow and its capabilities, refer to the [MLflow Documentation](https://www.mlflow.org/docs/latest/index.html). + +## Contribution + +Feel free to contribute to this project! If you have suggestions or improvements, open an issue or pull request. + +## License + +This project is licensed under the Apache-2.0 License. + #### wandb -=== "Simple Deployment Configuration" - ```yaml - --8<-- "docs/examples/cloud_vm/wandb/aws-wandb.yaml" - ``` -=== "Advanced Deployment Configuration" - ```yaml - --8<-- "docs/examples/cloud_vm/wandb/aws-wandb-advanced.yaml" - ``` +# Weights and Biases Deployment on AWS 🌊 + +This directory contains YAML configurations for deploying Weights and Biases (Wand) on AWS. Weights and Biases is a powerful tool designed for managing and monitoring machine learning experiments, helping teams to track metrics and visualize results effectively. + +## Directory Structure + +- **`aws-wand.yaml`**: This file contains the basic configuration for deploying Weights and Biases on AWS. +- **`aws-wand-advanced.yaml`**: This file includes advanced configurations for a more robust Weights and Biases deployment, incorporating features for scalability and performance. + +## Deployment Instructions + +### 1. Prerequisites + +Before deploying, ensure you have the following: + +- An active AWS account with the necessary permissions. +- AWS CLI configured with your credentials. +- Python 3.x installed. + +### πŸ“¦ Installation + +To install the `platinfra` package, you can use pip. It's recommended to create a Python virtual environment first: + +```bash +python -m venv venv +source venv/bin/activate # On Windows use `venv\Scripts\activate` +pip install platinfra +``` + +### 2. Basic Deployment + +To deploy using the basic configuration, run: + +```bash +platinfra terraform --action apply --stack-config-path aws-wand.yaml +``` + +### 3. Advanced Deployment + +For an advanced setup with additional features, execute: + +```bash +platinfra terraform --action apply --stack-config-path aws-wand-advanced.yaml +``` + +### 4. Verifying Deployment + +After applying the configurations, check the status of the Weights and Biases services: + +```bash +aws cloudformation describe-stacks --stack-name wand-stack +``` + +### 5. Accessing Weights and Biases UI + +You can access the Weights and Biases UI to monitor your machine learning experiments. Depending on your service configuration, you may need to set up an appropriate networking method, such as an Elastic Load Balancer (ELB) or an ingress point. + +Once set up, visit the ELB URL or the endpoint you configured to access the Weights and Biases UI. + +## Additional Resources + +- [Weights and Biases Documentation](https://docs.wandb.ai/) +- [AWS CLI Documentation](https://docs.aws.amazon.com/cli/latest/userguide/cli-configure-quickstart.html) + +## License πŸ“„ + +This project is licensed under the Apache-2 License. ## orchestrator #### prefect -=== "Simple Deployment Configuration" - ```yaml - --8<-- "docs/examples/cloud_vm/prefect/aws-prefect.yaml" - ``` -=== "Advanced Deployment Configuration" - ```yaml - --8<-- "docs/examples/cloud_vm/prefect/aws-prefect-advanced.yaml" - ``` +# Deploying Prefect on AWS ☁️ + +This guide provides instructions for deploying Prefect on Amazon Web Services (AWS) using the provided YAML configuration files. + +## Overview πŸ“Š + +Prefect is a modern workflow orchestration tool designed to enable efficient data pipeline management. This deployment will help you run Prefect on AWS, providing scalability and reliability for your workflows. + +## Requirements βœ”οΈ + +Before you start, ensure you have the following: +- **Terraform**: Version **>= 1.4.0** installed on your machine. +- **AWS Account**: A valid AWS account with necessary permissions to create resources. + +## Installation βš™οΈ + +1. **Create a Python Virtual Environment:** + ```bash + python -m venv venv + source venv/bin/activate + ``` + +2. **Install the Required Python Package:** + ```bash + pip install platinfra + ``` + +## Deployment Configuration πŸ“„ + +In the `prefect` folder, you have the following configuration files: + +1. **aws-prefect.yaml**: Basic configuration for deploying Prefect on AWS. +2. **aws-prefect-advanced.yaml**: Advanced configuration with additional settings and optimizations. + +### Configuration File Details + +- **aws-prefect.yaml**: This file sets up a simple Prefect server instance on AWS with default settings. + +- **aws-prefect-advanced.yaml**: This file includes advanced features, such as a larger instance type, additional storage options, and monitoring configurations for enhanced performance. + +## Running the Deployment πŸš€ + +To deploy Prefect on AWS, follow these steps: + +1. **Copy the Configuration File:** + Choose one of the configuration files (basic or advanced) and modify it with your AWS account details and desired configurations. + +2. **Deploy the Configuration:** + Run the following command to apply the configuration: + ```bash + platinfra terraform --action apply --stack-config-path + ``` + Replace `` with the path to either `aws-prefect.yaml` or `aws-prefect-advanced.yaml`. + +## Conclusion πŸŽ‰ + +Congratulations! You have successfully deployed Prefect on AWS. You can now start orchestrating your workflows in the cloud. For further configurations and integrations, refer to the Prefect documentation. #### dagster -=== "Simple Deployment Configuration" - ```yaml - --8<-- "docs/examples/cloud_vm/dagster/aws-dagster.yaml" +# πŸš€ Dagster on AWS Cloud Setup + +Welcome to the **Dagster AWS Cloud Setup** documentation! This guide will help you deploy Dagster on AWS using the provided configurations for a seamless data orchestration experience. Let's get started! πŸ› οΈ + +## πŸ“‚ Configuration Files Overview + +### 1. `aws-dagster.yaml` +This file provides a **basic setup** for deploying Dagster on AWS. It includes essential configurations to quickly get you up and running with Dagster. Use this file if you are just starting out or need a straightforward deployment. + +**Features:** +- πŸ—οΈ Basic infrastructure setup +- 🧩 Core components of Dagster +- πŸ”Œ Integration with AWS services like S3 and CloudWatch + +### 2. `aws-dagster-advanced.yaml` +This file offers a **more advanced setup** for deploying Dagster on AWS, with additional configurations for better scalability, security, and performance. Use this file if you need more control and features for your deployment. + +**Features:** +- πŸ“ˆ Enhanced scalability and performance configurations +- πŸ”’ Advanced security settings +- πŸ”„ Custom integrations and multi-environment support + +## πŸ“¦ Prerequisites + +Before deploying Dagster on AWS, make sure you have: +1. 🏷️ AWS Account with appropriate permissions. +2. πŸ—οΈ [AWS CLI](https://aws.amazon.com/cli/) installed and configured. +3. 🧩 [Dagster](https://dagster.io/) installed on your local machine. +4. πŸ”§ Proper IAM roles and security groups set up. + +## πŸš€ Deployment Instructions + +### Basic Setup (`aws-dagster.yaml`) +1. **Configure your AWS CLI:** + ```bash + aws configure + ``` +2. **Deploy the infrastructure:** + ```bash + aws cloudformation deploy --template-file aws-dagster.yaml --stack-name DagsterBasicSetup ``` -=== "Advanced Deployment Configuration" - ```yaml - --8<-- "docs/examples/cloud_vm/dagster/aws-dagster-advanced.yaml" +3. πŸŽ‰ **You're all set!** Access Dagster via the provided endpoint. + +### Advanced Setup (`aws-dagster-advanced.yaml`) +1. **Configure your AWS CLI:** + ```bash + aws configure + ``` +2. **Deploy the infrastructure:** + ```bash + aws cloudformation deploy --template-file aws-dagster-advanced.yaml --stack-name DagsterAdvancedSetup ``` +3. **Customize the parameters if needed:** + Modify the `.yaml` file to suit your specific needs, including scaling options, security settings, and more. +4. πŸŽ‰ **Enjoy your advanced setup!** + +## πŸ“ Notes +- Ensure the **VPC settings** and **subnets** are correctly configured to avoid deployment issues. +- For the advanced setup, you might need to adjust **IAM roles** and **network configurations**. + +## πŸ“š Additional Resources +- πŸ“– [Dagster Documentation](https://docs.dagster.io/) +- πŸ“– [AWS CloudFormation Documentation](https://docs.aws.amazon.com/cloudformation/) + +Feel free to customize the `.yaml` files to suit your specific requirements. πŸ› οΈ Happy deploying! \ No newline at end of file diff --git a/docs/code/aws/kubernetes.md b/docs/code/aws/kubernetes.md index 7e3a7451..61a63d0a 100644 --- a/docs/code/aws/kubernetes.md +++ b/docs/code/aws/kubernetes.md @@ -1,46 +1,442 @@ `kubernetes` deploys MLOps `stack` on top of Cloud provider's kubernetes. In case of AWS, its EKS. +# 🐳 Kubernetes Examples -#### Complete +Welcome to the **Kubernetes** examples! This section provides configurations and guides to help you deploy scalable MLOps stacks on Kubernetes clusters. Kubernetes is ideal for managing containerized applications, and these examples will help you harness its power for your machine learning workflows. -===+ "Simple Deployment Configuration" - ```yaml - --8<-- "docs/examples/kubernetes/complete/aws-complete.yaml" +## πŸš€ Getting Started + +### Prerequisites +Before deploying the examples, make sure you have: +- **Kubernetes Cluster**: A running Kubernetes cluster (minikube, EKS, GKE, or AKS). +- **kubectl**: Command-line tool to interact with your Kubernetes cluster. [Install kubectl](https://kubernetes.io/docs/tasks/tools/install-kubectl/) +- **Helm**: Package manager for Kubernetes. [Install Helm](https://helm.sh/docs/intro/install/) +- **Terraform**: Version `>= 1.8.0` installed. [Install Terraform](https://learn.hashicorp.com/tutorials/terraform/install-cli) + +### πŸ“¦ Installation + +1. **Set Up Kubernetes Cluster**: + - If you don’t have a cluster, create one using your preferred platform. For example, using **minikube**: + ```bash + minikube start + ``` + +2. **Install `mlinfra`**: + ```bash + pip install mlinfra ``` -=== "Advanced Deployment Configuration" - ```yaml - --8<-- "docs/examples/kubernetes/complete/aws-complete-advanced.yaml" + +3. **Choose an Example**: + Navigate to the example directory of your choice: + ```bash + cd examples/kubernetes/lakefs ``` +4. **Deploy to Kubernetes**: + Use `mlinfra` to deploy the chosen configuration: + ```bash + mlinfra terraform --action apply --stack-config-path ./config/k8s-config.yaml + ``` + +## πŸ“‚ Available Examples + +| Example File | Description | +| ------------------------| ----------- | +## πŸ“‚ Available Examples + +| Example File | Description | +| ------------------------| ------------------------------------------------------------- | +| `Complete` | Complete configuration for deploying ML projects on Kubernetes | +| `LakeFS` | Basic configuration for deploying LakeFS on Kubernetes | +| `LakeFS Advanced` | Advanced configuration for LakeFS with additional cloud database storage options | +| `MLflow` | Configuration for deploying MLflow on Kubernetes | +| `Prefect` | Configuration for deploying Prefect on Kubernetes | + + +### ℹ️ Note: +- Ensure your **VPC** has a **NAT Gateway** configured so that the node groups can access the EKS cluster. [Configure a NAT Gateway](https://docs.aws.amazon.com/vpc/latest/userguide/vpc-nat-gateway.html) +- You can choose between creating a **single NAT gateway** or **one NAT gateway per Availability Zone (AZ)**. + +## 🌟 Why Use Kubernetes? +Using Kubernetes brings several benefits: +- **Scalability**: Automatically scale containers based on load and resource utilization. +- **Portability**: Deploy your MLOps stack across multiple cloud providers or on-premise. +- **Resilience**: Kubernetes ensures your services remain available by self-healing when issues occur. + +## πŸ†˜ Need Assistance? +For help, open an issue or join our [Discord community](#) for support and to share your feedback. + +#### Complete + +# Complete Deployment on Kubernetes πŸš€ + +This directory contains YAML configurations for deploying a complete solution on a Kubernetes cluster. These configurations are designed to help you set up an entire application stack using Kubernetes. + +## πŸ“‚ Files Included + +- **`aws-complete.yaml`**: Basic configuration for deploying a complete solution on a Kubernetes cluster. +- **`aws-complete-advanced.yaml`**: Advanced configuration for deploying a complete solution with additional features and optimizations. + +## βš™οΈ Prerequisites + +Before starting, ensure you have: + +- An existing Kubernetes cluster (e.g., Amazon EKS, GKE, AKS, or any self-managed cluster). +- `kubectl` installed and configured to connect to your Kubernetes cluster. +- Necessary permissions for creating resources in the Kubernetes cluster. +- Docker images for your application available, or use existing images. + +## πŸ“¦ Deployment Instructions + +### 1. Basic Complete Deployment + +To deploy the complete solution using the basic setup, run: + +```bash +kubectl apply -f aws-complete.yaml +``` + +This configuration is suitable for basic usage and testing. + +### 2. Advanced Complete Deployment + +For a more robust deployment that includes optimizations and additional features, use the advanced configuration: + +```bash +kubectl apply -f aws-complete-advanced.yaml +``` + +This file allows for customizations such as: + +- **Increased resource requests/limits for better performance** +- **Integration with external services** +- **Enhanced monitoring and logging features** + +### 3. Verifying Your Deployment + +To check the status of your deployments, use: + +```bash +kubectl get pods +``` + +You can also check the services to see if they are running correctly: + +```bash +kubectl get svc +``` + +### 4. Accessing the Application + +Once deployed, you can access your application via the service endpoints. If the services are not exposed externally, you may need to set up port forwarding: + +```bash +kubectl port-forward svc/ 8080:80 +``` + +Replace `` with the name of your service. Visit `http://localhost:8080` to access your application. + +## πŸ”§ Customization Tips + +- **Scaling**: Modify replica counts in the YAML files to adjust the number of instances running. +- **Resource Allocation**: Tweak resource requests and limits based on application needs. +- **Environment Variables**: Set any required environment variables directly in the YAML files. + +## 🧹 Cleanup + +To delete your complete deployment: + +```bash +kubectl delete -f aws-complete.yaml +kubectl delete -f aws-complete-advanced.yaml +``` + +## πŸ“„ Additional Resources + +- [Kubernetes Documentation](https://kubernetes.io/docs/) +- [AWS Documentation](https://aws.amazon.com/documentation/) + #### lakefs -===+ "Simple Deployment Configuration" - ```yaml - --8<-- "docs/examples/kubernetes/lakefs/aws-lakefs.yaml" - ``` -=== "Advanced Deployment Configuration" - ```yaml - --8<-- "docs/examples/kubernetes/lakefs/aws-lakefs-advanced.yaml" - ``` +# 🌊 Deploying LakeFS on Kubernetes + +This directory contains YAML configurations for deploying LakeFS on a Kubernetes cluster. LakeFS is an open-source data version control system designed for managing and versioning data in object storage, enabling efficient data workflows. + +## πŸ“ Files Included + +- **`lakefs.yaml`**: Basic configuration for deploying LakeFS on Kubernetes. +- **`lakefs-advanced.yaml`**: Advanced configuration for deploying LakeFS with additional features and custom settings. + +## πŸš€ Getting Started + +To deploy LakeFS on a Kubernetes cluster, ensure you have the following prerequisites: + +- An active Kubernetes cluster (e.g., using EKS, GKE, AKS). +- `kubectl` installed and configured to interact with your cluster. +- Necessary permissions to create resources in the cluster. + +## 🌐 Deployment Instructions + +### 1. Basic LakeFS Setup + +The `lakefs.yaml` file provides a straightforward setup for deploying LakeFS. This configuration is suitable for initial testing and smaller workloads. + +**To deploy:** + +```bash +kubectl apply -f lakefs.yaml +``` + +This command will create the necessary Kubernetes resources and deploy LakeFS in your cluster. + +### 2. Advanced LakeFS Setup + +The `lakefs-advanced.yaml` file contains a more comprehensive configuration, which may include: + +- Enhanced scaling options +- Custom resource requests and limits +- Integrations with additional services (e.g., S3, RDS) + +**To deploy:** + +```bash +kubectl apply -f lakefs-advanced.yaml +``` + +Feel free to modify the advanced configuration according to your specific use cases and resource requirements. + +## πŸ”§ Customization Tips + +- **Storage Configuration**: Ensure you configure persistent storage for LakeFS to manage data effectively. +- **Environment Variables**: Adjust any environment variables in the YAML files for configuring LakeFS settings. +- **Service Configuration**: Modify service settings if you need to expose LakeFS externally or set up an ingress controller. + +## πŸ“¦ Managing Your Deployment + +- **Update Configuration**: To apply changes, use: + ```bash + kubectl apply -f .yaml + ``` +- **Check Status**: Monitor the deployment using: + ```bash + kubectl get pods + ``` +- **Logs**: View logs for troubleshooting: + ```bash + kubectl logs + ``` + +## 🧹 Cleanup + +To remove the LakeFS deployment, execute: + +```bash +kubectl delete -f lakefs.yaml +kubectl delete -f lakefs-advanced.yaml +``` + +## πŸ“ž Support & Resources + +For additional support, consider the following resources: + +- [LakeFS Documentation](https://docs.lakefs.io/) +- [Kubernetes Documentation](https://kubernetes.io/docs/home/) + +## License πŸ“„ + +This project is licensed under the Apache-2 License. + #### mlflow ===+ "Simple Deployment Configuration" - ```yaml - --8<-- "docs/examples/kubernetes/mlflow/aws-mlflow.yaml" +# MLflow Deployment on AWS πŸš€ + +This directory contains configuration files for deploying [MLflow](https://mlflow.org/), an open-source platform for managing the end-to-end machine learning lifecycle, on AWS infrastructure. + +## Directory Structure + +- `aws-mlflow.yaml`: Basic setup for deploying MLflow on AWS. +- `aws-mlflow-advanced.yaml`: Advanced setup with additional configurations for enhanced performance and scalability on AWS. + +## Prerequisites + +Before deploying MLflow on AWS, ensure you have: + +- An AWS account with sufficient permissions to create resources (e.g., EC2, S3, RDS). +- AWS CLI installed and configured (`aws configure`). +- `kubectl` installed and set up to interact with your Kubernetes cluster on AWS (e.g., EKS). +- Optional: Docker installed locally if you plan to build custom MLflow images. + +## Deployment Instructions + +### 1. Set Up AWS Resources + +Ensure that the following AWS resources are set up: + +- **S3 Bucket**: For storing MLflow artifacts. Create a bucket or use an existing one. +- **RDS (Optional)**: For storing MLflow experiment metadata. Alternatively, use a local or other supported database. +- **IAM Role**: Make sure your EC2 or EKS nodes have the necessary IAM roles to access S3, RDS, and any other AWS services you plan to use. + +### 2. Basic MLflow Deployment on AWS + +Use the `aws-mlflow.yaml` file for a straightforward MLflow deployment on AWS. + +1. Update the `aws-mlflow.yaml` file to include your S3 bucket information and any environment variables. +2. Apply the deployment using `kubectl`: + + ```bash + kubectl apply -f aws-mlflow.yaml ``` -=== "Advanced Deployment Configuration" - ```yaml - --8<-- "docs/examples/kubernetes/mlflow/aws-mlflow-advanced.yaml" + +This will create an MLflow server, using S3 as the artifact store and a local backend for experiment metadata (or any database you configure). + +### 3. Advanced MLflow Deployment on AWS + +For more advanced deployment, use `aws-mlflow-advanced.yaml`. This setup may include features like: + +- Autoscaling for the MLflow server. +- Enhanced resource allocation for performance. +- Integration with managed databases (e.g., RDS). +- Custom Docker images. + +1. Update the `aws-mlflow-advanced.yaml` file as needed: + - Specify environment variables, including database connection strings. + - Adjust resource requests and limits. + - Configure autoscaling if required. +2. Apply the deployment: + + ```bash + kubectl apply -f aws-mlflow-advanced.yaml ``` +### 4. Configure Access to MLflow + +- **NodePort**: Use NodePort in your YAML files to expose the MLflow server on a specific port. +- **LoadBalancer**: Expose the MLflow server using a LoadBalancer service for easier access from outside the cluster. +- **Ingress**: For more control over routing and domain configuration, use an Ingress controller. + +After deployment, check the status: + +```bash +kubectl get pods +kubectl get services +``` + +### 5. Customize MLflow Configuration + +You can modify environment variables in the YAML files to adjust the following: + +- **Backend Store**: Specify a database connection (e.g., MySQL, PostgreSQL, or AWS RDS) for experiment tracking. +- **Artifact Store**: Use S3 by configuring the `MLFLOW_S3_ENDPOINT_URL` and `AWS_ACCESS_KEY_ID` and `AWS_SECRET_ACCESS_KEY` if needed. +- **Environment Variables**: Configure authentication and other settings as per your requirements. + +### 6. Monitoring and Logging + +To view the logs of the MLflow server: + +```bash +kubectl logs +``` + +For real-time monitoring, consider integrating with AWS CloudWatch. + +## Additional Resources + +- [MLflow Documentation](https://mlflow.org/docs/latest/index.html) +- [AWS Documentation](https://docs.aws.amazon.com/) +- [Kubectl Cheat Sheet](https://kubernetes.io/docs/reference/kubectl/cheatsheet/) + +## License πŸ“„ + +This project is licensed under the Apache-2 License. + #### Prefect (Server & Worker) -===+ "Simple Deployment Configuration" - ```yaml - --8<-- "docs/examples/kubernetes/prefect/aws-prefect.yaml" - ``` -=== "Advanced Deployment Configuration" - ```yaml - --8<-- "docs/examples/kubernetes/prefect/aws-prefect-advanced.yaml" - ``` +# Prefect Deployment on Kubernetes πŸš€ + +This directory contains YAML configurations for deploying Prefect on a Kubernetes cluster. Prefect is a workflow orchestration tool that allows you to schedule, monitor, and manage data pipelines with ease. + +## πŸ“‚ Files Included + +- **`aws-prefect.yaml`**: Basic configuration for deploying Prefect on a Kubernetes cluster. +- **`aws-prefect-advanced.yaml`**: Advanced configuration for deploying Prefect with additional customizations and features. + +## βš™οΈ Prerequisites + +Before starting, ensure you have: + +- An existing Kubernetes cluster (e.g., Amazon EKS, GKE, AKS, or any self-managed cluster). +- `kubectl` installed and configured to connect to your Kubernetes cluster. +- Necessary permissions for creating resources in the Kubernetes cluster. +- Docker images for Prefect available, or use the official Prefect images. + +## πŸ“¦ Deployment Instructions + +### 1. Basic Prefect Deployment + +The `aws-prefect.yaml` file provides a straightforward setup for deploying Prefect. This configuration is ideal for small to medium workloads. + +To deploy Prefect using the basic setup: + +```bash +kubectl apply -f aws-prefect.yaml +``` + +### 2. Advanced Prefect Deployment + +For a more robust deployment with enhanced features like resource allocation, autoscaling, or custom environment variables, use the `aws-prefect-advanced.yaml`: + +```bash +kubectl apply -f aws-prefect-advanced.yaml +``` + +This file can be modified to include specific customizations such as: + +- **Increased resource requests/limits for scalability** +- **Integration with external monitoring or logging systems** +- **Custom environment variables and secrets management** + +### 3. Verifying Your Deployment + +To check if your Prefect pods are running correctly, use: + +```bash +kubectl get pods +``` + +You can also describe the Prefect services to get more information: + +```bash +kubectl get svc +``` + +### 4. Accessing the Prefect UI + +Once Prefect is deployed, you can access the Prefect UI for monitoring workflows. You may need to set up port forwarding if the service is not exposed externally: + +```bash +kubectl port-forward svc/prefect-service 8080:80 +``` + +Visit `http://localhost:8080` to access the Prefect UI. + +## πŸ”§ Customization Tips + +- **Scaling**: Modify replica counts in the YAML files to scale Prefect components as needed. +- **Resource Allocation**: Adjust resource requests and limits for memory and CPU based on workload requirements. +- **Secrets**: Configure any necessary secrets for external services directly in Kubernetes or using a tool like `kubectl` or `helm`. + +## 🧹 Cleanup + +To delete your Prefect deployment: + +```bash +kubectl delete -f aws-prefect.yaml +kubectl delete -f aws-prefect-advanced.yaml +``` + +## πŸ“„ Additional Resources + +- [Prefect Documentation](https://docs.prefect.io/) +- [Kubernetes Documentation](https://kubernetes.io/docs/) \ No newline at end of file diff --git a/docs/code/local/kind.md b/docs/code/local/kind.md index dd2ac84f..b064e557 100644 --- a/docs/code/local/kind.md +++ b/docs/code/local/kind.md @@ -1,13 +1,80 @@ `kind` deploys MLOps `stack` on top of [kind cluster](https://kind.sigs.k8s.io/) on your machine. Please ensure that docker is installed and running on your machine, along with the latest version of kind. +# Kind Deployment in Local Environment πŸ–₯️ -#### Complete +This directory contains YAML configurations for deploying applications using **Kind** (Kubernetes IN Docker). Kind allows you to run Kubernetes clusters in Docker containers, making it an excellent tool for local development and testing. -===+ "Simple Deployment Configuration" - ```yaml - --8<-- "docs/examples/local/kind.yaml" - ``` -=== "Advanced Deployment Configuration" - ```yaml - --8<-- "docs/examples/local/kind-advanced.yaml" - ``` +## πŸ“‚ Files Included + +- **`kind.yaml`**: Basic configuration for deploying an application using Kind. +- **`kind-advanced.yaml`**: Advanced configuration with optimizations and additional features for a more robust Kind deployment. + +## βš™οΈ Prerequisites + +Before you begin, ensure you have the following: + +- **Docker**: Installed and running on your machine. +- **Kind**: Installed. You can install Kind by following the instructions on the [Kind installation page](https://kind.sigs.k8s.io/docs/user/quick-start/#installation). +- **kubectl**: Installed to interact with your Kind cluster. + +## πŸš€ Deployment Instructions + +### 1. Create a Kind Cluster + +To create a Kind cluster using the basic configuration, run the following command: + +```bash +kind create cluster --config kind.yaml +``` + +For an advanced setup, use: + +```bash +kind create cluster --config kind-advanced.yaml +``` + +### 2. Verifying Your Cluster + +To check the status of your Kind cluster, run: + +```bash +kubectl cluster-info +``` + +You should see details about the Kubernetes cluster and its components. + +### 3. Deploying Applications + +Once the cluster is up and running, you can deploy your applications using `kubectl`. For example, to apply a deployment configuration file: + +```bash +kubectl apply -f .yaml +``` + +### 4. Accessing Your Application + +If you need to access your application, you can use port forwarding: + +```bash +kubectl port-forward svc/ 8080:80 +``` + +Replace `` with the name of your service. Access your application at `http://localhost:8080`. + +## πŸ”§ Customization Tips + +- **Cluster Configuration**: Modify the YAML files to adjust resource allocations, networking, or add additional features as needed. +- **Docker Resources**: Ensure your Docker daemon has enough resources (CPU, memory) allocated to handle the Kind cluster. + +## 🧹 Cleanup + +To delete your Kind cluster when you are finished, run: + +```bash +kind delete cluster +``` + +## πŸ“„ Additional Resources + +- [Kind Documentation](https://kind.sigs.k8s.io/docs/) +- [Kubernetes Documentation](https://kubernetes.io/docs/) \ No newline at end of file diff --git a/docs/code/local/minikube.md b/docs/code/local/minikube.md index e24143be..f6244550 100644 --- a/docs/code/local/minikube.md +++ b/docs/code/local/minikube.md @@ -1,15 +1,86 @@ `minikube` deploys MLOps `stack` on top of [minikube cluster](https://minikube.sigs.k8s.io/) on your machine. Please ensure that docker is installed and running on your machine, along with the latest version of minikube. +# Minikube Deployment in Local Environment πŸš€ -#### Complete +This directory contains YAML configurations for deploying applications using **Minikube**. Minikube is a tool that enables you to run Kubernetes clusters locally, making it an excellent choice for development and testing. -===+ "Simple Deployment Configuration" - ```yaml - --8<-- "docs/examples/local/minikube.yaml" - ``` -=== "Advanced Deployment Configuration" - ```yaml - --8<-- "docs/examples/local/minikube-advanced.yaml" - ``` +## πŸ“‚ Files Included -- Once the stacks have been deployed, there is one last step that allows the deployed stacks to be accessed on your local machine. You need to open a new terminal window and find the minikube profile that you're using `minikube profile list` and run `minikube tunnel --profile=''` to allow the services to be accessible on the local instance. +- **`minikube.yaml`**: Basic configuration for deploying an application using Minikube. +- **`minikube-advanced.yaml`**: Advanced configuration with optimizations and additional features for a more robust Minikube deployment. + +## βš™οΈ Prerequisites + +Before you begin, ensure you have the following: + +- **Docker**: Installed and running on your machine (as Minikube can use Docker as a driver). +- **Minikube**: Installed. You can install Minikube by following the instructions on the [Minikube installation page](https://minikube.sigs.k8s.io/docs/start/). +- **kubectl**: Installed to interact with your Minikube cluster. + +## πŸš€ Deployment Instructions + +### 1. Start Minikube + +To start a Minikube cluster using the basic configuration, run the following command: + +```bash +minikube start --config minikube.yaml +``` + +For an advanced setup, use: + +```bash +minikube start --config minikube-advanced.yaml +``` + +### 2. Verifying Your Cluster + +To check the status of your Minikube cluster, run: + +```bash +minikube status +``` + +You should see details about the Minikube cluster and its components. + +### 3. Deploying Applications + +Once the cluster is up and running, you can deploy your applications using `kubectl`. For example, to apply a deployment configuration file: + +```bash +kubectl apply -f .yaml +``` + +### 4. Accessing Your Application + +To access your application, you can use: + +```bash +minikube service +``` + +Replace `` with the name of your service. This command will open your default web browser and direct you to your application. + +## πŸ”§ Customization Tips + +- **Cluster Configuration**: Modify the YAML files to adjust resource allocations, networking, or add additional features as needed. +- **Minikube Configurations**: You can adjust Minikube settings such as the driver or resources using: + +```bash +minikube config set +``` + +## 🧹 Cleanup + +To delete your Minikube cluster when you are finished, run: + +```bash +minikube delete +``` + +## πŸ“„ Additional Resources + +- [Minikube Documentation](https://minikube.sigs.k8s.io/docs/) +- [Kubernetes Documentation](https://kubernetes.io/docs/) + +Once the stacks have been deployed, there is one last step that allows the deployed stacks to be accessed on your local machine. You need to open a new terminal window and find the minikube profile that you're using `minikube profile list` and run `minikube tunnel --profile=''` to allow the services to be accessible on the local instance. diff --git a/mlinfra b/mlinfra new file mode 160000 index 00000000..41a320c9 --- /dev/null +++ b/mlinfra @@ -0,0 +1 @@ +Subproject commit 41a320c91ba5885e8c2c00e999227a37b9c5ccca diff --git a/src/mlinfra/__pycache__/__init__.cpython-311.pyc b/src/mlinfra/__pycache__/__init__.cpython-311.pyc new file mode 100644 index 0000000000000000000000000000000000000000..e5d2038e7df5e0fadbbd3f2f732bf4afb5204baf GIT binary patch literal 172 zcmZ3^%ge<81brL%(%YCA7#@Q-Fu(|9d}aeOrZc241T$zd`mJOr0xD$q43hk1pKKKq zTAW%`9OIi`oLa2lo0wgmni=DgpIn-onpaXBlbe&7msXS*Q(Tk`XUE59=4F<|$LkeT l{^GF7%}*)KNwq6t1)2x4v6vr7d|+l|Wc