diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md
index f774b84d05..e085749b68 100644
--- a/CONTRIBUTING.md
+++ b/CONTRIBUTING.md
@@ -1,3 +1,3 @@
# Contributing to Flyte
-For information related to contributing to Flyte, please check out the [Contributing to Flyte](https://docs.flyte.org/en/latest/community/contribute.html) section of the documentation at the Flyte site.
+For information related to contributing to Flyte, please check out the [Contributing to Flyte](https://docs.flyte.org/en/latest/community/contribute.html) section of the documentation.
diff --git a/Makefile b/Makefile
index 290669ef0e..6acccb1452 100644
--- a/Makefile
+++ b/Makefile
@@ -56,10 +56,6 @@ install-piptools: ## Install pip-tools
doc-requirements.txt: doc-requirements.in install-piptools
$(call PIP_COMPILE,doc-requirements.in)
-.PHONY: requirements.txt
-requirements.txt: requirements.in install-piptools
- $(call PIP_COMPILE,requirements.in)
-
.PHONY: stats
stats:
@generate-dashboard -o deployment/stats/prometheus/flytepropeller-dashboard.json stats/flytepropeller_dashboard.py
diff --git a/NOTICE b/NOTICE
deleted file mode 100644
index 43f19fee61..0000000000
--- a/NOTICE
+++ /dev/null
@@ -1,4 +0,0 @@
-flyte
-Copyright 2019 Lyft Inc.
-
-This product includes software developed at Lyft Inc.
diff --git a/README.md b/README.md
index ac2a7a314f..5904b608a3 100644
--- a/README.md
+++ b/README.md
@@ -1,4 +1,3 @@
-
@@ -8,338 +7,130 @@
- ๐ป ๐ณ ๐
+ ๐๏ธ ๐ ๐
-
-Code. Ship. Scale.
-
+
+ Build. Deploy. Scale.
+
-
- Flyte is a workflow automation platform for complex, mission-critical data, and ML processes at scale
-
+Flyte is an **open-source orchestrator** that facilitates building production-grade data and ML pipelines. It is built for scalability and reproducibility, leveraging Kubernetes as its underlying platform. With Flyte, user teams can construct pipelines using the Python SDK, and seamlessly deploy them on both cloud and on-premises environments, enabling distributed processing and efficient resource utilization.
- 
+ 
- 
+ 
- 
+ 
- 
-
- 
-
- 
-
- 
-
- 
-
- 
+ 
- 
+ 
- 
+ 
- 
+ 
- 
+ 
-
-
-
----
-# What is Flyte?
-Flyte is a structured programming and distributed processing platform that enables highly concurrent, scalable, and maintainable workflows for `Machine Learning` and `Data Processing`. It is a fabric that connects disparate computation backends using a type-safe data dependency graph. It records all changes to a pipeline, making it possible to rewind time. It also stores
-a history of all executions and provides an intuitive UI, CLI, and REST/gRPC API to interact with the computation.
-
-Flyte is more than a workflow engine -- it uses `workflow` as a core concept, and `task` (a single unit of execution) as a top-level concept. Multiple tasks arranged in a data
-producer-consumer order creates a workflow.
-
-`Workflows` and `Tasks` can be written in any language, with out-of-the-box support for [Python](https://github.com/flyteorg/flytekit), [Java and Scala](https://github.com/spotify/flytekit-java).
-Flyte was designed to manage the complexity that arises in Data and ML teams and ensure they keep up their high velocity of delivering business impacting features. One way it achieves this is by separating
-the control-plane from the user-plane. Thus, every organization can offer Flyte as a service to their end-users where the service is managed by folks who are more infrastructure-focused, while the users use the intuitive interface of Flytekit.
-
-## โ๏ธ Why Flyte
-
-- Kubernetes-native workflow automation platform
-- Ergonomic SDKs in Python, Java, and Scala
-- Versioned, auditable, and reproducible pipelines
-- Data-aware and strongly-typed
-- Resource-aware and deployments at scale
-
-|  |
-| :--------------------------------------------------------------------------------------------------------: |
-| *โจ Flyte UI โจ* |
-___
-# Quickstart
-
-**NOTE**: If you want to try Flyte in the browser without installing anything locally, visit the [Flyte Hosted Sandbox](https://sandbox.union.ai/)
-___
-1. Ensure you're using Python version >=`3.7`, <=`3.10`
-2. Install Flytekit, Flyte's Python SDK:
-```bash
-$ pip install flytekit
-```
-3. Then install flytectl, the CLI for interacting with a Flyte backend:
-```bash
-brew install flyteorg/homebrew-tap/flytectl
-```
-**You're all set!**
-
-For installation instructions for other operating systems and how to run an example workflow, visit the [Getting Started guide](https://docs.flyte.org/projects/cookbook/en/latest/index.html#)!
-## โญ๏ธ Organizations/projects using Flyte (excerpt)
-
-
-
-- [appliedAI Initiative](https://appliedai.de)
-- [Blackshark.ai](https://blackshark.ai/)
-- [Freenome](https://www.freenome.com/)
-- [Gojek](https://www.gojek.io/)
-- [Hurb](https://www.hurb.com/)
-- [Intel](https://www.intel.com/)
-- [LatchBio](https://www.latch.bio/)
-- [Level 5 Global Autonomous (Woven Planet)](https://level-5.global/)
-(...)
-**Check [ADOPTERS](https://github.com/flyteorg/community/blob/main/ADOPTERS.md) for the full list and how to add your org/project**
-
-
- ๐ฅ Features
-
-
-
-
-
-- Used at _Scale_ in production by multiple firms and projects.
-- Proved to scale to more than **1 million** executions, and **40+ million** containers
-- Data Aware and Resource Aware (Allows organizations to separate concerns - users can use the API, platforms/infra teams can manage the deployments and scaling)
-- Enables **collaboration across your organization** by:
- - Executing distributed data pipelines/workflows
- - Making it easy to stitch together workflows from different teams and domain experts and share them across teams
- - Comparing results of training workflows over time and across pipelines
- - Simplifying the complexity of multi-step, multi-owner workflows
-- **[Get Started quickly](https://docs.flyte.org/projects/cookbook/en/latest/index.html)** -- start locally and scale to the cloud instantly
-- **gRPC / REST** interface to define and execute tasks and workflows
-- **Typesafe** construction of pipelines -- each task has an interface characterized by its input and output, so illegal construction of pipelines fails during declaration, rather than at runtime
-- Supports multiple **[data types](https://docs.flyte.org/projects/cookbook/en/latest/auto/core/type_system/index.html)** for machine learning and data processing pipelines, such as Blobs (images, arbitrary files), Directories, Schema (columnar structured data), collections, maps, etc.
-- Memoization and Lineage tracking
-- Provides logging and observability
-- Workflow features:
- - Start with one task, convert to a pipeline, attach **[multiple schedules](https://docs.flyte.org/projects/cookbook/en/latest/auto/core/scheduled_workflows/lp_schedules.html#sphx-glr-auto-core-scheduled-workflows-lp-schedules-py)**, trigger using a programmatic API, or on-demand
- - Parallel step execution
- - Extensible backend to add **[customized plugin](https://docs.flyte.org/projects/flytekit/en/latest/tasks.extend.html#custom-tasks)** experience (with simplified user experience)
- - **[Branching](https://docs.flyte.org/projects/cookbook/en/latest/auto/core/control_flow/conditions.html)**
- - Workflow of workflows - **[subworkflows](https://docs.flyte.org/projects/cookbook/en/latest/auto/core/control_flow/subworkflows.html)** (a workflow can be embedded within one node of the top-level workflow)
- - Distributed **remote external workflows** (a remote workflow can be triggered and statically verified at compile time)
- - **[Array Tasks](https://docs.flyte.org/projects/cookbook/en/latest/auto/core/control_flow/map_task.html)** (map a function over a large dataset -- ensures controlled execution of thousands of containers)
- - **[Dynamic workflows](https://docs.flyte.org/projects/cookbook/en/latest/auto/core/control_flow/dynamics.html)** creation and execution with runtime type safety
- - [Flytekit plugins](https://docs.flyte.org/projects/flytekit/en/latest/plugins/index.html#plugins) with first-class support in Python
- - Arbitrary Flytekit-less containers tasks ([RawContainer](https://docs.flyte.org/projects/cookbook/en/latest/auto/core/containerization/raw_container.html))
-- Guaranteed **[reproducibility](https://docs.flyte.org/projects/cookbook/en/latest/auto/core/flyte_basics/task_cache.html)** of pipelines via:
- - Versioned data, code, and models
- - Automatically tracked executions
- - Declarative pipelines
-- **Multi-cloud support** (AWS, GCP, and others)
-- No single point of failure, and is resilient by design
-- Automated notifications to Slack, Email, and Pagerduty
-- [Multi K8s cluster support](https://docs.flyte.org/projects/cookbook/en/latest/auto/integrations/kubernetes/pod/index.html)
-- Out of the box support to run **[Spark jobs on K8s](https://docs.flyte.org/projects/cookbook/en/latest/auto/integrations/kubernetes/k8s_spark/index.html)**, **[Dask jobs on K8s](https://docs.flyte.org/projects/cookbook/en/latest/auto/integrations/kubernetes/k8s_dask/index.html)**, **[Hive queries](https://docs.flyte.org/projects/cookbook/en/latest/auto/integrations/external_services/hive/index.html)**, etc.
-- Snappy Console & Golang CLI (Flytectl)
-- Written in **Golang** and optimized for jobs that run for a long period of time.
-- [Grafana templates](https://grafana.com/orgs/flyte) (user/system observability)
-- Deploy with Helm and Kustomize
-
-
-
-## ๐ Available Plugins
-
-Check out the [current list of plugins](https://docs.flyte.org/projects/flytekit/en/latest/plugins/index.html#plugins) and how to use them
-
-
-
-
-
-____
-## Contributing
-
-There are many ways in which you can participate in this project, including:
-
-* Submit [bugs](https://github.com/flyteorg/flyte/issues/new?assignees=&labels=bug%2Cuntriaged&template=bug_report.yaml&title=%5BBUG%5D+) and [feature requests](https://github.com/flyteorg/flyte/issues/new?assignees=&labels=enhancement%2Cuntriaged&template=feature_request.yaml&title=%5BCore+feature%5D+) for different components
-
-* Review [the documentation](https://docs.flyte.org/en/latest/) and make [pull requests](https://github.com/flyteorg/flytesnacks) for anything from typos to additional and new content
-* Speak or write about Flyte or any other ecosystem integration and [let us know](https://flyte-org.slack.com/archives/C02JMT8KTEE)!
-* Grab a [`help wanted`](https://github.com/flyteorg/flyte/issues?q=is%3Aopen+is%3Aissue+label%3A%22help+wanted%22+) or [`good-first-issue`](https://github.com/flyteorg/flyte/issues?q=is%3Aopen+is%3Aissue+label%3A%22good+first+issue%22) and follow the [CONTRIBUTING](https://docs.flyte.org/en/latest/community/contribute.html#contribute-flyte) guide to submit changes to the codebase
-* Upvote [popular feature requests](https://github.com/flyteorg/flyte/issues?q=is%3Aopen+is%3Aissue+label%3Aenhancement+sort%3Areactions-%2B1-desc)
-
-
-
-
- ๐ฃ๏ธ Live Roadmap
-
-
-
-Live roadmap for the project can be found @[Github Live Roadmap](https://github.com/orgs/flyteorg/projects/3)
-
-
-
-
-
- ๐ RFCs (Request for Comments) & Proposals
-
-
-
-Flyte is a community-driven and community-owned software. It is managed using
-a steering committee and encourages collaboration. The community has a long
-roadmap for Flyte, but there might be interesting
-ideas, extensions, or additions that you may want to propose. This is
-usually done by starting with:
-
-- [Github Issue](https://github.com/flyteorg/flyte/issues): we maintain issues for all repos in the main flyte repo.
-- Writing down your proposal using a [documented RFC process](rfc/.).
-
-RFCs are encouraged for larger changes.
-You are welcome to hop into our [Slack](https://flyte-org.slack.com/archives/C04NJPLRWUX) and talk to the community
-if you want to test the waters before proposing.
-
-
- ๐ฆ Component Repos
+ Resources
-
-
+
-| Repo | Language | Purpose | Status |
-| ------------------------------------------------------------ | ------------- | ---------------------------------------------- | ---------------- |
-| [flyte](https://github.com/flyteorg/flyte) | Kustomize,RST | deployment, documentation, issues | Production-grade |
-| [flyteidl](https://github.com/flyteorg/flyteidl) | Protobuf | gRPC/REST API, Workflow language spec | Production-grade |
-| [flytepropeller](https://github.com/flyteorg/flytepropeller) | Go | execution engine | Production-grade |
-| [flyteadmin](https://github.com/flyteorg/flyteadmin) | Go | control plane | Production-grade |
-| [flytekit](https://github.com/flyteorg/flytekit) | Python | python SDK and tools | Production-grade |
-| [flyteconsole](https://github.com/flyteorg/flyteconsole) | Typescript | Flyte UI | Production-grade |
-| [datacatalog](https://github.com/flyteorg/datacatalog) | Go | manage input & output artifacts | Production-grade |
-| [flyteplugins](https://github.com/flyteorg/flyteplugins) | Go | Flyte Backend plugins | Production-grade |
-| [flytecopilot](https://github.com/flyteorg/flytecopilot) | Go | Sidecar to manage input/output for sdk-less | Production-grade |
-| [flytestdlib](https://github.com/flyteorg/flytestdlib) | Go | standard library | Production-grade |
-| [flytesnacks](https://github.com/flyteorg/flytesnacks) | Python | examples, tips, and tricks | Maintained |
-| [flytekit-java](https://github.com/flyteorg/flytekit-java) | Java/Scala | Java & scala SDK for authoring Flyte workflows | Incubating |
-| [flytectl](https://github.com/flyteorg/flytectl) | Go | A standalone Flyte CLI | Production-grade |
-| [homebrew-tap](https://github.com/flyteorg/homebrew-tap) | Ruby | Tap for downloadable flyte tools (cli etc) | Production-grade |
-| [bazel-rules](https://github.com/flyteorg/bazelrules_flyte) | skylark/py | Use Bazel to build Flyte workflows and tasks | Incubating |
+
-
-
-_____
-
-
-
-
-Use the following resources to communicate with Flyte maintainers, contributors and other community members to ask questions and enhance your learning experience:
-
-### Asynchronous
-
-- [Slack](https://slack.flyte.org)
-- [Mailing list](https://lists.lfaidata.foundation/g/flyte-announce/join)
-- [GitHub Discussions](https://github.com/flyteorg/flyte/discussions)
-
-### Social media
-- [Twitter](https://twitter.com/flyteorg)
-- [LinkedIn Discussion Group](https://www.linkedin.com/groups/13962256/)
-
-## Meetings
-### Biweekly Community Sync
-
-- ๐ฃโ**Flyte OSS Community Sync**
-Join this meeting to learn from other users how they are leveraging Flyte for different use cases, ask questions or continue community-related discussions
-* When: every other Tuesday, 9:00 am - 10:00 am PT.
-* Where: [Zoom](https://us04web.zoom.us/j/71298741279?pwd=TDR1RUppQmxGaDRFdzBOa2lHN1dsZz09) bridge
-* Previous meetings: [[notes](https://docs.google.com/document/d/1EPR-BqWi8t7OF_forVflM8lWLm1T-_IE2_5_HGeyGlo/edit?usp=sharing)| [recordings](https://www.youtube.com/@flyteorg/playlists)]
-
-* Subscribe to the [calendar](https://lists.lfaidata.foundation/g/flyte-announce/ics/12031983/2145304139/feed.ics)
-
-### Office Hours
+---
-Receive support from Flyte maintainers:
+## Features
-* When: weekly on Wednesdays
-* How: schedule a 30 min session from one of the available slots (7AM, 1PM and 9PM PT) using [this link](https://calendly.com/flyte-office-hours-01/30min)
+We ship new features, bug fixes and performance improvements regularly. Read our [release notes](https://github.com/flyteorg/flyte/releases) to stay updated.
-### Knowledge Base
+๐ **Strongly typed interfaces**: Validate your data at every step of the workflow by defining data guardrails using Flyte types.
+๐ **Any language**: Write code in any language using raw containers, or choose [Python](https://github.com/flyteorg/flytekit), [Java](https://github.com/flyteorg/flytekit-java), [Scala](https://github.com/flyteorg/flytekit-java) or [JavaScript](https://github.com/NotMatthewGriffin/pterodactyl) SDKs to develop your Flyte workflows.
+๐ **Map tasks**: Achieve parallel code execution with minimal configuration using [map tasks](https://docs.flyte.org/projects/cookbook/en/latest/auto/core/control_flow/map_task.html).
+๐ **Dynamic workflows**: [Build flexible and adaptable workflows](https://docs.flyte.org/projects/cookbook/en/latest/auto/core/control_flow/dynamics.html) that can change and evolve as needed, making it easier to respond to changing requirements.
+๐ณ **Branching**: [Selectively execute branches](https://docs.flyte.org/projects/cookbook/en/latest/auto/core/control_flow/conditions.html) of your workflow based on static or dynamic data produced by other tasks or input data.
+๐ **FlyteFile & FlyteDirectory**: Transfer [files](https://docs.flyte.org/projects/cookbook/en/latest/auto/core/flyte_basics/files.html) and [directories](https://docs.flyte.org/projects/cookbook/en/latest/auto/core/flyte_basics/folders.html) between local and cloud storage.
+๐๏ธ **Structured dataset**: Convert dataframes between types and enforce column-level type checking using the abstract 2D representation provided by [Structured Dataset](https://docs.flyte.org/projects/cookbook/en/latest/auto/core/type_system/structured_dataset.html).
+๐ก๏ธ **Recover from failures**: Recover only the failed tasks.
+๐ **Rerun a single task**: Rerun workflows at the most granular level without modifying the previous state of a data/ML workflow.
+๐ฆ **Versioned workflows**: Reproduce results and roll back to a previous workflow version any time.
+๐ **Cache outputs**: Cache task outputs by passing `cache=True` to the task decorator.
+๐ฉ **Intra-task checkpointing**: [Checkpoint progress](https://docs.flyte.org/projects/cookbook/en/latest/auto/core/control_flow/checkpoint.html) within a task execution.
+๐ **Multi-tenancy**: Multiple users can share the same platform while maintaining their own distinct data and configurations.
+โฐ **Timeout**: Define a timeout period, after which the task is marked as failure.
+๐ **Immutability**: Immutable executions help ensure reproducibility by preventing any changes to the state of an execution.
+๐งฌ **Data lineage**: Track the movement and transformation of data throughout the lifecycle of your data and ML workflows.
+๐ **Data visualization**: Visualize data, monitor models and view training history through plots.
+๐ญ **Dev to prod**: As simple as changing your [domain](https://docs.flyte.org/en/latest/concepts/domains.html) from development or staging to production.
+๐ธ **Spot or preemptible instances**: Schedule your workflows on spot instances by setting `interruptible` to `True` in the task decorator.
+โ๏ธ **Cloud-native deployment**: Deploy Flyte on AWS, GCP, Azure and other cloud services.
+๐
**Scheduling**: [Schedule](https://docs.flyte.org/projects/cookbook/en/latest/auto/core/scheduled_workflows/lp_schedules.html) your data and ML workflows to run at a specific time.
+๐ข **Notifications**: Stay informed about changes to your workflow's state by configuring [notifications](https://docs.flyte.org/projects/cookbook/en/latest/auto/deployment/lp_notifications.html) through Slack, PagerDuty or email.
+โ๏ธ **Timeline view**: Evaluate the duration of each of your Flyte tasks and identify potential bottlenecks.
+๐จ **GPU acceleration**: Enable and control your tasksโ GPU demands by requesting resources in the task decorator.
+๐ณ **Dependency isolation via containers**: Maintain separate sets of dependencies for your tasks so no dependency conflicts arise.
+๐ **Parallelism**: Flyte tasks are inherently parallel to optimize resource consumption and improve performance.
+๐พ **Allocate resources dynamically** at the task level.
+โฏ๏ธ [Wait](https://docs.flyte.org/projects/cookbook/en/latest/auto/core/control_flow/waiting_for_external_inputs.html) for **external inputs** before proceeding with the execution.
-Find answers to the FAQs at [Knowledge Base](https://discuss.flyte.org): our minified StackOverflow and magnified Slack.
+## Quick start
-### Conference Talks & Podcasts
+If you want to try out Flyte, the easiest way to get started is by using the Flyte Hosted Sandbox, available at https://sandbox.union.ai/. It allows you to experiment with Flyte's capabilities without installing anything on your local machine.
-Videos and recordings can be found on Flyte's YouTube channel under the [Conference Talks and Podcasts](https://www.youtube.com/playlist?list=PLmQd1BBY9MWouwgSeivMoA4ldSyRcRj_-) playlist.
+However, if you prefer to install Flyte locally and run a workflow, our [getting started guide](https://docs.flyte.org/projects/cookbook/en/latest/index.html) is the best place to start. It provides step-by-step instructions on how to install Flyte locally and run your first workflow.
-### Podcasts
-
-- MLOps Coffee Session - [Why You Need More Than Airflow](http://go.mlops.community/Oz48gY)
-- Kelsey Hightower Twitter Space - [Machine Learning in Production](https://twitter.com/i/spaces/1ZkKzbXLekWKv)
-- Contributor.fyi - [Flyte with Ketan Umare](https://www.contributor.fyi/flyte)
-- TWIML&AI - [Scalable and Maintainable ML Workflows at Lyft - Flyte](https://twimlai.com/twiml-talk-343-scalable-and-maintainable-workflows-at-lyft-with-flyte-w-haytham-abuelfutuh-and-ketan-umare/)
-- Software Engineering Daily - [Flyte: Lyft Data Processing Platform](https://softwareengineeringdaily.com/2020/03/12/flyte-lyft-data-processing-platform-with-allyson-gale-and-ketan-umare/)
-- MLOps Coffee session - [Flyte: an open-source tool for scalable, extensible, and portable workflows](https://anchor.fm/mlops/episodes/MLOps-Coffee-Sessions-12-Flyte-an-open-source-tool-for-scalable--extensible---and-portable-workflows-eksa5k)
-- Open Data Science - [West Warm Up session with Ketan Umare - Creator of Flyte](https://twitter.com/odsc/status/1451594432369758212)
+> If you're unsure what a Flyte [task](https://docs.flyte.org/projects/cookbook/en/latest/auto/core/flyte_basics/task.html) and [workflow](https://docs.flyte.org/projects/cookbook/en/latest/auto/core/flyte_basics/basic_workflow.html) are, be sure to check out our guides on both!
-
+### Deploy Flyte to production
-## ๐งช Functional Tests Matrix
+The [deployment guide](https://docs.flyte.org/en/latest/deployment/index.html) provides useful information on how to self-host and manage Flyte.
-We run a suite of [tests](https://github.com/flyteorg/flytesnacks/blob/master/cookbook/flyte_tests_manifest.json) to ensure that basic functionality and a subset of the integrations work across a variety of release versions. Those tests are run in a cluster where specific versions of the Flyte components (such as flyteconsole, flyteadmin, datacatalog, and flytepropeller) are installed, including the released versions and also the latest versions in the case of the nighly runs.
+## Adopters
-The table below has different release versions as the columns and the result of each test suite as rows:
+Join the likes of LinkedIn, Spotify, Freenome, Pachama, Gojek, and Woven Planet in adopting Flyte. For a full list of adopters and information on how to add your organization or project, please visit our [ADOPTERS](https://github.com/flyteorg/community/blob/main/ADOPTERS.md) page.
-
-
+## Roadmap
-| workflow group | nightly | v1.0.1 | v1.0.0 | v0.19.4 |
-| ------------------------------- | ------------------------------------------------------------------------------------------------------------------ | ----------------------------------------------------------------------------------------------------------------- | ----------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------ |
-| core |  |  |  |  |
-| integrations-hive |  |  |  |  |
-| integrations-k8s-spark |  |  |  |  |
-| integrations-kfpytorch |  |  |  |  |
-| integrations-pod |  |  |  |  |
-| integrations-pandera_examples |  |  |  |  |
-| integrations-papermilltasks |  |  |  |  |
-| integrations-greatexpectations |  |  |  |  |
-| integrations-sagemaker-pytorch |  |  |  |  |
-| integrations-sagemaker-training |  |  |  |  |
+Stay up to date with the latest project roadmap by checking out our live [roadmap](https://github.com/orgs/flyteorg/projects/3) on GitHub.
-
-
+## Resources
+๐ [Weekly office hours](https://calendly.com/flyte-office-hours-01/30min): Live informal sessions with the Flyte team held every week. Book a 30-minute slot and get your questions answered.
+๐ฅ [Biweekly community sync](https://www.addevent.com/event/EA7823958): A biweekly event where the Flyte team provides updates on the project and community members can share their progress and ask questions.
+๐ฌ [Slack](https://slack.flyte.org/): Join the Flyte community on Slack to chat with other users, ask questions, and get help.
+โ ๏ธ [GitHub](https://github.com/flyteorg/flyte): Check out the Flyte project on GitHub to file issues, contribute code, and stay up to date on the latest development.
+๐น [Youtube](https://www.youtube.com/channel/UCNduEoLOToNo3nFVly-vUTQ): Tune into panel discussions, customer success stories, community updates and feature deep dives.
+๐ [Blog](https://flyte.org/blog): Here, you can find tutorials and feature deep dives to help you learn more about Flyte.
+๐ก [RFCs](rfc/.): RFCs are used for proposing new ideas and features to improve Flyte. You can refer to them to stay updated on the latest developments and contribute to the growth of the platform.
+## How to contribute
+There are many ways to get involved in Flyte, including:
-## ๐ Contributors
+* Submitting [bugs](https://github.com/flyteorg/flyte/issues/new?assignees=&labels=bug%2Cuntriaged&template=bug_report.yaml&title=%5BBUG%5D+) and [feature requests](https://github.com/flyteorg/flyte/issues/new?assignees=&labels=enhancement%2Cuntriaged&template=feature_request.yaml&title=%5BCore+feature%5D+) for various components.
+* Reviewing [the documentation](https://docs.flyte.org/en/latest/) and submitting [pull requests](https://github.com/flyteorg/flytesnacks) for anything from fixing typos to adding new content.
+* Speaking or writing about Flyte or any other ecosystem integration and [letting us know](https://flyte-org.slack.com/archives/C02JMT8KTEE)!
+* Taking on a [`help wanted`](https://github.com/flyteorg/flyte/issues?q=is%3Aopen+is%3Aissue+label%3A%22help+wanted%22+) or [`good-first-issue`](https://github.com/flyteorg/flyte/issues?q=is%3Aopen+is%3Aissue+label%3A%22good+first+issue%22) and following the [CONTRIBUTING](https://docs.flyte.org/en/latest/community/contribute.html#contribute-flyte) guide to submit changes to the codebase.
+* Upvoting [popular feature requests](https://github.com/flyteorg/flyte/issues?q=is%3Aopen+is%3Aissue+label%3Aenhancement+sort%3Areactions-%2B1-desc) to show your support.
-A big thank you to the community for making Flyte possible!
+### We โค๏ธ our contributors
[](https://github.com/katrogan)[](https://github.com/lyft-metaservice-3)[](https://github.com/matthewphsmith)[](https://github.com/EngHabu)[](https://github.com/goreleaserbot)[](https://github.com/evalsocket)[](https://github.com/hamersaw)[](https://github.com/flyte-bot)[](https://github.com/honnix)[](https://github.com/anandswaminathan)[](https://github.com/wild-endeavor)[](https://github.com/bnsblue)[](https://github.com/pingsutw)[](https://github.com/iaroslav-ciupin)[](https://github.com/kumare3)[](https://github.com/samhita-alla)[](https://github.com/MorpheusXAUT)[](https://github.com/aliabbasjaffri)[](https://github.com/ckiosidis)[](https://github.com/mayitbeegh)[](https://github.com/alexlipa91)[](https://github.com/brucearctor)[](https://github.com/pmahindrakar-oss)[](https://github.com/Antaxify)[](https://github.com/eapolinario)[](https://github.com/andrewwdye)[](https://github.com/ariefrahmansyah)[](https://github.com/jeevb)[](https://github.com/jonathanburns)[](https://github.com/lu4nm3)[](https://github.com/lyft-metaservice-2)[](https://github.com/2uasimojo)[](https://github.com/veggiemonk)[](https://github.com/schottra)[](https://github.com/supreeth7)[](https://github.com/cosmicBboy)[](https://github.com/migueltol22)[](https://github.com/milton0825)[](https://github.com/slai)[](https://github.com/SmritiSatyanV)[](https://github.com/surindersinghp)[](https://github.com/TheYk98)[](https://github.com/kosigz-lyft)[](https://github.com/chanadian)[](https://github.com/kanterov)[](https://github.com/hanzo)[](https://github.com/igorvalko)[](https://github.com/kennyworkman)[](https://github.com/maximsmol)[](https://github.com/vsbus)[](https://github.com/akhurana001)[](https://github.com/bstadlbauer)[](https://github.com/jerempy)[](https://github.com/tnsetting)[](https://github.com/catalinii)[](https://github.com/ByronHsu)[](https://github.com/chetcode)[](https://github.com/regadas)[](https://github.com/fg91)[](https://github.com/aybidi)[](https://github.com/akashkatipally)[](https://github.com/goyalankit)[](https://github.com/clairemcginty)[](https://github.com/flixr)[](https://github.com/akumor)[](https://github.com/niliayu)[](https://github.com/snyk-bot)[](https://github.com/derwiki)[](https://github.com/th0114nd)[](https://github.com/AlekhyaSasi)[](https://github.com/apps/dependabot)[](https://github.com/asottile)[](https://github.com/SandraGH5)[](https://github.com/sbrunk)[](https://github.com/sonjaer)[](https://github.com/fediazgon)[](https://github.com/rahul-theorem)[](https://github.com/ryankarlos)[](https://github.com/ddhirajkumar)[](https://github.com/max-hoffman)[](https://github.com/AdrianoKF)[](https://github.com/dennisobrien)[](https://github.com/Ln11211)[](https://github.com/aeioulisa)[](https://github.com/michaels-lyft)[](https://github.com/murilommen)[](https://github.com/RobertoRRW)[](https://github.com/bimtauer)[](https://github.com/esadler-hbo)[](https://github.com/ggydush-fn)[](https://github.com/kiliangojek)[](https://github.com/bethebunny)[](https://github.com/aalavian)[](https://github.com/convexquad)[](https://github.com/andresgomezfrr)[](https://github.com/asahalyft)[](https://github.com/apatel-fn)[](https://github.com/arpitbhardwaj)[](https://github.com/lordnodd)[](https://github.com/bryanwweber)[](https://github.com/CalvinLeather)[](https://github.com/YmirKhang)[](https://github.com/franco-bocci)[](https://github.com/frsann)[](https://github.com/hfurkanvural)[](https://github.com/jbrambleDC)[](https://github.com/jcugat)[](https://github.com/madhur-tandon)[](https://github.com/matheusMoreno)[](https://github.com/NotMatthewGriffin)[](https://github.com/myz540)[](https://github.com/tekumara)[](https://github.com/ppiegaze)[](https://github.com/Qiwen-Yu)[](https://github.com/RobinKa)[](https://github.com/rubenbarragan)[](https://github.com/sugatoray)[](https://github.com/sushrut111)[](https://github.com/Tat-V)[](https://github.com/TeoZosa)[](https://github.com/ThomVett)[](https://github.com/datability-io)[](https://github.com/techytushar)[](https://github.com/vchowdhary)[](https://github.com/varshaparthay)[](https://github.com/vvasavada-fn)[](https://github.com/ybubnov)[](https://github.com/Yicheng-Lu-llll)[](https://github.com/palchicz)[](https://github.com/ajsalow)[](https://github.com/ggydush)[](https://github.com/martinlyra)[](https://github.com/mcloney-ddm)[](https://github.com/pbrogan12)[](https://github.com/stef-stripe)[](https://github.com/charlie0220)[](https://github.com/stephen37)[](https://github.com/anrusina)[](https://github.com/service-github-lyft-semantic-release)[](https://github.com/ursucarina)[](https://github.com/jsonporter)[](https://github.com/csirius)[](https://github.com/olga-union)[](https://github.com/Pianist038801)[](https://github.com/james-union)[](https://github.com/eugenejahn)[](https://github.com/4nalog)[](https://github.com/apTalya)[](https://github.com/aviaviavi)[](https://github.com/Professional0321)[](https://github.com/HiromuHota)[](https://github.com/rafaelraposospot)[](https://github.com/daniel-shuy)[](https://github.com/live-wire)[](https://github.com/sisco0)[](https://github.com/jimbobby5)[](https://github.com/pradithya)[](https://github.com/skiptomyliu)[](https://github.com/haoyuez)[](https://github.com/lupasarin)[](https://github.com/Dread1982)[](https://github.com/narape)[](https://github.com/alexapdev)[](https://github.com/3t8)[](https://github.com/zeryx)[](https://github.com/arturdryomov)[](https://github.com/ChickenTarm)[](https://github.com/DavidMertz)[](https://github.com/felixwang9817)[](https://github.com/juandiegopalomino)[](https://github.com/kanyesthaker)[](https://github.com/marc-union)[](https://github.com/michaeltinsley)[](https://github.com/mucahitkantepe)[](https://github.com/oyevtushok)[](https://github.com/trishitapingolia)[](https://github.com/Smartmind12)[](https://github.com/huxuan)[](https://github.com/privatedumbo)[](https://github.com/tjKairos)[](https://github.com/georgesnelling)[](https://github.com/dschaller)[](https://github.com/davidmirror-ops)[](https://github.com/davidxia)[](https://github.com/hoyajigi)[](https://github.com/MrKrishnaAgarwal)[](https://github.com/NitinAgg)[](https://github.com/noobkid2411)[](https://github.com/yongchand)[](https://github.com/nicklofaso)[](https://github.com/mounesi)[](https://github.com/eanakhl)[](https://github.com/adinin)[](https://github.com/anton-malakhov)[](https://github.com/avan-sh)[](https://github.com/kinow)[](https://github.com/Daeruin)[](https://github.com/dav009)[](https://github.com/idivyanshbansal)[](https://github.com/fvde)[](https://github.com/Lundez)[](https://github.com/hasukmistry)[](https://github.com/rokrokss)[](https://github.com/jeremydonahue)[](https://github.com/jonasdebeukelaer)[](https://github.com/jmcarp)[](https://github.com/kazesberger)[](https://github.com/lsena)[](https://github.com/mishmanners)[](https://github.com/paravatha)[](https://github.com/uschi2000)[](https://github.com/rodrigobaron)[](https://github.com/ronaldosaheki)[](https://github.com/shahwar9)[](https://github.com/shihgianlee)[](https://github.com/SKalt)[](https://github.com/samuel-sujith)[](https://github.com/ilikedata)[](https://github.com/orf)[](https://github.com/vijaysaravana)[](https://github.com/yubofredwang)[](https://github.com/fsz285)[](https://github.com/gigi-at-zymergen)[](https://github.com/hampusrosvall)[](https://github.com/hitarth01)[](https://github.com/jcourteau)[](https://github.com/jw0515)[](https://github.com/leorleor)[](https://github.com/moose007)[](https://github.com/sanjaychouhan-adf)[](https://github.com/v01dXYZ)[](https://github.com/wanderer163)[](https://github.com/kylewaynebenson)[](https://github.com/Gui11aum3)[](https://github.com/JakeNeyer)[](https://github.com/abhijeet007rocks8)[](https://github.com/mouuff)[](https://github.com/Juneezee)[](https://github.com/goodgravy)[](https://github.com/radiantly)[](https://github.com/nishantwrp)[](https://github.com/sighingnow)[](https://github.com/vglocus)[](https://github.com/RustedBones)[](https://github.com/pablocasares)[](https://github.com/evdokim)[](https://github.com/stormy-ua)[](https://github.com/marschall)[](https://github.com/gdungca-fn)[](https://github.com/ttanay)[](https://github.com/Abdullahi-Ahmed)[](https://github.com/amaleelhamri)[](https://github.com/pradyunsg)[](https://github.com/apps/pre-commit-ci)[](https://github.com/jdknight)[](https://github.com/kmike)[](https://github.com/hugovk)[](https://github.com/sirosen)[](https://github.com/humitos)[](https://github.com/bastimeyer)[](https://github.com/asmeurer)[](https://github.com/drewyh)[](https://github.com/polyzen)[](https://github.com/dvarrazzo)[](https://github.com/dbitouze)[](https://github.com/idryzhov)[](https://github.com/pauloxnet)[](https://github.com/ichard26)[](https://github.com/sethmlarson)[](https://github.com/graingert)[](https://github.com/stonecharioteer)[](https://github.com/yeraydiazdiaz)[](https://github.com/eviau-sat)[](https://github.com/rozsasarpi)[](https://github.com/estan)[](https://github.com/pseudomuto)[](https://github.com/htdvisser)[](https://github.com/jacobtolar)[](https://github.com/ezimanyi)[](https://github.com/lpabon)[](https://github.com/ArcEye)[](https://github.com/mingrammer)[](https://github.com/aschrijver)[](https://github.com/panzerfahrer)[](https://github.com/glasser)[](https://github.com/murph0)[](https://github.com/zetaron)[](https://github.com/sunfmin)[](https://github.com/guozheng)[](https://github.com/suusan2go)[](https://github.com/mhaberler)[](https://github.com/s4ichi)[](https://github.com/dreampuf)[](https://github.com/UnicodingUnicorn)[](https://github.com/philiptzou)[](https://github.com/timabell)[](https://github.com/jasonhancock)[](https://github.com/matryer)[](https://github.com/piotrrojek)[](https://github.com/jasonsattler)[](https://github.com/sbward)[](https://github.com/Pisush)[](https://github.com/tamalsaha)[](https://github.com/marianina8)[](https://github.com/ernesto-jimenez)[](https://github.com/jsteenb2)[](https://github.com/darwayne)[](https://github.com/naysayer)[](https://github.com/AgrimPrasad)[](https://github.com/dahernan)[](https://github.com/jtarchie)[](https://github.com/jdtobe)[](https://github.com/alrs)[](https://github.com/urisimchoni)[](https://github.com/Xercoy)[](https://github.com/marbergq)[](https://github.com/anothrNick)[](https://github.com/fermoya)[](https://github.com/sbe-arg)[](https://github.com/PeerXu)[](https://github.com/reececomo)[](https://github.com/dmerrick)[](https://github.com/andrewcole)[](https://github.com/phish108)[](https://github.com/endrjuskr)[](https://github.com/bevans-HD)[](https://github.com/gukoff)[](https://github.com/lovromazgon)[](https://github.com/117)[](https://github.com/tomsolem)[](https://github.com/vq-ambiata)[](https://github.com/sjauld)[](https://github.com/adam-berrio)[](https://github.com/zsedem)[](https://github.com/imdanielsp)[](https://github.com/fabricepipart)[](https://github.com/ivanpk)[](https://github.com/JeremyLWright)[](https://github.com/OskarStark)[](https://github.com/pavyarov)[](https://github.com/pellared)[](https://github.com/cuttingedge1109)[](https://github.com/okozachenko1203)[](https://github.com/zero-below)[](https://github.com/asford)[](https://github.com/bmcconeghy)[](https://github.com/conda-forge-admin)[](https://github.com/regro-cf-autotick-bot)[](https://github.com/apps/conda-forge-curator)[](https://github.com/apps/github-actions)[](https://github.com/conda-forge-linter)[](https://github.com/fellhorn)
+
+
+## License
+
+Flyte is available under the Apache License 2.0. Use it wisely.
diff --git a/eks/README.md b/eks/README.md
deleted file mode 100644
index f0e7c0d611..0000000000
--- a/eks/README.md
+++ /dev/null
@@ -1,25 +0,0 @@
-
-Steps:
-- Make sure kubectl and aws-cli are installed and working
-- Make sure terraform is installed and working and references aws-cli permissions
-- Run Terraform files. (There seems to be a race condition in one of the IAM role creation steps - you may need to run it twice.)
-- Copy or update the kubectl config file and switch to that context.
-- Create the webhook
- - Create ECR repo for the webhook
- - Build the image and push
- - Run the make cluster-up command with the right image
-- Create the example 2048 game on the EKS IAM page linked above. Keep in mind that even after an address shows up in the ingress, it may take a while to provision.
-- Delete the game
-- Create the spare datacatalog reference in the db.
-- Follow the [Installation portion](https://github.com/aws/amazon-eks-pod-identity-webhook/blob/95808cffe6d801822dae122f2f2c87a258d70bb8/README.md#installation) of the webhook readme. You will need to make sure to use your own AWS account number, and will also need to build your own image and upload it to your ECR, which will probably require you to create that repository in your ECR.
-- Go through all the overlays in the `kustomize/overlays/eks` folder and make sure all the service accounts and RDS addresses reference yours. (Do a grep for `111222333456` and `456123e6ivib`).
-- Install Flyte with `kubectl apply -f deployment/eks/flyte_generated.yaml`
-
-This is the webhook used to inject IAM role credentials into pods.
-https://github.com/aws/amazon-eks-pod-identity-webhook
-
-This is how you get pods to use the proper roles. (This is the KIAM replacement.)
-https://docs.aws.amazon.com/eks/latest/userguide/iam-roles-for-service-accounts-technical-overview.html
-The implementation of these steps is done for you in the `alb-ingress` submodule.
-
-
diff --git a/eks/tf/main.tf b/eks/tf/main.tf
deleted file mode 100644
index 0e05fc6d6b..0000000000
--- a/eks/tf/main.tf
+++ /dev/null
@@ -1,161 +0,0 @@
-terraform {
- required_version = ">= 0.12.0"
-}
-
-provider "aws" {
- profile = "default"
- region = var.region
-}
-
-# Use the internal module to create an RDS instance with a user-supplied VPC
-module "flyte_rds" {
- source = "./modules/flyte-rds"
-
- rds_vpc = var.rds_vpc
-}
-
-# Use the internal module to create an EKS cluster, which has its own VPC
-module "flyte_eks" {
- source = "./modules/flyte-eks"
-
- eks_cluster_name = var.eks_cluster_name
-}
-
-# Get information about the two VPCs
-data "aws_vpc" "rds_vpc" {
- id = var.rds_vpc
-}
-
-data "aws_vpc" "eks_vpc" {
- id = module.flyte_eks.eks_vpc_id
-}
-
-# Get information about the RDS instance
-data "aws_db_instance" "flyte_rds" {
- db_instance_identifier = module.flyte_rds.admin_rds_instance_id
-}
-
-resource "aws_vpc_peering_connection" "eks_to_main_peering" {
- peer_vpc_id = var.rds_vpc
- vpc_id = module.flyte_eks.eks_vpc_id
- auto_accept = true
-
- tags = {
- Name = "VPC peering connection between Flyte RDS and EKS"
- }
-
- accepter {
- allow_remote_vpc_dns_resolution = true
- }
-
- requester {
- allow_remote_vpc_dns_resolution = true
- }
-}
-
-data "aws_route_table" "eks_public_route_table" {
- vpc_id = module.flyte_eks.eks_vpc_id
- filter {
- name = "tag:Name"
- values = ["${var.eks_cluster_name}-vpc-public"]
- }
-}
-
-resource "aws_route" "route_rds_cidr" {
- route_table_id = data.aws_route_table.eks_public_route_table.id
- destination_cidr_block = data.aws_vpc.rds_vpc.cidr_block
- vpc_peering_connection_id = aws_vpc_peering_connection.eks_to_main_peering.id
-}
-
-# Add a rule to the RDS security group to allow access from the EKS VPC
-resource "aws_security_group_rule" "allow_eks_to_rds" {
- type = "ingress"
- from_port = 5432
- to_port = 5432
- protocol = "tcp"
- cidr_blocks = [data.aws_vpc.eks_vpc.cidr_block]
- security_group_id = data.aws_db_instance.flyte_rds.vpc_security_groups[0]
-}
-
-# The following implements the instructions set forth by:
-# https://github.com/aws/amazon-eks-pod-identity-webhook/blob/95808cffe6d801822dae122f2f2c87a258d70bb8/README.md
-# This is a webhook that will allow pods to assume arbitrarily constrained roles via their service account.
-# TODO: This should be moved into a separate module probably but will require further refactoring as the assume role
-# policy used is also used further below in the ALB ingress module.
-
-# Create an oidc provider using the EKS cluster's public OIDC discovery endpoint
-resource "aws_iam_openid_connect_provider" "eks_oidc_connection" {
- client_id_list = ["sts.amazonaws.com"]
- thumbprint_list = ["9e99a48a9960b14926bb7f3b02e22da2b0ab7280"]
- url = module.flyte_eks.eks_oidc_issuer
-}
-
-locals {
- issuer_parsed = regex("^arn.*(?Poidc.eks.*)", aws_iam_openid_connect_provider.eks_oidc_connection.arn)
-}
-
-# This is the trust document that will allow pods to use roles that they specify in their service account
-data "aws_iam_policy_document" "let_pods_assume_roles" {
- statement {
- actions = ["sts:AssumeRoleWithWebIdentity"]
-
- principals {
- type = "Federated"
- identifiers = [aws_iam_openid_connect_provider.eks_oidc_connection.arn]
- }
-
- condition {
- test = "StringLike"
- variable = "${local.issuer_parsed.trailing}:sub"
-
- values = [
- "system:serviceaccount:*:*",
- ]
- }
- }
-}
-
-# Make a role for Flyte components themselves to use
-resource "aws_iam_role" "flyte_operator" {
- name = "flyte-operator"
- assume_role_policy = data.aws_iam_policy_document.let_pods_assume_roles.json
-}
-
-
-# Make a policy document
-# TODO: Scope this down later
-data "aws_iam_policy_document" "all_s3_access" {
- statement {
- actions = [
- "s3:*",
- ]
-
- resources = [
- "*",
- ]
- }
-}
-
-# Use the policy document to create a policy
-resource "aws_iam_policy" "flyte_operator_s3_access" {
- name = "flyte_operator_s3_access"
- path = "/"
- policy = data.aws_iam_policy_document.all_s3_access.json
-}
-
-# Attach the policy to the flyte operator role
-resource "aws_iam_role_policy_attachment" "flyte_operator_s3_attach" {
- role = aws_iam_role.flyte_operator.name
- policy_arn = aws_iam_policy.flyte_operator_s3_access.arn
-}
-
-module "alb_ingress" {
- source = "./modules/alb-ingress"
-
- region = var.region
- eks_cluster_name = var.eks_cluster_name
- cluster_id = module.flyte_eks.cluster_id
- eks_vpc_id = module.flyte_eks.eks_vpc_id
- assume_role_policy_string = data.aws_iam_policy_document.let_pods_assume_roles.json
-}
-
diff --git a/eks/tf/modules/alb-ingress/main.tf b/eks/tf/modules/alb-ingress/main.tf
deleted file mode 100644
index 2624ed9c43..0000000000
--- a/eks/tf/modules/alb-ingress/main.tf
+++ /dev/null
@@ -1,183 +0,0 @@
-# As required by https://docs.aws.amazon.com/eks/latest/userguide/alb-ingress.html
-data "http" "alb_ingress_policy" {
- url = "https://raw.githubusercontent.com/kubernetes-sigs/aws-alb-ingress-controller/v1.1.4/docs/examples/iam-policy.json"
-
- request_headers = {
- Accept = "application/json"
- }
-}
-
-resource "aws_iam_policy" "k8s_alb_ingress_controller" {
- name = "ALBIngressControllerIAMPolicy"
- path = "/"
- policy = data.http.alb_ingress_policy.body
-}
-
-resource "aws_iam_role" "eks_alb_ingress_controller" {
- name = "eks-alb-ingress-controller"
- assume_role_policy = var.assume_role_policy_string
-}
-
-# Attach the policy to the flyte operator role
-resource "aws_iam_role_policy_attachment" "eks_alb_attachment" {
- role = aws_iam_role.eks_alb_ingress_controller.name
- policy_arn = aws_iam_policy.k8s_alb_ingress_controller.arn
-}
-
-data "aws_eks_cluster" "cluster" {
- name = var.cluster_id
-}
-
-data "aws_eks_cluster_auth" "cluster" {
- name = var.cluster_id
-}
-
-provider "kubernetes" {
- host = data.aws_eks_cluster.cluster.endpoint
- cluster_ca_certificate = base64decode(data.aws_eks_cluster.cluster.certificate_authority.0.data)
- token = data.aws_eks_cluster_auth.cluster.token
- load_config_file = false
- version = "~> 1.9"
-}
-
-resource "kubernetes_cluster_role" "alb_ingress_controller" {
- metadata {
- name = "alb-ingress-controller"
-
- labels = {
- "app.kubernetes.io/name" : "alb-ingress-controller"
- }
- }
-
- rule {
- api_groups = [
- "",
- "extensions",
- ]
- resources = [
- "configmaps",
- "endpoints",
- "events",
- "ingresses",
- "ingresses/status",
- "services",
- ]
- verbs = [
- "create",
- "get",
- "list",
- "update",
- "watch",
- "patch",
- ]
- }
-
- rule {
- api_groups = [
- "",
- "extensions",
- ]
- resources = [
- "nodes",
- "pods",
- "secrets",
- "services",
- "namespaces",
- ]
- verbs = [
- "get",
- "list",
- "watch",
- ]
- }
-}
-
-resource "kubernetes_service_account" "alb_ingress_controller" {
- metadata {
- name = "alb-ingress-controller"
- namespace = "kube-system"
-
- labels = {
- "app.kubernetes.io/name" = "alb-ingress-controller"
- }
-
- annotations = {
- "eks.amazonaws.com/role-arn" = aws_iam_role.eks_alb_ingress_controller.arn
- }
- }
-}
-
-resource "kubernetes_cluster_role_binding" "alb_ingress_controller" {
- metadata {
- name = "alb-ingress-controller"
-
- labels = {
- "app.kubernetes.io/name" = "alb-ingress-controller"
- }
- }
-
- role_ref {
- api_group = "rbac.authorization.k8s.io"
- kind = "ClusterRole"
- name = kubernetes_cluster_role.alb_ingress_controller.metadata[0].name
- }
-
- subject {
- kind = "ServiceAccount"
- name = kubernetes_service_account.alb_ingress_controller.metadata[0].name
- namespace = "kube-system"
- }
-}
-
-resource "kubernetes_deployment" "alb_ingress_controller" {
- metadata {
- name = "alb-ingress-controller"
- namespace = "kube-system"
-
- labels = {
- "app.kubernetes.io/name" = "alb-ingress-controller"
- }
- }
-
- spec {
- selector {
- match_labels = {
- "app.kubernetes.io/name" = "alb-ingress-controller"
- }
- }
-
- template {
- metadata {
- name = "alb-ingress-controller"
- namespace = "kube-system"
-
- labels = {
- "app.kubernetes.io/name" = "alb-ingress-controller"
- }
- }
-
- spec {
- container {
- name = "alb-ingress-controller"
- image = "docker.io/amazon/aws-alb-ingress-controller:v1.1.4"
- args = [
- "--ingress-class=alb",
- "--cluster-name=${var.eks_cluster_name}",
- "--aws-vpc-id=${var.eks_vpc_id}",
- "--aws-region=${var.region}",
- "--feature-gates=waf=false",
- ]
- }
-
- service_account_name = kubernetes_service_account.alb_ingress_controller.metadata[0].name
- automount_service_account_token = true
-
- node_selector = {
- "beta.kubernetes.io/os" = "linux"
- }
- }
- }
- }
-}
-
-
diff --git a/eks/tf/modules/alb-ingress/outputs.tf b/eks/tf/modules/alb-ingress/outputs.tf
deleted file mode 100644
index e69de29bb2..0000000000
diff --git a/eks/tf/modules/alb-ingress/variables.tf b/eks/tf/modules/alb-ingress/variables.tf
deleted file mode 100644
index 43a0e9219c..0000000000
--- a/eks/tf/modules/alb-ingress/variables.tf
+++ /dev/null
@@ -1,20 +0,0 @@
-variable "region" {
- type = string
-}
-
-variable "eks_cluster_name" {
- type = string
-}
-
-variable "eks_vpc_id" {
- type = string
-}
-
-variable "cluster_id" {
- type = string
-}
-
-variable "assume_role_policy_string" {
- type = string
-}
-
diff --git a/eks/tf/modules/flyte-eks/main.tf b/eks/tf/modules/flyte-eks/main.tf
deleted file mode 100644
index edecace66a..0000000000
--- a/eks/tf/modules/flyte-eks/main.tf
+++ /dev/null
@@ -1,111 +0,0 @@
-data "aws_eks_cluster" "cluster" {
- name = module.eks.cluster_id
-}
-
-data "aws_availability_zones" "available" {
-}
-
-resource "aws_security_group" "worker_group_mgmt_one" {
- name_prefix = "worker_group_mgmt_one"
- vpc_id = module.vpc.vpc_id
-
- ingress {
- from_port = 22
- to_port = 22
- protocol = "tcp"
-
- cidr_blocks = [
- "10.0.0.0/8",
- ]
- }
-}
-
-resource "aws_security_group" "worker_group_mgmt_two" {
- name_prefix = "worker_group_mgmt_two"
- vpc_id = module.vpc.vpc_id
-
- ingress {
- from_port = 22
- to_port = 22
- protocol = "tcp"
-
- cidr_blocks = [
- "192.168.0.0/16",
- ]
- }
-}
-
-resource "aws_security_group" "all_worker_mgmt" {
- name_prefix = "all_worker_management"
- vpc_id = module.vpc.vpc_id
-
- ingress {
- from_port = 22
- to_port = 22
- protocol = "tcp"
-
- cidr_blocks = [
- "10.0.0.0/8",
- "172.16.0.0/12",
- "192.168.0.0/16",
- ]
- }
-}
-
-module "vpc" {
- source = "git::https://github.com/terraform-aws-modules/terraform-aws-vpc.git?ref=v2.6.0"
- name = "${var.eks_cluster_name}-vpc"
-
- cidr = "10.0.0.0/16"
- azs = data.aws_availability_zones.available.names
- private_subnets = ["10.0.1.0/24", "10.0.2.0/24", "10.0.3.0/24"]
- public_subnets = ["10.0.4.0/24", "10.0.5.0/24", "10.0.6.0/24"]
- enable_nat_gateway = true
- single_nat_gateway = true
- enable_dns_hostnames = true
-
- public_subnet_tags = {
- "kubernetes.io/cluster/${var.eks_cluster_name}" = "shared"
- "kubernetes.io/role/elb" = "1"
- }
-
- private_subnet_tags = {
- "kubernetes.io/cluster/${var.eks_cluster_name}" = "shared"
- "kubernetes.io/role/internal-elb" = "1"
- }
-}
-
-module "eks" {
- source = "git::https://github.com/terraform-aws-modules/terraform-aws-eks.git?ref=v11.0.0"
-
- cluster_name = var.eks_cluster_name
- subnets = module.vpc.private_subnets
-
- tags = {
- Environment = "test"
- GithubRepo = "terraform-aws-eks"
- GithubOrg = "terraform-aws-modules"
- }
-
- vpc_id = module.vpc.vpc_id
-
- worker_groups = [
- {
- name = "worker-group-1"
- instance_type = "t2.small"
- additional_userdata = "flyte t2.small"
- asg_desired_capacity = 2
- additional_security_group_ids = [aws_security_group.worker_group_mgmt_one.id]
- },
- {
- name = "worker-group-2"
- instance_type = "t2.medium"
- additional_userdata = "flyte t2.medium"
- additional_security_group_ids = [aws_security_group.worker_group_mgmt_two.id]
- asg_desired_capacity = 1
- },
- ]
-
- worker_additional_security_group_ids = [aws_security_group.all_worker_mgmt.id]
-}
-
diff --git a/eks/tf/modules/flyte-eks/outputs.tf b/eks/tf/modules/flyte-eks/outputs.tf
deleted file mode 100644
index 9e7cd30b94..0000000000
--- a/eks/tf/modules/flyte-eks/outputs.tf
+++ /dev/null
@@ -1,36 +0,0 @@
-output "cluster_endpoint" {
- description = "Endpoint for EKS control plane."
- value = module.eks.cluster_endpoint
-}
-
-output "cluster_security_group_id" {
- description = "Security group ids attached to the cluster control plane."
- value = module.eks.cluster_security_group_id
-}
-
-output "cluster_id" {
- description = "The id of the EKS cluster created"
- value = module.eks.cluster_id
-}
-
-output "kubectl_config" {
- description = "kubectl config as generated by the module."
- value = module.eks.kubeconfig
-}
-
-output "config_map_aws_auth" {
- description = "A kubernetes configuration to authenticate to this EKS cluster."
- value = module.eks.config_map_aws_auth
-}
-
-output "eks_vpc_id" {
- description = "The id of the VPC created for the EKS cluster"
- value = module.vpc.vpc_id
-}
-
-output "eks_oidc_issuer" {
- # Basically this is the output of this command
- # aws --region us-west-2 eks describe-cluster --name your-cluster --query cluster.identity.oidc.issuer
- description = "Issuer endpoint provided by EKS"
- value = data.aws_eks_cluster.cluster.identity.0.oidc.0.issuer
-}
diff --git a/eks/tf/modules/flyte-eks/variables.tf b/eks/tf/modules/flyte-eks/variables.tf
deleted file mode 100644
index 810e6fd537..0000000000
--- a/eks/tf/modules/flyte-eks/variables.tf
+++ /dev/null
@@ -1,4 +0,0 @@
-variable "eks_cluster_name" {
- type = string
-}
-
diff --git a/eks/tf/modules/flyte-rds/main.tf b/eks/tf/modules/flyte-rds/main.tf
deleted file mode 100644
index adceaad42d..0000000000
--- a/eks/tf/modules/flyte-rds/main.tf
+++ /dev/null
@@ -1,22 +0,0 @@
-resource "aws_rds_cluster_instance" "flyte_instances" {
- count = 1
- identifier = "flyteadmin-instances-${count.index}"
- cluster_identifier = aws_rds_cluster.flyteadmin.id
- instance_class = "db.t3.medium"
- engine = "aurora-postgresql"
- engine_version = "11.6"
-}
-
-resource "aws_rds_cluster" "flyteadmin" {
- cluster_identifier = "flyteadmin-cluster"
- engine = "aurora-postgresql"
- engine_version = "11.6"
- availability_zones = ["us-west-2a", "us-west-2b", "us-west-2c"]
- database_name = "flyteadmin"
- master_username = "flyteadmin"
- master_password = "spongebob"
- backup_retention_period = 1
- preferred_backup_window = "07:00-09:00"
- skip_final_snapshot = true
-}
-
diff --git a/eks/tf/modules/flyte-rds/outputs.tf b/eks/tf/modules/flyte-rds/outputs.tf
deleted file mode 100644
index 10a40713e2..0000000000
--- a/eks/tf/modules/flyte-rds/outputs.tf
+++ /dev/null
@@ -1,8 +0,0 @@
-output "admin_rds_name" {
- value = aws_rds_cluster.flyteadmin.endpoint
-}
-
-output "admin_rds_instance_id" {
- value = aws_rds_cluster_instance.flyte_instances[0].identifier
-}
-
diff --git a/eks/tf/modules/flyte-rds/variables.tf b/eks/tf/modules/flyte-rds/variables.tf
deleted file mode 100644
index 7f569fa0fc..0000000000
--- a/eks/tf/modules/flyte-rds/variables.tf
+++ /dev/null
@@ -1,4 +0,0 @@
-variable "rds_vpc" {
- type = string
-}
-
diff --git a/eks/tf/outputs.tf b/eks/tf/outputs.tf
deleted file mode 100644
index 7e7105832a..0000000000
--- a/eks/tf/outputs.tf
+++ /dev/null
@@ -1,49 +0,0 @@
-output "rds_vpc_cidr_block" {
- value = data.aws_vpc.rds_vpc.cidr_block
-}
-
-output "cluster_endpoint" {
- description = "Endpoint for EKS control plane."
- value = module.flyte_eks.cluster_endpoint
-}
-
-output "cluster_security_group_id" {
- description = "Security group ids attached to the cluster control plane."
- value = module.flyte_eks.cluster_security_group_id
-}
-
-output "kubectl_config" {
- description = "kubectl config as generated by the module."
- value = module.flyte_eks.kubectl_config
-}
-
-output "config_map_aws_auth" {
- description = "A kubernetes configuration to authenticate to this EKS cluster."
- value = module.flyte_eks.config_map_aws_auth
-}
-
-output "eks_vpc_id" {
- description = "The id of the VPC created for the EKS cluster"
- value = module.flyte_eks.eks_vpc_id
-}
-
-output "admin_rds_name" {
- value = module.flyte_rds.admin_rds_name
-}
-
-output "admin_rds_instance_id" {
- value = module.flyte_rds.admin_rds_instance_id
-}
-
-output "eks_cluster_oidc_connection_arn" {
- value = aws_iam_openid_connect_provider.eks_oidc_connection.arn
-}
-
-output "eks_cluster_oidc" {
- value = module.flyte_eks.eks_oidc_issuer
-}
-
-output "role_arn" {
- value = aws_iam_role.flyte_operator.arn
-}
-
diff --git a/eks/tf/values.auto.tfvars b/eks/tf/values.auto.tfvars
deleted file mode 100644
index 68f3e57973..0000000000
--- a/eks/tf/values.auto.tfvars
+++ /dev/null
@@ -1,4 +0,0 @@
-region = "us-west-2"
-rds_vpc = "vpc-1234567a"
-eks_cluster_name = "flyte-demo"
-
diff --git a/eks/tf/variables.tf b/eks/tf/variables.tf
deleted file mode 100644
index 640bb8c6da..0000000000
--- a/eks/tf/variables.tf
+++ /dev/null
@@ -1,12 +0,0 @@
-variable "region" {
- type = string
-}
-
-variable "rds_vpc" {
- type = string
-}
-
-variable "eks_cluster_name" {
- type = string
-}
-
diff --git a/rsts/community/contribute.rst b/rsts/community/contribute.rst
index d018dadac9..942fee29b1 100644
--- a/rsts/community/contribute.rst
+++ b/rsts/community/contribute.rst
@@ -13,15 +13,15 @@ Here are some guidelines for you to follow, which will make your first and follo
TL;DR: Find the repo-specific contribution guidelines in the `Component Reference <#component-reference>`__ section.
-๐ป Becoming a Contributor
+๐ป Becoming a contributor
=========================
-An issue tagged with ``good first issue`` is the best place to start for first-time contributors. You can find them `here `__.
+An issue tagged with `good first issue `__ is the best place to start for first-time contributors.
**Appetizer for every repo: Fork and clone the concerned repository. Create a new branch on your fork and make the required changes. Create a pull request once your work is ready for review.**
.. note::
- To open a pull request, follow `this guide `__.
+ To open a pull request, refer to `GitHub's guide `__ for detailed instructions.
Example PR for your reference: `GitHub PR `__.
A couple of checks are introduced to help maintain the robustness of the project.
@@ -34,7 +34,8 @@ On a side note, format your Go code with ``golangci-lint`` followed by ``goimpor
If make targets are not available, you can manually format the code.
Refer to `Effective Go `__, `Black `__, and `Isort `__ for full coding standards.
-As you become more involved with the project you may be able to be added as a contributor to the repos you're working on, but there is a medium term effort to move all development to forks โจ.
+As you become more involved with the project, you may be able to be added as a contributor to the repos you're working on,
+but there is a medium term effort to move all development to forks.
๐ Documentation
================
@@ -46,14 +47,7 @@ For API- and code-related content, it extracts docstrings from the code files.
To get started, refer to the `reStructuredText reference `__.
-For minor edits that donโt require a local setup, you can edit the GitHub page in the documentation to propose improvements.
-
-The edit option can be found at the bottom of a page, as shown below.
-
-.. figure:: https://raw.githubusercontent.com/flyteorg/static-resources/main/flyte/contribution_guide/docs_edit.png
- :alt: GitHub edit option for Documentation
- :align: center
- :figclass: align-center
+For minor edits that don't require a local setup, you can edit the GitHub page in the documentation to propose improvements.
Intersphinx
***********
@@ -138,18 +132,17 @@ Output:
:py:meth:`Method `
-๐งฑ Component Reference
+๐งฑ Component reference
======================
-To understand how the below components interact with each other, refer to :ref:`Understand the lifecycle of a workflow `
+To understand how the below components interact with each other, refer to :ref:`Understand the lifecycle of a workflow `.
.. figure:: https://raw.githubusercontent.com/flyteorg/static-resources/main/flyte/contribution_guide/dependency_graph.png
:alt: Dependency graph between various flyteorg repos
:align: center
:figclass: align-center
- The dependency graph between various flyte.org repos
-
+ The dependency graph between various flyte repos
``flyte``
*********
@@ -291,54 +284,73 @@ To understand how the below components interact with each other, refer to :ref:`
* - **Guidelines**: Refer to the `FlyteCTL Contribution Guide `__
-๐ฎ Recommended Iteration Cycle
+๐ฎ Recommended iteration cycle
==============================
-As you may have read in other parts of the documentation, this repo contains go code as well that pulls in all the backend components (admin, propeller, data catalog, console) into one executable.
-The Flyte team is also working on migrating the core backend repositories into one repo, some time in 2023. For the time being, you can still contribute by making changes to the individual repos, and then bringing them into this ``flyte`` repo. This setup works well for backend golang development, but is not tested for ``flyteconsole`` development. That development cycle will look a bit different. This setup here will allow you to run the Flyte binary from your IDE, hitting any breakpoints you may have, but will connect you to all the other resources of the demo environment (like postgres, rds).
-Setup
-*****
+As you may have already read in other parts of the documentation, the `Flyte repository `__ includes Go code
+that integrates all backend components (admin, propeller, data catalog, console) into a single executable.
+The Flyte team is currently working on consolidating the core backend repositories into one repository, which is expected to be completed by 2023.
+In the meantime, you can contribute to the individual repositories and then merge your changes into the `Flyte repository `__.
+This setup is suitable for Go-based backend development, but it has not been tested for Flyteconsole development, which has a different development cycle.
+Nonetheless, this setup allows you to run the Flyte binary from your IDE, enabling you to debug your code effectively by setting breakpoints.
+Additionally, this setup connects you to all other resources in the demo environment, such as PostgreSQL and RDS.
-Dev Mode Cluster
-----------------
-To launch the dependencies, teardown any old sandboxes you may have, and then run ::
+Dev mode cluster
+****************
+
+To launch the dependencies, teardown any old sandboxes you may have, and then run:
- ``flytectl demo start --dev``
+.. code-block::
-This will run the demo environment but without Flyte itself. This makes it so that you the developer can run it later, on your host machine.
+ flytectl demo start --dev
+
+This command will launch the demo environment without running Flyte. By doing so, developers can run Flyte later on their host machine.
Set up Flyte configuration
---------------------------
-#. Copy the file ``flyte-single-binary-local.yaml`` to ``~/.flyte/local-dev-config.yaml``
-#. Replace instances of ``$HOME`` with the real path of your home directory.
+**************************
+
+#. Copy the file ``flyte-single-binary-local.yaml`` to ``~/.flyte/local-dev-config.yaml``.
+#. Replace occurrences of ``$HOME`` with the actual path of your home directory.
-Cluster Resources
------------------
-You'll notice one of the entries in the config is ``cluster_resources.templatePath``. In this folder you should put the templates for the cluster resource controller to use. For now, it's okay to just start with the namespace one. Create a file called ``~/.flyte/cluster-resource-templates/00_namespace.yaml`` with the following ::
+Cluster resources
+*****************
- .. literalinclude:: ../../charts/flyte-binary/eks-production.yaml
- :lines: 81-85
+One of the configuration entries you will notice is ``cluster_resources.templatePath``.
+This folder should contain the templates that the cluster resource controller will use.
+To begin, you can create a file called ``~/.flyte/cluster-resource-templates/00_namespace.yaml`` with the following content:
-Pull Console Artifacts
-----------------------
-Run the following command from the base folder of this repo to pull in the static assets for Flyteconsole ::
+.. literalinclude:: ../../charts/flyte-binary/eks-production.yaml
+ :lines: 81-87
+
+Pull console artifacts
+**********************
+
+Run the following command from the base folder of the Flyte repository to pull in the static assets for Flyteconsole:
+
+.. code-block::
make cmd/single/dist
-Build & Iterate
----------------
-Update using ``go get github.com/flyteorg/&gitsha`` to bring in the code of the component you're testing. From there, you can run ::
+Build and iterate
+*****************
+
+To bring in the code of the component you are testing, use the command go get ``github.com/flyteorg/&gitsha``.
+Once you have done that, you can run the following command:
+
+.. code-block::
POD_NAMESPACE=flyte go run -tags console cmd/main.go start --config ~/.flyte/local-dev-config.yaml
-The ``POD_NAMESPACE`` environment is required for the webhook to work. Of course you can create a build target in your IDE with the same effective command as well.
+The ``POD_NAMESPACE`` environment variable is necessary for the webhook to function correctly.
+You can also create a build target in your IDE with the same command.
-After it's running, you should still be able to go to ``localhost:30080/console`` to see Flyte but now hosted by your host computer. It relies on a Docker host mapping to get the correct IP for your local host.
+Once it is up and running, you can access Flyte hosted by your local machine by going to ``localhost:30080/console``.
+The Docker host mapping is used to obtain the correct IP address for your local host.
-๐ File an Issue
+๐ File an issue
================
-`GitHub Issues `__ is used for issue tracking. The following are available issue types that you could use for filing an issue.
+We use `GitHub Issues `__ for issue tracking. The following issue types are available for filing an issue:
* `Plugin Request `__
* `Bug Report `__
@@ -351,7 +363,4 @@ After it's running, you should still be able to go to ``localhost:30080/console`
If none of the above fit your requirements, file a `blank `__ issue.
Also, add relevant labels to your issue. For example, if you are filing a Flytekit plugin request, add the ``flytekit`` label.
-|
-
For feedback at any point in the contribution process, feel free to reach out to us on `Slack `__.
-
diff --git a/requirements.in b/stats/requirements.in
similarity index 100%
rename from requirements.in
rename to stats/requirements.in
diff --git a/requirements.txt b/stats/requirements.txt
similarity index 100%
rename from requirements.txt
rename to stats/requirements.txt