Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add KuberenetesWorker.submit for ad-hoc submission to a Kubernetes work pool #17218

Merged
merged 12 commits into from
Feb 25, 2025

Conversation

desertaxle
Copy link
Member

@desertaxle desertaxle commented Feb 20, 2025

This PR adds an experimental .submit method to the KubernetesWorker class to allow a flow to be submitted to a Kubernetes cluster. The updates to the work pool API haven't been made yet, so the steps for bundle upload and execution are stored within the work pool's env field on the base job template.

Here's an example:

import asyncio
import json
import os

from prefect_kubernetes import KubernetesWorker

from prefect import flow

# These steps are stored in `env` in the work pool's base job template
UPLOAD_STEP = {
    "prefect_aws.experimental.bundles.upload": {
        "requires": "prefect-aws==0.5.5",
        "bucket": "storage-blocks-test-bucket",
        "aws_credentials_block_name": "my-creds",
    }
}

EXECUTE_STEP = {
    "prefect_aws.experimental.bundles.execute": {
        "requires": "prefect-aws==0.5.5",
        "bucket": "storage-blocks-test-bucket",
        "aws_credentials_block_name": "my-creds",
    }
}


@flow(log_prints=True, result_storage="s3-bucket/results-bucket")
def my_flow():
    print("Returning greeting...")
    return "Hello, world!"


async def main():
    async with KubernetesWorker(work_pool_name="olympic") as worker:
        future = await worker.submit(flow=my_flow)

    print(future.result()) # prints "Hello, world!"


if __name__ == "__main__":
    asyncio.run(main())

Related to #17194

Copy link

codspeed-hq bot commented Feb 20, 2025

CodSpeed Performance Report

Merging #17218 will not alter performance

Comparing k8s-worker-submit (d18f090) with main (ab2634c)

Summary

✅ 2 untouched benchmarks

@desertaxle desertaxle changed the base branch from main to flow-run-watcher February 21, 2025 19:23
Base automatically changed from flow-run-watcher to main February 21, 2025 21:16
@desertaxle desertaxle marked this pull request as ready for review February 21, 2025 21:57
@desertaxle desertaxle requested a review from cicdw February 24, 2025 21:59
Copy link
Member

@cicdw cicdw left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

noticed a few things but overall is looking good

@desertaxle desertaxle requested a review from cicdw February 25, 2025 00:45
@desertaxle desertaxle merged commit 88a030e into main Feb 25, 2025
51 checks passed
@desertaxle desertaxle deleted the k8s-worker-submit branch February 25, 2025 18:00
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants