-
Notifications
You must be signed in to change notification settings - Fork 1.7k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Add @kubernetes
decorator
#17248
Add @kubernetes
decorator
#17248
Conversation
future = await worker.submit( | ||
flow=flow, parameters=parameters, job_variables=job_variables | ||
) | ||
return await future.aresult() |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
note from discussion: we probably want to consider adding a .submit
method to the decorated flow object that returns this future
directly when called and otherwise, when the new flow-object is called directly it automatically resolves the future as it does here.
d01c250
to
cfe0db8
Compare
d9ade40
to
83a3584
Compare
def decorator(flow: Flow[P, R]) -> InfrastructureBoundFlow[P, R]: | ||
return InfrastructureBoundFlow.from_flow( | ||
flow, | ||
work_pool=work_pool, | ||
job_variables=job_variables, | ||
worker_cls=KubernetesWorker, | ||
) |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
looking trim!
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
lgtm!
|
||
|
||
def kubernetes( | ||
work_pool: str, **job_variables: Any |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
it'd be nice to have a TypedDict
here to Unpack
someday, not sure how we'd manage that, but just throwing it out there
work_pool: str, **job_variables: Any | |
work_pool: str, **job_variables: Any |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I could see a world where we can generate a decorator for a specific work pool that includes the job variables as named and type kwargs. We're probably a ways off from that though.
16d4a52
to
8b3d8d3
Compare
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
love it
This PR introduces a
@kubernetes
decorator that will bind a flow to run in a given work pool.Here's what it looks like in action:
The goal of this decorator is to allow decorating flows and bind them to running in a Kubernetes cluster, but not require any code changes where the flow is called. This is achieved in this PR except for a few exceptions:
wait_for
kwarg is unused right now.Additional work will be necessary to address these limitations.
Related to #17218