[Core feature] Override task secret_requests using with_overrides

[EternalDeiwos]

Motivation: Why do you think this is important?

Not being able to generally override properties of a task using with_overrides is not intuitive and in this case is forcing me to have multiple copies of identical code.

Goal: What should the final outcome look like, ideally?

I would like to do something like this:

@workflow
def run():
    data = get_data()
    overrides = { 
        "secret_requests": [
            Secret(key="username", group="db"),
            Secret(key="password", group="db"),
        ],
    }
    common.generic_task(**params).with_overrides(**overrides)

It would be most intuitive to have anything passed into with_overrides override the equivalently named property of the task, however it appears here that properties are being handled explicitly. There is also the ability to override task_config but there is no documentation for how this will behave.

Describe alternatives you've considered

Create separate tasks with identical logic with different secrets attached. This works but is not DRY.

Propose: Link/Inline OR Additional context

No response

Are you sure this issue hasn't been raised already?

• Yes

Have you read the Code of Conduct?

• Yes

[dosubot]

Hey there @EternalDeiwos! Great to see you back and diving into some interesting challenges with us.

Thank you for your detailed request. I understand your need to override the secret_requests property of a task using the with_overrides function in Flyte. This would indeed make your code more DRY and intuitive.

Currently, the with_overrides function in Flyte allows for node-level override for workflows at compile time. However, it does not explicitly handle the secret_requests property. To enable runtime overrides, you can use a dynamic workflow that can override the secret_requests at runtime. Here's an example:

import flytekit
from flytekit import CronSchedule, LaunchPlan, Secret, task, workflow, dynamic

SECRET_NAME = "user_secret"
SECRET_GROUP = "user-info"

# Original task definition remains the same
@task(secret_requests=[Secret(group=SECRET_GROUP, key=SECRET_NAME)])
def secret_task() -> str:
    secret_val = flytekit.current_context().secrets.get(SECRET_GROUP, SECRET_NAME)
    print(secret_val)
    return secret_val

# Dynamic workflow to override secret_requests
@dynamic
def dynamic_wf(secret_name: str, secret_group: str) -> str:
    # Override the secret_requests for the task
    overridden_secret_task = secret_task.with_overrides(secret_requests=[Secret(group=secret_group, key=secret_name)])
    x = overridden_secret_task()
    return x

@workflow
def wf(secret_name: str = SECRET_NAME, secret_group: str = SECRET_GROUP) -> str:
    x = dynamic_wf(secret_name=secret_name, secret_group=secret_group)
    return x

sslp = LaunchPlan.get_or_create(
    name="scheduled_secrets",
    workflow=wf,
    schedule=CronSchedule(schedule="0/1 * * * *"),
)

In this example, the dynamic_wf function is a dynamic workflow that takes secret_name and secret_group as inputs. It uses these inputs to override the secret_requests of the secret_task by calling with_overrides on the task. This allows you to dynamically specify different secrets for the task at runtime without needing to duplicate the task code for each secret.

You can find more information about dynamic workflows in the Flyte documentation.

I hope this helps! If you have any further questions or need more clarification, please don't hesitate to ask.

Sources

• docker/sandbox-bundled/test/secrets_wf.py
• rfc/system/3553-config-override.md

About Dosu

This response is meant to be useful and save you time. It is not meant to be a precise solution, but rather a starting point for your own research.

Help me be more useful! Please leave a 👍 if this is helpful and 👎 if it is irrelevant.

If you want to continue the conversation, start your reply with @dosu-bot.

[pingsutw]

@EternalDeiwos qq: if you override the secret, how do you get the value from it in the task? Let's say if we have a task like

@task(secret_requests=[Secret(group=SECRET_GROUP, key=SECRET_NAME)])
def secret_task() -> str:
    context = flytekit.current_context()
    secret_val = context.secrets.get(SECRET_GROUP, SECRET_NAME)
    print(secret_val)
    return secret_val

if you override it, context.secrets.get won't work, right?

[EternalDeiwos]

@pingsutw I haven’t tried since I made this issue hit why would it not? Is the context not updated by overrides? If not then it certainly should be.

[Future-Outlier]

@pingsutw I haven’t tried since I made this issue hit why would it not? Is the context not updated by overrides? If not then it certainly should be.

I think it's doable to support this PR since we can use try except to get different secrets with overrides.

SECRET_NAME = "user_secret"
SECRET_GROUP = "user-info"

# Original task definition remains the same
@task(secret_requests=[Secret(group=SECRET_GROUP, key=SECRET_NAME)])
def secret_task() -> str:
    # add more try except logic to do get diiferent secret with different groups and names.
    secret_val = flytekit.current_context().secrets.get(SECRET_GROUP, SECRET_NAME)
    print(secret_val)
    return secret_val

[asd0300]

#take