Pipeline with configured resource fails to start

[szalai1]

I preconfigured our k8s executor the following way:

def get_k8s_executor_config(image_name: str = "dagster"):
    return {
        "backend": "amqp",
        "broker": "pyamqp://xxxxx@dagster-rabbitmq:5672//",
        "env_config_maps": [
            "dagster-pipeline-env"
        ],
        "env_secrets": [
            "dagster-pipeline-secrets"
        ],
        "job_image": f"xxxx.dkr.ecr.eu-west-1.amazonaws.com/{image_name}:latest",
        "job_namespace": "dagster",
        "image_pull_policy": "Always"
    }


prod_k8s_executor = celery_k8s_job_executor.configured(get_k8s_executor_config())
prod_kyber_k8s_executor = celery_k8s_job_executor.configured(get_k8s_executor_config('dagster-kyber'))

and used prod_k8s_executor as executor_defs=[prod_k8s_executor]. but when I want to run the pipeline (no error on the UI editor) I got this:
dagster.check.CheckError: Invariant failed. Description: celery-k8s execution must be configured in pipeline execution config to launch runs with CeleryK8sRunLauncher

so I added to the run_config:

execution:
  celery-k8s: {}

Then I got dagster.core.errors.DagsterInvariantViolationError: Cannot find job_image in celery-k8s executor config.

It seems the executor is not configured, what am I doing wrong?

[alangenfeld]

what am I doing wrong?

Nothing, this is a bug in our system with celery-k8s in particular #2979

[szalai1]

Thanks :)