allow for setting variables for future steps from within a step

[gapodo]

Clear and concise description of the problem

Sometimes a lot of work has to be done to prepare environment variables for their future use, while I can mostly do this within the step (just run some commands, potentially even from a yaml variable), it becomes problematic when using plugins, as they don't (usually) support running commands in their environment before running.

This problem currently affects me the most with the plugin-docker-buildx, where we need to prepare some build-args to be passed into the plugin. Though it would also ease many other things I'm currently doing in pipelines.

Suggested solution

Allow a step to export variables for use in the following steps, this could be done by either providing a way to write them to a file, which is injected as environment in the following steps, or by an explicit mechanism (i.e. calling a specific command, which woodpecker reacts to).

To properly steer / control this, a flag within the step should exist, which either allows "any" exports, or take the variable names, which are exported to the next steps.

a short (likely not syntactically correct) idea... just as a discussion start
I've left out the "normal" options (this should just be an add-on)

pipeline:
  # potentially some steps that run before
  # a step which is allowed to provide any keys...
  standard_env:
    image: some/image
    allow_export: true
    commands:
      - what ever prepares and exports the needed key-value pairs
  
  # a step that is just allowed to provide specific keys
  debugging_env:
    image: some/other:image
    allow_exports:
      - DEBUGGING_FLAGS
      - GOPROXY
    commands:
      - ......
  
  # potentially some other steps

  # use the variables provided by previous steps
  use_env:
    image: what/ever
    include_env_from:
      standard_env: # filter values to include from standard_env
        - OUR_TAG
        - SOME_OTHER_VALUE
      debugging_env # include anything provided by debugging_env

  # use prepared variables in a step, where "normal" processing isn't possible
  use_plugin:
    image: some/plugin
    include_env_from:
      - standard_env # include all variables provided by standard_env
    settings:
      tag: $${OUR_TAG}
      label: $${SOME_OTHER_VALUE}

While being able to also use the variables directly (within the pipeline yaml and not the executed image) would be nice, I believe this could overcomplicate things.

Alternative

This could also be implemented in a more hands off way, by providing a second mountpoint (additional to the normal working directory), which a step can write into a file, which could then be included in the starting-script of steps (by woodpecker)...

Basic example (as always, just the addition and usecase, may not be valid yaml as I'm just quickly typing this up in the browser...)

env_export_mount: /env_export

pipeline:
  standard_env:
    image: ...
    export_mount: true
    commands:
      - echo "somekey=value" > /env_export/standard_env
  
  debugging_env:
    image: ...
    export_mount: true
    commands:
      - echo "debug=trace" > /env_export/debugging_env
  
  using_env:
    image: ...
    include_env:
      - standard_env # include_env would load this file into the env of the container (this should likely be done by the backend)
      - debugging_env
  
  plugin:
    image: some/plugin
    include_env:
      - standard_env # this is where the backend handling injection is important, as plugins would need to be modified otherways
    settings:
      tag: $${somekey} # this should(TM) be evaluated within the plugin, though this is where the agent doing the work would be nice for ensuring it also works in env-less containers (single binary)

The first approach would IMHO be more solid, while this one would likely be easier to implement (except the potential issue with plugins), though it could potentially be "bodged" by just running an additional command in the script_.go / script.go

Additional context

Some other CIs have this feature either by design (explicitly implemented for the purpose) or by happy accident (e.g. Gitlab-CI when not using containers, as steps share the env...).

Being able to prepare variables becomes especially important, when using plugins that are privileged, as the user cannot replace them easily with something that prepares the env as he needs it (esp. so when used in a shared setting like e.g. at Codeberg or in a company/community setting, where devs should not be allowed to declare their own things as privileged).

Validations

• Read the Contributing Guidelines.
• Read the docs.
• Check that there isn't already an issue that request the same feature to avoid creating a duplicate.

[nightly-brew]

Did this get introduced in some version?
Or maybe, did someone find a way to do this?

Especially with plugins, I can't find a way to share a value from a previous step, and reuse it in the settings section...

[nitezs]

Has there been any substantive progress in this discussion?
Now how should I share variables in different steps?

[zc-devs]

Now how should I share variables in different steps?

Via a file in the shared workspace.

[dvjn]

I am really interested in writing a proposal/implementation for this feature with something more in-line with tools like GitHub Actions.

Would just like to get opinions from any of the maintainers if this is something we would like to spend time on before putting it that effort.

[dvjn]

I really like the http server idea. It is very neat. This would solve problems like shared parameters, artifacts, etc.

However, if we are looking for a simpler solution, we can use the filesystem.
Writing parameters can be a direct echo "test=hello" >> "$WOODPECKER_OUTPUT" instead of a curl call.
Saving artifacts can be a direct cp output.zip "$WOODPECKER_ARTIFACTS".
This could be implemented using a private shared volume, like the one we use to share the workspace.
This would also eliminate the need of tools like curl.

Just some rough thoughts.

[zc-devs]

What's hold you from doing this right now? You have a shell and a volume (workspace).

[dvjn]

Plugins.

I don't want to create plugins which read variables from arbitrary files. It would be better if Woodpecker could manage these outputs somehow and provide a way to safely expose them to a subsequent step. I have explained one approach to implement this in #2266 (reply in thread)

[lafriks]

The problem is that this can have security implications also, so this is something that really needs to be thought through before implementing

[dvjn]

The problem is that this can have security implications also

I think my approach is being misunderstood. The point of echo "test=hello" >> "$WOODPECKER_OUTPUT" was to be an easier API for the user to do curl --request POST --data '{"output_var": "value"}' "${CALLBACK_URL}". This will still only be able to set key value pair outputs for this step. This shouldn't be misunderstood as "writing arbitrary data on a shared filesystem". This might as well be implemented using a unix socket file. Each step will have its own file which the agent will listen on and save the data in something like a map[string]map[string]string{"step-1": {"out_var": "value"}, "step-2": {}} and make these values available to subsequent steps using some syntax similar to from_secret but evaluated on runtime of the pipeline.

[zc-devs]

Plugins cannot use arbitrary envs

We could agree (make a "standard") on where to place env-file.
For example, /woodpecker/.env or directory /woodpecker/env.
Updating the environment would be done by general step

echo "TEST=hello" >> '/woodpecker/.env'
echo "GOPATH=/go/path" >> '/woodpecker/env/build'
echo "REGISTRY=registry.example.com" >> '/woodpecker/env/image'

Importing from a general step would be like

. /woodpecker/.env
for path in /woodpecker/env/*
  . $path
done

And for plugins we could add to plugin-template some code, which load envs from standard directory or file (/woodpecker/env or /woodpecker/.env). Documentation for plugins devs could be changed (or some announcement) to encourage them to process this files.

I think this discussion could be closed as wont fix, because:

1. General steps can transfer environment from step to step right now without introducing special pipeline syntax;
2. Plugins are prohibited from using environments.

Or move to something like how to set / share plugins settings. But, probably, answer is External Configuration API.

[dvjn]

Using environment variables was just one way to implement this, but there are more robust solutions out there.

For example, we could design something like this:

steps:
  - name: generate-metadata
    image: ubuntu:latest
    commands:
      - tags=$(generate-tags.sh)
      - labels=$(generate-labels.sh)
      # set outputs using file syntax
      - echo "tags=${tags}" >> "$WOODPECKER_OUTPUT"
      - echo "labels=${labels}" >> "$WOODPECKER_OUTPUT"
      # or using a callback server
      - curl \
        --header "Content-Type: application/json" \
        --request POST \
        --data '{"tags": "'"${tags}"'", "labels": "'"${labels}"'"}' \
        "${WOODPECKER_CALLBACK_SERVER}"

  - name: build
    image: woodpecker/buildx
    settings:
      tags:
        from_output: generate-metadata.tags
      labels:
        from_output: generate-metadata.labels

Even plugins could set their own outputs:

steps:
  - name: generate-metadata
    image: dvjn/woodpecker-metadata
    settings:
      ...
  - name: build
    image: woodpecker/buildx
    settings:
      tags:
        from_output: generate-metadata.tags
      labels:
        from_output: generate-metadata.labels

This approach would significantly enhance the plugin ecosystem by allowing outputs to be set and then used as inputs in subsequent steps.

It would also solve the issue where plugins need to handle parameters in two different ways (like reading tags from an environment variable or a file in the Woodpecker buildx plugin).

[zc-devs]

Plugins

So, we are talking about

how to set / share plugins settings

---

I see the chicken and egg problem here. Pipeline is processed by Server before execution by Agent. But to substitute settings, Server (or Agent?) needs "first" step's execution result.

[zc-devs]

And for plugins we could add to plugin-template some code, which load envs from standard directory or file (/woodpecker/env or /woodpecker/.env)

What if the plugins could be set up not only by settings (PLUGIN_* envs), but also by a config file (JSON, YAML, whatever)? Some preparation step could produce this config file (like /woodpecker/settings/buildx.yaml) and plugin consume it. Obviously, only plugin's external API is involved here, so should not be problems like it was with envs.

---

Edit 1

Wait 🤔 PLUGIN_* envs are settings -> external API.
Therefore it could be env file, but plugin has to filter/consume only PLUGIN_* envs. Also we have to separate settings for each plugin-step. It could be like:

steps:
  - name: generate-metadata
    image: dvjn/woodpecker-metadata
    settings:
      buildx-settings-file: /woodpecker/settings/buildx.env
  - name: build
    image: woodpecker/buildx
    settings:
      file: /woodpecker/settings/buildx.env

/woodpecker/settings/buildx.env:

PLUGIN_TAGS='1.1.1'
PLUGIN_LABELS='some-label

[dvjn]

I see the chicken and egg problem here.

Yes this is a problem, but it seems solvable with some additional logic on the agent.

Some preparation step could produce this config file

I strongly disagree with this idea as each plugin which is generating something would need to be tightly coupled with the subsequent plugin's required inputs. This means plugins would be not at all composable by nature and create a closed ecosystem.

[zc-devs]

Plugin settings could be reworked to use JSON/YAML. YAML is more convenient as WP uses it in its pipelines. It would prevent issues like #4146.
All below could be implemented as complying to convention (plugin template):

1. WP mounts settings as file in a plugin's container into well-known location. That is settings from WP.
2. Some previous step writes additional settings in well-known location or it could be some setting in the plugin like additional_options_file(s).
3. The plugin loads settings from WP and merges it with additional ones from previous steps.