[BUG] V2.7.0 breaks JSON secrets

[jakubdrabik]

Describe the bug
After the latest update (v2.7.0) our pipelines started failing due to unexpected tokens in JSON. Downgrading to v2.6.0 solved the issue.

To Reproduce
As an example, in the Vault, in the key-value engine, there's Google Service Account JSON content. It's been working fine, but now when it's passed to google-github-actions/auth action, it fails with:

google-github-actions/auth failed with: retry function failed after 4 attempts: failed to parse service account key JSON credentials: unexpected token \ in JSON at position 1

Additional context
Related to this: #466

cc @fairclothjm

[fairclothjm]

@jakubdrabik Thanks for reporting your issue. Could you please provide more information on how the JSON is stored in Vault? Specifically, what is the output of the following commands (with sensitive data redacted):

vault kv get -mount=<MOUNT_PATH> <KEY>
vault kv get -mount=<MOUNT_PATH> -format=json <KEY>

[jakubdrabik]

@fairclothjm of course

vault kv get -mount=<MOUNT_PATH>

GCP_SA_KEY     {
  "type": "service_account",
  "project_id": "...",
  "private_key_id": "...",
  "private_key": "-----BEGIN PRIVATE KEY-----\n...\n-----END PRIVATE KEY-----\n",
  "client_email": "..."
}

vault kv get -mount=<MOUNT_PATH> -format=json

{
  "data": {
    "GCP_SA_KEY": "{\n  \"type\": \"service_account\",\n  \"project_id\": \"...\",\n  \"private_key_id\": \"...\",\n  \"private_key\": \"-----BEGIN PRIVATE KEY-----\\n...\\n-----END PRIVATE KEY-----\\n\"\n}\n"
  }
}

[fairclothjm]

Thanks @jakubdrabik!

Are you doing any processing on the data before it is passed to the google-github-actions/auth action? Or are you simply doing "${{ steps.import-secrets.outputs.KEY }}"?

[sergeykuprash]

We have the same issue. We are doing:

- name: Authenticate to Google Cloud
        uses: google-github-actions/[email protected]
        with:
          credentials_json: ${{ steps.vault_secrets.outputs.google_credentials }}

[MattiDeGrauwe]

Same issue, broke my JSON secrets. Using 2.6 for now.

[DanieleCali]

Same issue, we are retrieving all the secrets at once by doing:

- name: Fetch secrets from vault
   id: fetch_secrets_from_vault
   uses: actions/vault-action@v2
   with:
      ...
      secrets: |
        <path_to_secrets>/secrets $.$ | SECRETS;

We downgraded the version to the 2.6.0 for now.

[fairclothjm]

We will work on fixing this. The issues looks to be with secrets stored in Vault as multi-line JSON strings.

There are a few options to work around this at the moment:

Pin your vault-action to v2.6.0

uses: actions/[email protected]

Or, if you want to keep using v2.7.0 (or v2), store your JSON string as a single line in Vault (this is the way google-github-actions/auth recommends storing secrets)

vault kv put -mount=secret singleline key="$(cat file.json | jq -r tostring)"

[fairclothjm]

Released Vault GitHub Action v2.7.1 which fixes this regression

[mweber15]

I have a secret stored like this:

{
  "certdata": [
    "-----BEGIN CERTIFICATE-----\n...\n-----END CERTIFICATE-----",
    "-----BEGIN CERTIFICATE-----\n...\n-----END CERTIFICATE-----"
  ],
}

This is working with v2.6.0 and v2.7.1, but is failing in v2.7.2.

[fairclothjm]

@mweber15 Hi, thanks for reporting. Can you please provide more details:

• what command you use to store the data in vault
• the expected output of vault-action

[mweber15]

I'm not sure what the store command looked like. I do know it was stored as multi-line data, and I'm reasonably certain changing it to be stored as a single line would avoid the problem.

My usage looks similar to this:

    secrets: secret/data/my-secrets secret1 | SECRET1 ;
  secret/data/my-secrets secret2 | SECRET2 ;
  secret/data/my-secrets secret3 | SECRET3
    exportEnv: true

SECRET3 is the certdata above, and I use that in a later step:

  run: |
    set -e
    echo 1
    echo "$SECRET2" > data0.pem
    echo 2
    echo "$SECRET3" | jq -r .[0] >> data0.pem
    echo 3
    echo "$SECRET3" | jq -r .[1] > data1.pem
    echo 4

This works with 2.6.0 and 2.7.1, but fails with 2.7.0 or 2.7.2 with this output:

Run set -e
1
2
parse error: Invalid numeric literal at line 1, column 3
Error: Process completed with exit code 4.

Sorry if this isn't enough to be actionable. I'm a bit limited in what I can provide right now.

[fairclothjm]

@mweber15 Thanks, what is the output of the following commands (with sensitive data redacted):

vault kv get -mount=<MOUNT_PATH> <KEY>
vault kv get -mount=<MOUNT_PATH> -format=json <KEY>

[mweber15]

vault kv get -mount=<MOUNT_PATH> <KEY>

========== Data ==========
Key                 Value
---                 -----
SECRET3             [-----BEGIN CERTIFICATE-----
...
-----END CERTIFICATE----- -----BEGIN CERTIFICATE-----
...
-----END CERTIFICATE-----]

vault kv get -mount=<MOUNT_PATH> -format=json <KEY>

{
  "request_id": "...",
  "lease_id": "",
  "lease_duration": 0,
  "renewable": false,
  "data": {
    "data": {
      "SECRET3": [
        "-----BEGIN CERTIFICATE-----\n...\n-----END CERTIFICATE-----",
        "-----BEGIN CERTIFICATE-----\n...\n-----END CERTIFICATE-----"
      ],
      "certificate": "-----BEGIN CERTIFICATE-----\n...\n-----END CERTIFICATE-----",
      "expiration": ...,
      "issuing_ca": "-----BEGIN CERTIFICATE-----\n...\n-----END CERTIFICATE-----",
      "private_key": "...",
      "private_key_type": "...",
      "serial_number": "..."
    },
    "metadata": {
      "created_time": "...",
      "deletion_time": "",
      "destroyed": false,
      "version": 2
    }
  },
  "warnings": null
}

[fairclothjm]

thanks @mweber15! I am beginning to think that vault-action may need a JSON format option to better handle JSON secrets. Vault already has this, and in fact does not support your use-case without the format=json flag.

JSON data stored in Vault (as a map):

$ vault kv get  -mount=secret  -field=certdata cert
[-----BEGIN CERTIFICATE-----
...
-----END CERTIFICATE----- -----BEGIN CERTIFICATE-----
...
-----END CERTIFICATE-----]

Without format=json:

$ vault kv get -mount=secret -field=certdata cert |jq
parse error: Invalid numeric literal at line 1, column 12

With format=json:

$ vault kv get -mount=secret -format=json -field=certdata cert |jq
[
  "-----BEGIN CERTIFICATE-----\n...\n-----END CERTIFICATE-----",
  "-----BEGIN CERTIFICATE-----\n...\n-----END CERTIFICATE-----"
]

So, maybe vault-action needs something similar? Because I think that the current behavior as of v.2.7.2 is correct since it most closely matches Vault's behavior. I will discuss this with my team this week.

[fairclothjm]

Closed by #478. We reverted to the behavior vault-action has had since v2.1.2.

Updating to vault-action v2.7.3 will resolve this issue.