BUG - GitHub Enterprise Server Support

[ArkShocer]

Describe the bug

When using the action in a GitHub Enterprise Server Environment (github.yourcompany.com), the action cannot use the correct API via the GitHub CLI. Same issue when used in the new GHE.com Instances. It does work as expected when I'm using it in an Github.com Enterprise

      - uses: actions/checkout@v4
      - uses: hashicorp/setup-terraform@v3
      - uses: OP5dev/TF-via-PR@v13
        with:
          # Run plan by default, or apply on merge with lock.
          working-directory: terraform/azure-infra
          command: ${{ github.event_name == 'push' && 'apply' || 'plan' }}
          arg-lock: ${{ github.event_name == 'push' }}
          arg-var-file: variables/dev.tfvars

To Reproduce

Run the action in a GitHub Enterprise Server

Expected behavior

No GitHub CLI API Issue from the action.

Screenshots

Additional context

Looking through the action.yml, I couldn't find anything that the API uses the gh auth login command, so I guess it assumes it is always run on github.com and doesn't need an auth?. I can also provide more logs if needed (pure logs, no screenshot). Also want to point out that I really love this custom made solution! Covers our need perfectly, and no need to use atlantis/digger or some other terraform service which requires some maintenance!

[rdhar]

Also want to point out that I really love this custom made solution! Covers our need perfectly, and no need to use atlantis/digger or some other terraform service which requires some maintenance!

Thanks so much for the kind words! That's exactly what this GH Action is for: a lightweight alternative hosted on ephemeral runners for simple/secure provisioning. Atlantis is bullet-proof, but the lack of OIDC support made us rely on static secrets, which isn't ideal, and other Actions rarely go beyond producing pretty-plans, so TF-via-PR addresses that middle ground by reusing plan files during plan/apply along with all CLI arguments.

When using the action in a GitHub Enterprise Server Environment (github.yourcompany.com), the action cannot use the correct API via the GitHub CLI

On to the actual problem, I was worried this would eventually come up one day -- I'm afraid I don't have access to GH Enterprise to validate functionality. 😅

Going by the docs, it seems the easy approach is to just pass GH_HOST=internal.example.com environment variable (where the default value is github.com). It can be done like so:

- uses: op5dev/tf-via-pr@v13
  with:
    working-directory: terraform/azure-infra
    command: ${{ github.event_name == 'push' && 'apply' || 'plan' }}
    arg-lock: ${{ github.event_name == 'push' }}
    arg-var-file: variables/dev.tfvars
  env:
    GH_HOST: internal.example.com

I think that should alone cover it, and a separate GH_ENTERPRISE_TOKEN isn't required, as this comment suggests.

As an aside, I'm apprehensive of straight-up including gh auth login—not least because it looks complicated!—because I think it's already quite a stretch for users to trust a random GH Action with access to provision their infrastructure. I'd rather not push any further than needed, and that's largely informed how TF-via-PR is designed. Just a big ol' self-contained Bash script, rather than intermingling 3rd party dependencies within a JS/TS-compiled solution. Makes it as easy as possible for SecOps to review & approve.

[ArkShocer]

Thank you for the fast reply. I went ahead and already tried the suggestion. Setting GH_HOST alone resulted in the first error below

Adding GH_ENTERPRISE_TOKEN with enough permissions like this:

  - uses: hashicorp/setup-terraform@v3
  - uses: OP5dev/TF-via-PR@v13
    with:
      working-directory: terraform/azure-infra
      command: ${{ github.event_name == 'push' && 'apply' || 'plan' }}
      arg-lock: ${{ github.event_name == 'push' }}
      arg-var-file: variables/dev.tfvars
      # validate: true
    env:
      GH_HOST: internal.example.com
      GH_ENTERPRISE_TOKEN: ${{ secrets.GH_ENTERPRISE_TOKEN }}

results in a "better" response, as it goes and installs the providers but still errors out in the end :(

Maybe it's an issue with the API path as the server uses internal.example.com/api/v3 but also setting that as GH_HOST did not help. To me it looks like it can find some correct endpoints that are shared across cloud and server?

[rdhar]

Just as a sanity check for me, can you confirm that you've replaced the "internal.example.com" with your actual internal/enterprise GitHub URL? I'm almost certain that you have, and the screenshot is obfuscated (including the "random-id-hash" tfplan), but wanted to double-check, just in case!

Assuming that's all kosher, I think you've hit it on the head when you say, "it looks like it can find some correct endpoints that are shared across cloud and server." I think it might because I missed one of the defaults assumed by this Action.

Specifically, token input:

  - uses: op5dev/tf-via-pr@v13
    with:
      working-directory: terraform/azure-infra
      command: ${{ github.event_name == 'push' && 'apply' || 'plan' }}
      arg-lock: ${{ github.event_name == 'push' }}
      arg-var-file: variables/dev.tfvars
      token: ${{ secrets.GH_ENTERPRISE_TOKEN }} # <- this li'l stinker
    env:
      GH_HOST: internal.example.com
      GH_ENTERPRISE_TOKEN: ${{ secrets.GH_ENTERPRISE_TOKEN }} # <- might not be required then

Since token: ${{ github.token }} is set by default, it likely overrode the GH_ENTERPRISE_TOKEN environment variable. Lemme know how it works out!

[ArkShocer]

Yes, I can confirm I've replaced the internal.example.com with the actual domain (ghe.mycompany.com) of the GitHub Server instance. I've tried setting token: ${{ secrets.GH_ENTERPRISE_TOKEN }} and token: ${{ secrets.GITHUB_TOKEN }} both result in the same issue as before. I've noticed that if I don't set env: GH_ENTERPRISE_TOKEN: ${{ secrets.GH_ENTERPRISE_TOKEN }} at all, the error goes back to the first Image, so it looks like we are not on the wrong track so far.

[rdhar]

Just to humour me, what happens if you add the following gh auth step before calling TF-via-PR, along with all the GH_ environment variables?

I'm curious if GH CLI authentication is transferred across workflow steps.

- run: gh auth login --hostname ${{ env.GH_HOST }}
  env:
    GH_HOST: internal.example.com
    GH_TOKEN: ${{ secrets.GH_ENTERPRISE_TOKEN }}
    GH_ENTERPRISE_TOKEN: ${{ secrets.GH_ENTERPRISE_TOKEN }}

- uses: op5dev/tf-via-pr@v13
  with:
    working-directory: terraform/azure-infra
    command: ${{ github.event_name == 'push' && 'apply' || 'plan' }}
    arg-lock: ${{ github.event_name == 'push' }}
    arg-var-file: variables/dev.tfvars
    token: ${{ secrets.GH_ENTERPRISE_TOKEN }}
  env:
    GH_HOST: internal.example.com
    GH_TOKEN: ${{ secrets.GH_ENTERPRISE_TOKEN }}
    GH_ENTERPRISE_TOKEN: ${{ secrets.GH_ENTERPRISE_TOKEN }}

This configuration is definitely over-the-top, but I wanna see it working first before removing 1-line/variable at a time.

By the way, can you confirm the workflow job has the relevant permissions, and is compatible with GitHub Enterprise Server 2.20 and above?

permissions:
  actions: read # Required to identify workflow run.
  checks: write # Required to add status summary.
  contents: read # Required to checkout repository.
  id-token: write # Required to authenticate via OIDC (optional).
  pull-requests: write # Required to add comment and label.

[ArkShocer]

Running the following;

- run: gh auth login --hostname ${{ env.GH_HOST }}
  env:
    GH_HOST: internal.example.com
    GH_TOKEN: ${{ secrets.GH_ENTERPRISE_TOKEN }}
    GH_ENTERPRISE_TOKEN: ${{ secrets.GH_ENTERPRISE_TOKEN }}

Results in

Removing GH_ENTERPRISE_TOKEN: ${{ secrets.GH_ENTERPRISE_TOKEN }} results in a long run without any error as it waits for browser input, so I've aborted it. You also don't see which code it is until you abort the run.

I've now added the id-token: write permission, still same issues. We are running at nearly the latest Server Version (3.15.1)

[rdhar]

Alright, interesting, that'd seem to indicate to me that the gh auth login approach is a lost-cause since it's only applicable for local-use, whereas the CI context of GH Actions requires token-based authentication.

If you haven't already, would you mind removing the gh auth login step altogether, then just trying TF-via-PR with all the GH_ environment variables + token input like so:

- uses: op5dev/tf-via-pr@v13
  with:
    working-directory: terraform/azure-infra
    command: ${{ github.event_name == 'push' && 'apply' || 'plan' }}
    arg-lock: ${{ github.event_name == 'push' }}
    arg-var-file: variables/dev.tfvars
    token: ${{ secrets.GH_ENTERPRISE_TOKEN }}
  env:
    GH_HOST: internal.example.com
    GH_TOKEN: ${{ secrets.GH_ENTERPRISE_TOKEN }}
    GH_ENTERPRISE_TOKEN: ${{ secrets.GH_ENTERPRISE_TOKEN }}

[ArkShocer]

Okay, some big progress as it finally progressed to actually running terraform plan successfully! Sadly, the next Issue arrived as some actions that are used are not supported on GHES apparently (see: https://github.com/actions/upload-artifact?tab=readme-ov-file#v4---whats-new (upload-artifact@v4+ is not currently supported on GHES yet. If you are on GHES, you must use v3.)). I think it could also error out because some actions are locally cached with each version upgrade, which means it doesn't have the newest one from the cloud available. I can go into the Admin Config and delete the action cache so it uses the newest one from the cloud.

Current working one as you suggested

  - uses: OP5dev/TF-via-PR@v13
    with:
      # Run plan by default, or apply on merge with lock.
      working-directory: terraform/azure-infra
      command: plan
      arg-lock: ${{ github.event_name == 'push' }}
      arg-var-file: variables/dev.tfvars
      token: ${{ secrets.GH_ENTERPRISE_TOKEN }}
    env:
      GH_HOST: ghe.example.xyz
      GH_TOKEN: ${{ secrets.GH_ENTERPRISE_TOKEN }}
      GH_ENTERPRISE_TOKEN: ${{ secrets.GH_ENTERPRISE_TOKEN }}

[ArkShocer]

Guess v4 is not gonna be an Option github/roadmap#930 :/

[rdhar]

Gah, so close, yet so far!

Current working one as you suggested

• I'd start by removing the GH_TOKEN: line as I don't think it's contributing anything at all.
• I'd try to see if it works with token: only or GH_ENTERPRISE_TOKEN: only, instead of both together.

upload-artifact@v4+ is not currently supported on GHES yet

I can't believe that the one, the ONE, dependency I thought was safe due to 1st party/native support has let us down. How can GitHub deprecate v3 last year, without updating support for GHE as well—ridiculous!

At this point, I don't think your GHE instance will be able to make any use of TF-via-PR since "actions/upload-artifact" is a core dependency (for storage/retrieval of plan file)

… The only other viable option I can see is to include conditional support for upload-artifact@v3, primarily for GHE users. Unless I'm way-off, I don't think it'd be too involved to determine whether the Action is running within the context of a GHE and, if so, fallback on v3 instead of v4. 🤔

[rdhar]

Only if you feel comfortable, could I request you to try PR #401 branch in order to validate support for GHE as follows:

  - uses: op5dev/tf-via-pr@feat--ghe-support
    with:
      working-directory: terraform/azure-infra
      command: ${{ github.event_name == 'push' && 'apply' || 'plan' }}
      arg-lock: ${{ github.event_name == 'push' }}
      arg-var-file: variables/dev.tfvars
      token: ${{ secrets.GH_ENTERPRISE_TOKEN }}
    env:
      GH_ENTERPRISE_TOKEN: ${{ secrets.GH_ENTERPRISE_TOKEN }}

• Assuming that works, would you mind retrying without token input;
• Then retrying again with token input but without GH_ENTERPRISE_TOKEN env?

Once the minimum-necessary config is established, we can cut a new release of TF-via-PR with built-in GHE support & documentation.

[ArkShocer]

Hey, so I've tested the new branch with both token and GH_ENTERPRISE_TOKEN and this time it was able to create an artifact! There is still an error showing up in the last part of the action.

[rdhar]

Terrific, looks like progress! Albeit, the lack of helpful error message is far from ideal.

1. Was it at least able to create a PR comment?
2. Was it at least able to post on the workflow summary page?

The 403 error leads me to believe the last hurdle maybe related to the permissions granted to the GitHub token.

Would you mind retrying the PR branch as before, but without the token input to compare differences, if any?

[ArkShocer]

I've got an update. You writing ... maybe related to the permissions ... gave me the Idea of trying the env GH_ENTERPRISE_TOKEN with the global GITHUB_TOKEN secret - and it worked! So using a custom PAT-Token as suggested here (cli/cli#3522 (comment)) from the issue you linked earlier was not entirely wrong. Setting the env GH_TOKEN with ${{ secrets.GITHUB_TOKEN }} did nothing and gave me the same error I first encountered but with enterprise it worked. The PAT-Token {{ secrets.GH_ENTERPRISE_TOKEN }} would probably also work if I gave it more permissions.

Here is the full working yaml

jobs:
  provision:
    runs-on: ubuntu-latest

    permissions:
      actions: read        # Required to identify workflow run.
      checks: write        # Required to add status summary.
      contents: read       # Required to checkout repository.
      pull-requests: write # Required to add comment and label.

    env:
      ARM_CLIENT_ID: ${{ secrets.AZURE_CLIENT_ID }}
      ARM_TENANT_ID: ${{ secrets.AZURE_TENANT_ID }}
      ARM_SUBSCRIPTION_ID: ${{ secrets.AZURE_SUBSCRIPTION_ID }}
      ARM_CLIENT_SECRET: ${{ secrets.AZURE_CLIENT_SECRET }}

    steps:     
      - uses: actions/checkout@v4
      - uses: hashicorp/setup-terraform@v3
      - uses: OP5dev/TF-via-PR@feat--ghe-support
        with:
          # Run plan by default, or apply on merge with lock.
          working-directory: terraform/azure-infra
          command: plan
          arg-lock: ${{ github.event_name == 'push' }}
          arg-var-file: variables/dev.tfvars
        env:
          GH_ENTERPRISE_TOKEN: ${{ secrets.GITHUB_TOKEN }}

I do still see an API issue as before I tested it without a PR but now with one I can see the following in the logs

The API did set a label on the PR, guess it has an Issue with getting it again because it does not exist on the org-level bot on repo-level?

[rdhar]

That's it, I think you've done it!

That PR label creation "error" is only reported the first time that "tf:plan" and "tf:apply" labels are created. It's a harmless annotation that doesn't have any effect on the actual outcome of the workflow, and won't appear for that repo again.

Thanks once again for testing the inclusion/exclusion of various token inputs, we can package our learning into the next release today so you're free to pin TF-via-PR@v13 instead of a PR branch!

[ArkShocer]

Ah, okay, then I'm glad it ran through without any Issue 😄! I'm now going to build the actual full workflow for our needs. If you want, I can reply here with new Issues I encounter or you can close this one and I create a new Issue once I've encountered one. I'm fine with either way. Thank you for your fast help in solving this Issue!