[BUG] Flytectl authentication not working with GCP IAP

[gverkes]

Describe the bug

After having set up IAP with Flyte according to https://pypi.org/project/flytekitplugins-identity-aware-proxy/, the flytectl command during flyte authentication seems to fail due to IAP. During the Flyte authentication the initial IAP works fine, but during the callback I get the error: Couldn't get access token due to error: oauth2: cannot fetch token: 401 Unauthorized Response: Invalid IAP credentials: empty token. Connecting through Flytekit or the console works fine.

versions

$ flytectl version
{
  "App": "flytectl",
  "Build": "89efcc62a",
  "Version": "0.9.2",
  "BuildTime": "2024-12-06 10:06:41.50991 +0000 WET m=+0.017617709"
}

flyte-core: 1.13.3

Expected behavior

The proxyCommand provides a token for IAP, that should be properly propagated, such that the Flyte authentication succeeds. Just like Flytekit

Additional context to reproduce

1. Setup Flyte + IAP according to the tutorial on https://pypi.org/project/flytekitplugins-identity-aware-proxy/
2. Set config.yaml to something like (also used for flytekit, which does work):

admin:
  endpoint: dns:///example.com
  insecure: false
  insecureSkipVerify: false
  authType: Pkce
  proxyCommand: ["flyte-iap", "generate-user-id-token", "--desktop_client_id", "xxxxx.apps.googleusercontent.com", "--desktop_client_secret_gcp_secret_name", "flyte-desktop-oauth-client-secret", "--webapp_client_id", "xxxxx.apps.googleusercontent.com", "--project", "project-1"]

3. Try to run something like flytectl get project

Screenshots

Are you sure this issue hasn't been raised already?

• Yes

Have you read the Code of Conduct?

• Yes

[welcome]

Thank you for opening your first issue here! 🛠

[eapolinario]

@gverkes , can you confirm that running the flyte-iap command separately works?

Also, can you increase the log level in the invocation of flytectl? You can set the --logger.level flag to 5.

[ademariag]

Hello, I got to the bottom of this, and I can share exactly what is happening.

We followed these instructions to the letter, and we can confirm that the python aspect of things works, so we are able to run the following code successfully.

from flytekit.remote import FlyteRemote

from flytekit.configuration import Config


remote = FlyteRemote(
    config=Config.auto(),
    default_project="flytesnacks",
    default_domain="development",
)


print(remote.recent_executions())

with this configuration file

admin:
  endpoint: dns:///flyte.xxxx
  clientId: flytectl
  insecure: true
  insecureSkipVerify: true
  authType: Pkce
  proxyCommand: ["flyte-iap", "generate-user-id-token", "--desktop_client_id", "xxxxxx-xxxxx.apps.googleusercontent.com", "--desktop_client_secret_gcp_secret_name", "flyte-iap-secret", "--webapp_client_id", "xxxxx-xxxxxjlsgv1t6pj9dnp7khqe.apps.googleusercontent.com", "--project", "xxxxxxearch" ]

The problem we experience is actually only appearing with the golang based code (or at least, with flytectl) and I was able to run a mitmproxy process to see exactly where it is failing

in the specific, we can see that while the GRPC based connections have the correct proxy-authorization: Bearer XXXXXXXXX header, the HTTP connections are lacking that header, and so they fail to pass through the proxy

[ademariag]

According to copilot....

flyte/flyteidl/clients/go/admin/auth_interceptor.go

Line 84 in 588ceec

func (c *proxyAuthTransport) RoundTrip(req *http.Request) (*http.Response, error) {

Your observation is correct. The ProxyAuthorizationHeader is currently set only for grpc connections. To add support for http connections, you'll need to update the code to include the ProxyAuthorizationHeader in the HTTP client as well.

The issue with the current code is that it does not correctly handle the case when metadata[ProxyAuthorizationHeader] is empty or not set. This can happen if the proxy credentials future has not been properly initialized or if the token retrieval fails.

Here's an example of how you might update the auth_interceptor.go file to include the ProxyAuthorizationHeader in the HTTP client:

func (c *proxyAuthTransport) RoundTrip(req *http.Request) (*http.Response, error) {
    // check if the proxy credentials future is initialized
    if !c.proxyCredentialsFuture.IsInitialized() {
        return nil, errors.New("proxy credentials not initialized")
    }

    // Retrieve the token from the proxy credentials future
    token, err := c.proxyCredentialsFuture.Token()
    if err != nil {
        return nil, err
    }

    // Add the ProxyAuthorizationHeader to the request
    req.Header.Set(ProxyAuthorizationHeader, token.AccessToken)

    // Proceed with the round trip
    return c.transport.RoundTrip(req)
}

This modification ensures that the ProxyAuthorizationHeader is included in both grpc and http requests.

[eapolinario]

That is pretty cool, thanks for the investigation. @ademariag, would you be willing to contribute a PR?