AWS MWAA compatibility request

[dylan-shipwell]

the authentication modes:

• basic (username+password)
• oauth2_token

are incompatible with AWS managed Apache Airflow.

I went through the and issued an MWAA login token, aws doc, however this token is only valid for the forced api method /aws_mwaa/aws-console-sso?login=true#{token}. Which in turn probably vends an browser session cookie.

I could not find a way to convert an active MWAA session into either a username + password (creating a user is possible in MWAA but i didn't find a way to assign a password and even if I could, mwaa does not present a login page anywhere that I found), or into an oauth2 token, i was hoping there was a simple apache airflow method for vending oauth2 tokens but I did not see one.

My goal is to support MWAA's unusual login modes, though I understand if managing a cookie jar to make mwaa api requests is impossible due to technical reasons.

Thank You

[obax]

@dylan-shipwell Have you found a solution in the end?

[dylan-shipwell]

@obax sorry, no new info.

semi-related, word of caution, I have tried to use https://registry.terraform.io/providers/Mastercard/restapi/ as a terraform-facing HTTP client, Masterard/restapi which claims to be built with the goal of being the "curl of terraform" except it doesn't expose nearly any curl option that would be required to do anything that curl can conventionally do, it's really specifically for one and only one flavor of REST JSON Web API.

so, in cases where I need to orchestestrate http requests directly, i've been using a nasty hack like this, where i have a custom http client ex. named "my_http_client_script" that will receive the environment variable ex. named "TF_NULL_RESOURCE_JSON" which contains all of the dynamic parts of the payload that terraform is the input for.

that can do whatever, like make a proper curl request with sufficient options to accomplish the http state mutation intended.

# main.tf
resource "null_resource" "my_http_state_name" {
  triggers = {
    my_input_var = var.my_input_var
    my_http_payload = sha512(local.my_http_payload)
  
    # timestamp = "${timestamp()}" # debug, enable to always recreate
  }

  provisioner "local-exec" {
    command = "${path.module}/my_http_client_script"
    environment = {
      TF_NULL_RESOURCE_JSON = jsonencode({
        my_http_param = "https://example.org/my_api_method/"
        my_data_param = local.my_http_payload
        # ...
      }) # use an environ to transfer secrets like HTTP Authorization header value
    }
    on_failure = fail
  }
}

but, this is extremely DIY and fragile.
if using this pattern, i recommend logging when "my_http_client_script" starts running and just before it exits, as well as logging its PID on every log message because terraform will not report diagnostic info.

this doesn't solve anything to do with MWAA though. if you want to manage MWAA with terraform, it might be possible to manage MWAA with this script and have this script be auto-run by terraform with a hook like this. but this would 100% bypass the terraform-provider-airflow tooling specifically because

terraform-provider-airflow and MWAA have no compatible authentication modes that i'm aware of at time of writing 2024-02-13

Thanks!

[DrFaust92]

Out of the scope of this provider but i may be able to create a mwaa token data source in the aws provider to use it here :D how does that sound?

[obax]

Thanks, @dylan-shipwell, for the detailed explanation and the word of caution. I've also wondered if I could curl my way into making this happen. I wish the approach to getting this done was more abstracted instead of having to make scripts. @DrFaust92 - super interested in your solution

[cdeneen-adelaide]

AWS added support for restapi requests directly
https://aws.amazon.com/blogs/big-data/introducing-amazon-mwaa-support-for-the-airflow-rest-api-and-web-server-auto-scaling/

The authentication relies on a session token in the header.