Terraform Azure cloud export is not working

[miloskukoleca]

Hello team,

Farfetch is trying to use Terraform to create Azure Cloud export in Kentik. For that effort they were using Kentik provider snippet listed in Terraform provider resource registry
https://registry.terraform.io/providers/kentik/kentik-cloudexport/latest/docs/resources/item#schema

They ran the following terraform snippet and they for HTTP code 401 as a response:

Terraform used the selected providers to generate the following execution plan. Resource actions are indicated with the following symbols:
  + create

Terraform will perform the following actions:

  # kentik-cloudexport_item.this will be created
  + resource "kentik-cloudexport_item" "this" {
      + api_root       = (known after apply)
      + cloud_provider = "azure"
      + current_status = (known after apply)
      + description    = "terraform azure cloud export"
      + enabled        = true
      + flow_dest      = (known after apply)
      + id             = (known after apply)
      + name           = "we1-devops-live"
      + plan_id        = "7890"
      + type           = "CLOUD_EXPORT_TYPE_KENTIK_MANAGED"

      + azure {
          + location                   = "westeurope"
          + resource_group             = "we1con-prd-network-flowlogs-rg-1"
          + security_principal_enabled = true
          + storage_account            = "we1conprdflowlogsstd"
          + subscription_id            = "6ad1292c-bc8c-4400-b787-151192f74911"
        }
    }

Plan: 1 to add, 0 to change, 0 to destroy.

Do you want to perform these actions?
  Terraform will perform the actions described above.
  Only 'yes' will be accepted to approve.

  Enter a value: yes

kentik-cloudexport_item.this: Creating...
╷
│ Error: Failed to create cloud export
│ 
│   with kentik-cloudexport_item.this,
│   on nsg.tf line 1, in resource "kentik-cloudexport_item" "this":
│    1: resource "kentik-cloudexport_item" "this" {
│ 
│ 401 Unauthorized {{ "error": "Unauthorized", "reasonCode": 1, "errxid": "cg0cbb043ks66tqkotsg" }}
╵

In Kibana, we can see the following logs:

March 2nd, 16:36:29.834
{ error: { code: 'AuthenticationFailed', message: "Authentication failed. The 'Authorization' header is missing." }, message: '{"method":"GET","json":true,"timeout":120000,"uri":"https://management.azure.com/subscriptions/02598747-f6a7-49dc-b360-64279cf9badd?api-version=2020-01-01","resolveWithFullResponse":true}', start_time: '2023-03-02T16:36:29.653', categoryName: 'httpRequest', pid: 52008, level: 'INFO', req: '', user_ctx: '', mode: 'ui', worker_id: '' }

vpc-supervisor/transfluo-supervisor [Error] Supervisor>tagger [cid: 28698] <pkg/azurevpc/types/company_tags_manager.go: 102> Error submitting 0 Azure subnets to Portal: Error submitting 0 azure subnets to Portal API for user [email protected] in 532.986528ms: Received non-200/OK status code from HTTP request for Portal subnets API: 401
HTTP request for Portal subnets API: 401

Customer did attempt to use v6 API and managed to create Azure Cloud Export account on Kentik without issues.

In the end they used Hashicorp provider snippet, modified it and managed to create cloud export on Kentik portal:

Initializing the backend...

Initializing provider plugins...
- Reusing previous version of devops-rob/terracurl from the dependency lock file
- Reusing previous version of hashicorp/vault from the dependency lock file
- Using previously-installed devops-rob/terracurl v1.0.1
- Using previously-installed hashicorp/vault v3.10.0

Terraform has been successfully initialized!

You may now begin working with Terraform. Try running "terraform plan" to see
any changes that are required for your infrastructure. All Terraform commands
should now work.

If you ever set or change modules or backend configuration for Terraform,
rerun this command to reinitialize your working directory. If you forget, other
commands will detect it and remind you to do so if necessary.
data.vault_transit_decrypt.password: Reading...
data.vault_transit_decrypt.password: Read complete after 1s [id=dmF1bHQ6djE6TlRVK1BaUUZFdmpFcDdpSzFvS2RTVWpYaXNLQm1DeGdsTWtKenVwYWJJS1QyQ2ZLcmxVcXVibXlhUzlidzNCWmwzR2pzSlBjRWdYNDVCem0=]

Terraform used the selected providers to generate the following execution plan. Resource actions are indicated with the following symbols:
  + create

Terraform will perform the following actions:

  # terracurl_request.get will be created
  + resource "terracurl_request" "get" {
      + destroy_retry_interval = 10
      + headers                = {
          + "X-CH-Auth-API-Token" = (sensitive value)
          + "X-CH-Auth-Email"     = "[email protected]"
          + "accept"              = "application/json"
          + "name"                = "we1_devops_live"
        }
      + id                     = (known after apply)
      + method                 = "GET"
      + name                   = "we1_devops_live"
      + response               = (known after apply)
      + response_codes         = [
          + "200",
          + "204",
        ]
      + retry_interval         = 10
      + url                    = "https://grpc.api.kentik.eu/cloud_export/v202210/exports"
    }

  # terracurl_request.post will be created
  + resource "terracurl_request" "post" {
      + destroy_retry_interval = 10
      + headers                = {
          + "Content-Type"        = "application/json"
          + "X-CH-Auth-API-Token" = (sensitive value)
          + "X-CH-Auth-Email"     = "[email protected]"
          + "accept"              = "application/json"
        }
      + id                     = (known after apply)
      + max_retry              = 1
      + method                 = "POST"
      + name                   = "we1_devops_live"
      + request_body           = jsonencode(
            {
              + export = {
                  + azure         = {
                      + location                 = "westeurope"
                      + resourceGroup            = "we1con-prd-network-flowlogs-rg-1"
                      + securityPrincipalEnabled = true
                      + storageAccount           = "we1conprdflowlogsstd"
                      + subscriptionId           = "6ad1292c-bc8c-4400-b787-151192f74911"
                    }
                  + cloudProvider = "CLOUD_PROVIDER_AZURE"
                  + currentStatus = {}
                  + description   = "lol"
                  + enabled       = true
                  + name          = "we1_devops_live"
                  + planId        = "7890"
                  + type          = "CLOUD_EXPORT_TYPE_KENTIK_MANAGED"
                }
            }
        )
      + response               = (known after apply)
      + response_codes         = [
          + "200",
          + "204",
        ]
      + retry_interval         = 10
      + url                    = "https://grpc.api.kentik.eu/cloud_export/v202210/exports"
    }

Plan: 2 to add, 0 to change, 0 to destroy.

Changes to Outputs:
  + response = {
      + response = (known after apply)
    }

Do you want to perform these actions?
  Terraform will perform the actions described above.
  Only 'yes' will be accepted to approve.

  Enter a value: yes

terracurl_request.post: Creating...
terracurl_request.post: Creation complete after 0s [id=we1_devops_live]
terracurl_request.get: Creating...
terracurl_request.get: Creation complete after 0s [id=we1_devops_live]

Apply complete! Resources: 2 added, 0 changed, 0 destroyed.

Outputs:

response = {
  "response" = "{\"export\":{\"id\":\"392\",\"type\":\"CLOUD_EXPORT_TYPE_KENTIK_MANAGED\",\"enabled\":true,\"name\":\"we1_devops_live\",\"description\":\"lol\",\"planId\":\"7890\",\"cloudProvider\":\"CLOUD_PROVIDER_AZURE\",\"azure\":{\"location\":\"westeurope\",\"resourceGroup\":\"we1con-prd-network-flowlogs-rg-1\",\"storageAccount\":\"we1conprdflowlogsstd\",\"subscriptionId\":\"6ad1292c-bc8c-4400-b787-151192f74911\",\"securityPrincipalEnabled\":true},\"currentStatus\":{\"status\":\"\",\"errorMessage\":\"\",\"flowFound\":false,\"apiAccess\":false,\"storageAccountAccess\":false},\"cdate\":\"2023-03-02T20:25:11.486Z\",\"edate\":\"2023-03-02T20:25:11.486Z\"}}"
}

Kibana logs are also showing that creation was successful:

March 2nd @ 20:47:28.135
vpc-supervisor/transfluo-supervisor [Info] Supervisor <pkg/supervisor/supervisor.go: 435> startTasks: export:azure,s=fra1,u=37352,c=28698,id=393,n="we1_devops_live" -> [[email protected] -plan_id=7890 -export_id=393 -api_root=https://api.kentik.eu/ -dest=https://flow.kentik.eu/ -company_id=28698 -customer_azure_storage_account=we1conprdflowlogsstd -customer_azure_subscription_id=6ad1292c-bc8c-4400-b787-151192f74911 -customer_azure_resource_group=we1con-prd-network-flowlogs-rg-1 -customer_azure_location=westeurope -kentik_env=fra1 -cloud_env=main -device_shard_config= -sampling=false -sampling_rate=0 -log_level=info -metrics=tsdb:https://flow.kentik.com/tsdb -metalisten=:9001] [KENTIK_API_TOKEN=... KT_AZURE_KENTIK_CLIENT_APP_ID=a20ce222-63c0-46db-86d5-58551eeee89f KT_AZURE_KENTIK_CLIENT_APP_SECRET=... KT_AZURE_SERVICE_BUS_CONNECTION_STRING=... KT_AZURE_PREMIUM_SERVICE_BUS_CONNECTION_STRING=... AZURE_ENRICHMENT_SCOPE=null]

March 2nd @ 20:47:56.595
vpc-supervisor/transfluo-supervisor [Info] Supervisor>export:azure,s=fra1,u=37352,c=28698,id=393,n="we1_devops_live" <pkg/supervisor/supervisor.go: 242> started exporter pid:18 args:[[email protected] -plan_id=7890 -export_id=393 -api_root=https://api.kentik.eu/ -dest=https://flow.kentik.eu/ -company_id=28698 -customer_azure_storage_account=we1conprdflowlogsstd -customer_azure_subscription_id=6ad1292c-bc8c-4400-b787-151192f74911 -customer_azure_resource_group=we1con-prd-network-flowlogs-rg-1 -customer_azure_location=westeurope -kentik_env=fra1 -cloud_env=main -device_shard_config= -sampling=false -sampling_rate=0 -log_level=info -metrics=tsdb:https://flow.kentik.com/tsdb -metalisten=:9001]

I discussed this with Ted in #cloud slack channel and he believes that our Terraform implementation for Azure might need to be reviewed and fixed in API, Golang SDK and Terraform provider. He advised me to open an issue here.

Can you please have a look and fix this?

Thanks. Kind regards,
Milos Kukoleca
Kentik CSE EMEA

[stfp]

I'm just popping in here to point out that when you see an error with errxid, such as the one returned when they used our terraform provider, you can search for that ID in Kibana.

So here from the tf output we get an errxid:

401 Unauthorized {{ "error": "Unauthorized", "reasonCode": 1, "errxid": "cg0cbb043ks66tqkotsg" }}

Pasting cg0cbb043ks66tqkotsg in kibana gets us:

WRN not authorized c=server cmd="authserver svc auth" errxid=cg0cbb043ks66tqkotsg op=run reason=1 reasonMsg="bad credentials"

This points to an authentication problem. Not sure how credentials are passed in our tf provider? Maybe there's a typo somewhere?

I'm not sure if the other errors in the ticket are directly related to the tf failure, but I'll stop here. I just wanted to advertise the errxid thing as a way to find logs.

[kentikethan]

I was just able to run the single account example without any issues after a few mods. None of the mods had to do with authentication or the kentik export piece. That part worked without issue.