Azure vnet flow logs support in tf module (#78)

* Azure vnet flow logs support in tf module * filter rg's without any vnets * access vnet values * fix rg name, key values * fix rg name, key values * name * update provider versions in ci tests * update docs, replace nsg w/ vnet * vnet id outputs * remove local-exec provisioner null resources * retire get_nsg.py and remove unused requirements * cleanup service_principal creation * modify azure sa name generation * fix sa name * index sa's for uniqueness * fix * count with random_id on rg list' * add count index to local._names * more index * comment * add bd team as codeowners * comment --------- Co-authored-by: jksprattler <[email protected]>
kentik · Jan 24, 2025 · 058b50d · 058b50d
1 parent 867d757
commit 058b50d
Show file tree

Hide file tree

Showing 18 changed files with 104 additions and 187 deletions.
diff --git a/.github/CODEOWNERS b/.github/CODEOWNERS
@@ -0,0 +1 @@
+*	@kentik/bd
diff --git a/README.md b/README.md
@@ -12,7 +12,7 @@
 #### Single VPC, Single Region
 * [single-vpc](https://github.com/kentik/config-snippets-cloud/tree/master/cloud_AWS/terraform/module/examples/single-vpc)
 #### All VPC, Single Region
-* [all-vpc-from-region](https://github.com/kentik/config-snippets-cloud/tree/master/cloud_AWS/terraform/module/examples/all-vpc-from-region) 
+* [all-vpc-from-region](https://github.com/kentik/config-snippets-cloud/tree/master/cloud_AWS/terraform/module/examples/all-vpc-from-region)
 #### Deploy Sock Shop as an example micro-service architecture
 * [sock-shop-eks](https://github.com/kentik/config-snippets-cloud/tree/master/cloud_AWS/terraform/module/examples/sock-shop-eks)
 
@@ -28,8 +28,6 @@
 # Stage 2 - Automate GCP
 ## Terraform
 * [Terraform](https://github.com/kentik/config-snippets-cloud/tree/master/cloud_GCP/terraform)
-### Demo
-* [Terraform Demo](https://github.com/kentik/config-snippets-cloud/tree/master/cloud_GCP/terraform/module/demo) (TODO)
 ### Examples
 #### Subnet-list, Single region
 * [subnet-list](https://github.com/kentik/config-snippets-cloud/tree/master/cloud_GCP/terraform/module/examples/subnet-list)
@@ -38,27 +36,20 @@
 
 ## Ansible
 * [Ansible](https://github.com/kentik/config-snippets-cloud/tree/master/cloud_GCP/terraform)
-### Demo
-* [Ansible Demo](https://github.com/kentik/config-snippets-cloud/tree/master/cloud_GCP/terraform/module/demo)(TODO)
 
 # Stage 3 - Automate Azure
 ## Terraform
 * [Tearraform](https://github.com/kentik/config-snippets-cloud/tree/master/cloud_Azure/terraform)
-### Demo
-* [Terraform Demo](https://github.com/kentik/config-snippets-cloud/tree/master/cloud_Azure/terraform/module/demo) (TODO)
 ### Examples
-#### Subnet-list, Single region
-* [all_nsg](https://github.com/kentik/config-snippets-cloud/tree/master/cloud_Azure/terraform/module/examples/all_nsg)
+#### All Virtual Networks from multiple Resource Groups
+* [single_account_multiple_resource_groups](https://github.com/kentik/config-snippets-cloud/tree/master/cloud_Azure/terraform/module/examples/single_account_multiple_resource_groups)
 
 ## Ansible
 * [Ansible](cloud_Azure/ansible/roles/kentik_az)
 ### Examples
 #### All NSG from resource group
 * [all_nsg](cloud_Azure/ansible/examples/all_nsg)
 
-# Stage 4 - Automate IBM Cloud
-## Timing TBD
-
 # General needs for automation
 ## Identity and Access Management
 ## Creation of Storage location

diff --git a/cloud_Azure/terraform/module/README.md b/cloud_Azure/terraform/module/README.md
@@ -3,23 +3,23 @@
 Module supporting management of Azure and Kentik resources required for flow log export from Azure to Kentik.
 
 Module enables:
-* Flow logs in all Network Security Groups (NSG) found in requested Resource Groups
+* Flow logs in all Virtual Networks (VNets) found in requested Resource Groups
 
 Module creates:
-* Service Principal for Kentik NSG Flow Exporter application
+* Service Principal for Kentik VNet Flow Exporter application
 * Reader and Contributor Roles for above mentioned Service Principal
 * One Storage Account for flow logs per requested Resource Group
 * Registers flow in Kentik platform per requested Resource Group
 
 All resources created in Azure are tagged, see variable "resource_tag" in [variables.tf](./variables.tf)
 
-Module assumes that NetworkWatcher resource exists in NetworkWatcherRG resource group in specified Azure location (see variable "location" in [variables.tf](./variables.tf)).  
-For example, in location "eastus" there should be "NetworkWatcher_eastus" in "NetworkWatcherRG" resource group.  
+Module assumes that NetworkWatcher resource exists in NetworkWatcherRG resource group in specified Azure location (see variable "location" in [variables.tf](./variables.tf)).
+For example, in location "eastus" there should be "NetworkWatcher_eastus" in "NetworkWatcherRG" resource group.
 NetworkWatcher is automatically created by Azure when VirtualNetwork is created or updated, [as per documentation.](https://docs.microsoft.com/en-us/azure/network-watcher/network-watcher-create). This happens eg. when launching a new virtual machine.
 
 ## Usage examples
 
-* [All Network Security Groups in requested Resource Groups in single Azure Account](examples/single_account_multiple_resource_groups)
+* [All Virtual Networks in requested Resource Groups in single Azure Account](examples/single_account_multiple_resource_groups)
 * [All Network Security Groups in requested Resource Groups in multiple Azure Accounts](examples/multiple_accounts_multiple_resource_group)
 
 ## Requirements
@@ -40,27 +40,6 @@ NetworkWatcher is automatically created by Azure when VirtualNetwork is created
 | null | >= 2.1.2 |
 | external | >= 2.0.0 |
 
-## Python and dependencies
-
-This module uses Python script to list all Network Security Groups in specified Resource Groups and exposes the list to Terraform as external data source.  
-To install Python and required packages:
-* [Install Python and PIP](https://docs.python.org/3/using/index.html)
-* Install packages - in module directory, execute:  
-    PowerShell:
-    ```powershell
-    pip install virtualenv
-    virtualenv venv
-    .\venv\Scripts\activate
-    pip install -r requirements.txt
-    ```
-
-    or Bash:
-    ```bash
-    pip install virtualenv
-    virtualenv venv
-    source venv/bin/activate
-    pip install -r requirements.txt
-    ```
 ## Inputs
 
 | Name | Description | Type | Default | Required |
@@ -71,20 +50,20 @@ To install Python and required packages:
 | email | Kentik account email | `string` | none | yes |
 | token | Kentik account token | `string` | none | yes |
 | plan_id | Kentik billing plan ID | `string` | none | yes |
-| name | Cloudexport entry name in Kentik | `string` | none | yes |
+| name | Cloudexport entry name in Kentik will be appended with: resource_group_names and subscription_id to ensure uniqueness | `string` | none | yes |
 | enabled | Defines if cloud export to Kentik is enabled | `bool` | true | no |
 | description | Cloudexport entry description in Kentik | `string` | `Created using Terraform` | no |
 | resource_tag | Azure Tag value to apply to created resources | `string` | `flow_log_exporter` | no |
-| flow_exporter_application_id | Kentik NSG Flow Exporter application ID | `string` | `a20ce222-63c0-46db-86d5-58551eeee89f` | no |
-| storage_account_names | Names of Storage Accounts for storing flow logs. Names must meet Azure Storage Account naming restrictions.<br>The list should either contain 1 Storage Account name for each Resource Group, or be empty, in which case names will be generated automatically. | `list of strings` | `[]` | no |
+| flow_exporter_application_id | Kentik VNet Flow Exporter application ID | `string` | `a20ce222-63c0-46db-86d5-58551eeee89f` | no |
+| storage_account_names | Names of Storage Accounts to be created for storing flow logs. Names must meet Azure Storage Account naming restrictions.<br>The list should either contain 1 Storage Account name for each Resource Group, or be empty, in which case names will be generated automatically. Auto-generated names will use the first 12 characters of the `var.name` for the Cloudexport as a prefix appended with a random id of 12 characters for global uniqueness. | `list of strings` | `[]` | no |
 
 
 ## Outputs
 
 | Name | Description |
 |------|-------------|
-| network_security_groups | Id's of the Network Security Groups which flow logs will be collected |
+| vnet_ids | Id's of the Virtual Networks which to collect flow logs |
 | subscription_id | Azure subscription ID |
 | resource_group_names | Names of Resource Groups from which to collect flow logs |
 | storage_accounts | Storage Account names where flow logs will be collected |
-| principal_id | Service Principal ID created for Kentik NSG Flow Exporter application |
+| principal_id | Service Principal ID created for Kentik VNet Flow Exporter application |
diff --git a/cloud_Azure/terraform/module/cloudexport.tf b/cloud_Azure/terraform/module/cloudexport.tf
@@ -14,9 +14,9 @@ provider "kentik-cloudexport" {
 
 # Creates one Kentik CloudExport for every requested Resource Group
 resource "kentik-cloudexport_item" "azure_export" {
-  count = length(var.resource_group_names)
+  for_each = { for rg in var.resource_group_names : rg => rg }
 
-  name           = "${var.name}-${var.resource_group_names[count.index]}-${var.subscription_id}" # resource group name + subscription id make the name unique
+  name           = "${var.name}-${each.key}-${var.subscription_id}" # resource group name + subscription id make the name unique
   type           = "CLOUD_EXPORT_TYPE_KENTIK_MANAGED"
   enabled        = var.enabled
   description    = var.description
@@ -25,8 +25,8 @@ resource "kentik-cloudexport_item" "azure_export" {
   azure {
     subscription_id            = var.subscription_id
     location                   = var.location
-    resource_group             = var.resource_group_names[count.index]
-    storage_account            = azurerm_storage_account.logs_storage_account[count.index].name # storage accounts are mapped to resource groups 1:1
+    resource_group             = each.key
+    storage_account            = azurerm_storage_account.logs_storage_account[each.key].name # storage accounts are mapped to resource groups 1:1
     security_principal_enabled = true
   }
-}
+}
diff --git a/...ure/terraform/module/examples/single_account_multiple_resource_groups/README.md b/...ure/terraform/module/examples/single_account_multiple_resource_groups/README.md
@@ -15,7 +15,7 @@ None.
     location = "eastus"
     resource_group_names = ["resource-group-1", "resource-group-2", "resource-group-3"] # groups must exist in selected location
     storage_account_names = []
-    
+
     # Kentik
     email = "[email protected]"
     token = "dummy_token"
@@ -29,8 +29,6 @@ None.
 
 1. Execute:
     ```bash
-    virtualenv venv && source venv/bin/activate
-    pip install -r ../../requirements.txt
     terraform init
     ```
 

diff --git a/cloud_Azure/terraform/module/examples/single_account_multiple_resource_groups/main.tf b/cloud_Azure/terraform/module/examples/single_account_multiple_resource_groups/main.tf
@@ -3,11 +3,11 @@ terraform {
   required_providers {
     azurerm = {
       source  = "hashicorp/azurerm"
-      version = "~> 3.10"
+      version = "~> 4.15"
     }
     azuread = {
       source  = "hashicorp/azuread"
-      version = "~> 2.24"
+      version = "~> 3.0"
     }
     kentik-cloudexport = {
       source  = "kentik/kentik-cloudexport"

diff --git a/cloud_Azure/terraform/module/examples/single_account_multiple_resource_groups/output.tf b/cloud_Azure/terraform/module/examples/single_account_multiple_resource_groups/output.tf
@@ -1,7 +1,7 @@
 
-output "network_security_groups" {
-  value       = module.kentik_azure_integration.network_security_groups
-  description = "Id's of the Network Security Groups which flow logs will be collected"
+output "vnet_ids" {
+  value       = module.kentik_azure_integration.vnet_ids
+  description = "Id's of the Virtual Networks from which to collect flow logs"
 }
 
 output "subscription_id" {
@@ -22,4 +22,4 @@ output "storage_accounts" {
 output "principal_id" {
   value       = module.kentik_azure_integration.principal_id
   description = "Service Principal ID created for Kentik NSG Flow Exporter application"
-}
+}
diff --git a/cloud_Azure/terraform/module/get_nsg.py b/cloud_Azure/terraform/module/get_nsg.py
diff --git a/cloud_Azure/terraform/module/network_watcher.tf b/cloud_Azure/terraform/module/network_watcher.tf
@@ -5,55 +5,42 @@ data "azurerm_network_watcher" "network_watcher" {
   resource_group_name = "NetworkWatcherRG"
 }
 
-# Runs python script to gather network security groups from each requested resource group
-# This is required because no Terraform provider exposes such functionality
-# Resulting "data.external.nsg_data_source.results" is a map of string -> string, eg.
-# {
-#   "ResourceGroupName1" -> "NetworkSercurityGroupId1,NetworkSecurityGroupId2",
-#   "ResourceGroupName2" -> "NetworkSercurityGroupId3,NetworkSecurityGroupId4"
-# }
-data "external" "nsg_data_source" {
-  program = ["python3", "${path.module}/get_nsg.py"]
-  query = {
-    resource_group_names = join(",", var.resource_group_names)
-  }
+# Fetch all VNets for each resource group
+data "azurerm_resources" "vnet" {
+  for_each            = toset(var.resource_group_names)
+  type                = "Microsoft.Network/virtualNetworks"
+  resource_group_name = each.key
 }
 
-# Convert map of string -> string:
-# {
-#   "ResourceGroupName1" -> "NetworkSercurityGroupId1,NetworkSecurityGroupId2",
-#   "ResourceGroupName2" -> "NetworkSercurityGroupId3,NetworkSecurityGroupId4"
-# }
-# to list of objects:
-# [
-#   {rg = "ResourceGroupName1", nsg = "NetworkSercurityGroupId1"},
-#   {rg = "ResourceGroupName1", nsg = "NetworkSercurityGroupId2"},
-#   {rg = "ResourceGroupName2", nsg = "NetworkSercurityGroupId3"},
-#   {rg = "ResourceGroupName2", nsg = "NetworkSercurityGroupId4"}
-# ]
+# Map resource group names to their corresponding VNets
+# Flatten map to list of objects
 locals {
-  flat_nsgs = flatten([
-    for rg, nsg_list in data.external.nsg_data_source.result : [
-      for nsg in split(",", nsg_list) : {
-        rg  = rg  # Resource Group name
-        nsg = nsg # Network Security Group ID
+  flat_vnets = flatten([
+    for rg in var.resource_group_names : [
+      for vnet in data.azurerm_resources.vnet[rg].resources : {
+        key = "${rg}-${vnet.name}"
+        value = {
+          rg   = rg
+          name = vnet.name
+          id   = vnet.id
+        }
       }
-    ] if length(nsg_list) > 0 # filter out Resource Groups that have no Network Security Groups
+    ] if length(data.azurerm_resources.vnet[rg].resources) > 0 # filter out resource groups without VNets
   ])
 }
 
-# Turns on flow logs for all network security groups in requested resource groups
+# Turns on vnet flow logs for all vnets in requested resource groups
 resource "azurerm_network_watcher_flow_log" "kentik_network_flow_log" {
-  count = length(local.flat_nsgs)
+  for_each = { for vnet in local.flat_vnets : vnet.key => vnet.value }
 
-  name = "${var.name}_flow_log_${count.index}"
+  name                 = "${var.name}-flowLog-${each.value.name}"
   network_watcher_name = data.azurerm_network_watcher.network_watcher.name
-  resource_group_name  = data.azurerm_network_watcher.network_watcher.resource_group_name
+  resource_group_name  = "NetworkWatcherRG"
 
-  network_security_group_id = local.flat_nsgs[count.index].nsg
-  storage_account_id        = azurerm_storage_account.logs_storage_account[index(var.resource_group_names, local.flat_nsgs[count.index].rg)].id
-  enabled                   = true
-  version                   = 2
+  target_resource_id = each.value.id
+  storage_account_id = azurerm_storage_account.logs_storage_account[each.value.rg].id
+  enabled            = true
+  version            = 2
   retention_policy {
     enabled = true
     days    = 7

diff --git a/cloud_Azure/terraform/module/output.tf b/cloud_Azure/terraform/module/output.tf
@@ -1,6 +1,6 @@
-output "network_security_groups" {
-  value       = [for v in local.flat_nsgs : v.nsg]
-  description = "Id's of the Network Security Groups which flow logs will be collected"
+output "vnet_ids" {
+  value       = [for vnet in local.flat_vnets : vnet.value.id]
+  description = "Id's of the Virtual Networks from which to collect flow logs"
 }
 
 output "subscription_id" {
@@ -14,11 +14,11 @@ output "resource_group_names" {
 }
 
 output "storage_accounts" {
-  value       = azurerm_storage_account.logs_storage_account[*].name
+  value       = [for sa in azurerm_storage_account.logs_storage_account : sa.name]
   description = "Storage Account names where flow logs will be collected"
 }
 
 output "principal_id" {
-  value       = local.kentik_nsg_flow_exporter_id
-  description = "Service Principal ID created for Kentik NSG Flow Exporter application"
-}
+  value       = local.kentik_vnet_flow_exporter_id
+  description = "Service Principal ID created for Kentik VNet Flow Exporter application"
+}
diff --git a/cloud_Azure/terraform/module/providers.tf b/cloud_Azure/terraform/module/providers.tf
diff --git a/cloud_Azure/terraform/module/requirements.txt b/cloud_Azure/terraform/module/requirements.txt
diff --git a/cloud_Azure/terraform/module/roles.tf b/cloud_Azure/terraform/module/roles.tf
@@ -1,17 +1,17 @@
 # Provide service principal Contributor role to each storage account
-resource "azurerm_role_assignment" "kentic_role_contributor" {
-  count = length(azurerm_storage_account.logs_storage_account)
+resource "azurerm_role_assignment" "kentik_role_contributor" {
+  for_each = azurerm_storage_account.logs_storage_account
 
-  scope                = azurerm_storage_account.logs_storage_account[count.index].id
+  scope                = each.value.id
   role_definition_name = "Contributor"
-  principal_id         = local.kentik_nsg_flow_exporter_id
+  principal_id         = local.kentik_vnet_flow_exporter_id
 }
 
 # Provide service principal Reader role to each Resource Group
-resource "azurerm_role_assignment" "kentic_role_reader" {
-  count = length(var.resource_group_names)
+resource "azurerm_role_assignment" "kentik_role_reader" {
+  for_each = toset(var.resource_group_names)
 
-  scope                = "/subscriptions/${var.subscription_id}/resourceGroups/${var.resource_group_names[count.index]}"
+  scope                = "/subscriptions/${var.subscription_id}/resourceGroups/${each.value}"
   role_definition_name = "Reader"
-  principal_id         = local.kentik_nsg_flow_exporter_id
+  principal_id         = local.kentik_vnet_flow_exporter_id
 }