Bug Report - AMA Policy - UAMI Permission

[HeiGeorge]

Community Note

• Please vote on this issue by adding a 👍 reaction to the original issue to help the community and maintainers prioritize this request
• Please do not leave "+1" or "me too" comments, they generate extra noise for issue followers and do not help prioritize the request
• If you are interested in working on this issue or have submitted a pull request, please leave a comment

Versions

terraform: 1.5.7

azure provider: 3.90.0

module: "Azure/caf-enterprise-scale/azurerm" - 5.2.1

Description

Describe the bug

Assigning the policy "Enable Azure Monitor for VMs with Azure Monitoring Agent(AMA)" (/providers/Microsoft.Authorization/policySetDefinitions/924bfe3a-762f-40e7-86dd-5c8b95eb09e6) with an UAMI in a central subscription is restricting certain VM operations on Landing Zones like resizing a VM or moving a VM to an availability zone.

Steps to Reproduce

Policy Assignment

{
  "type": "Microsoft.Authorization/policyAssignments",
  "apiVersion": "2022-06-01",
  "name": "Deploy-AMA-Monitoring",
  "location": "${default_location}",
  "dependsOn": [],
  "identity": {
    "type": "SystemAssigned"
  },
  "properties": {
    "description": "Enable Azure Monitor for the virtual machines (VMs) in the specified scope (management group, subscription or resource group). Takes Log Analytics workspace as parameter.",
    "displayName": "Enable Azure Monitor Agent for VMs",
        "policyDefinitionId": "/providers/Microsoft.Authorization/policySetDefinitions/924bfe3a-762f-40e7-86dd-5c8b95eb09e6",
    "enforcementMode": "Default",
    "nonComplianceMessages": [
      {
        "message": "Azure Monitor {enforcementMode} be enabled for Virtual Machines."
      }
    ],
    "parameters": {
      "dcrResourceId": {
        "value": "/subscriptions/00000000-0000-0000-0000-000000000000/resourcegroups/resource_group-name/providers/microsoft.insights/datacollectionrules/DCR-name"
      },
      "bringYourOwnUserAssignedManagedIdentity": {
        "value": true
      },
      "restrictBringYourOwnUserAssignedIdentityToSubscription": {
        "value": false
      },
      "userAssignedIdentityResourceId": {
        "value": "/subscriptions/00000000-0000-0000-0000-000000000001/resourceGroups/resource_group-name/providers/Microsoft.ManagedIdentity/userAssignedIdentities/UAMI-name"
      },
      "enableProcessesAndDependencies": {
        "value": true
      },
      "scopeToSupportedImages": {
        "value": false
      }
    },
    "scope": "${current_scope_resource_id}",
    "notScopes": ${exclude_rgs}
  }
}

Issue 1 - VM Resize from CLI

ErrorCode: LinkedAuthorizationFailed
ErrorMessage: The client 'USER-NAME' with object id 'USER-ID' has permission to perform action 'Microsoft.Compute/virtualMachines/write' on scope '/subscriptions/subscriptions/00000000-0000-0000-0000-000000000002/resourceGroups/resource_group-name/providers/Microsoft.Compute/virtualMachines/VM-name'; however, it does not have permission to perform action(s) 'Microsoft.ManagedIdentity/userAssignedIdentities/assign/action' on the linked scope(s) '/subscriptions/00000000-0000-0000-0000-000000000001/resourceGroups/resource_group-name/providers/Microsoft.ManagedIdentity/userAssignedIdentities/UAMI-name' (respectively) or the linked scope(s) are invalid.
ErrorTarget:
StatusCode: 403
ReasonPhrase: Forbidden
OperationID : 

Issue 2 - Moving VM to an availability zone from the portal

Error Details: 

Error Id UserManagedIdentityOutsideSubscriptionScope
Error message 
The subscription of user-assigned managed identity - /subscriptions/00000000-0000-0000-0000-000000000001/resourceGroups/resource_group-name/providers/Microsoft.ManagedIdentity/userAssignedIdentities/UAMI-name and VM - /subscriptions/subscriptions/00000000-0000-0000-0000-000000000004/resourceGroups/resource_group-name/providers/Microsoft.Compute/virtualMachines/VM-name are different.
Possible Causes: The user-assigned managed identity - /subscriptions/00000000-0000-0000-0000-000000000001/resourceGroups/resource_group-name/providers/Microsoft.ManagedIdentity/userAssignedIdentities/UAMI-name should have the same subscription as the source VM - /subscriptions/subscriptions/00000000-0000-0000-0000-000000000004/resourceGroups/resource_group-name/providers/Microsoft.Compute/virtualMachines/VM-name.
Recommended Action: Remove the user-assigned managed identity - /subscriptions/00000000-0000-0000-0000-000000000001/resourceGroups/resource_group-name/providers/Microsoft.ManagedIdentity/userAssignedIdentities/UAMI-name from the resource settings and retry the operation

Additional context

We are a Platform Team and provisioned more than 100 Landing Zones using caf-enterprise-scale/azurerm. After deploying the AMA Policy, some Landing Zone users have reported that they are not able to perform the above actions on VMs because the UAMI is created in a different subscription on which they don't have access. We don't want to allow the access to the central subscription where the UAMI is created as it needs to be managed only by us.
For Issue 1 - The action can be performed from the Azure Portal but not from Azure CLI or PowerShell.
For Issue 2 - A workaround was to apply a temporary policy exemption but it is not sustainable on the long run.

[matt-FFFFFF]

Apologies. I thought this was getting logged upstream. Will move the issue and ping the relevant people

[matt-FFFFFF]

@arjenhuitema @Springstone

This is the issue that I flagged internally

[arjenhuitema]

Hi @HeiGeorge, Thank you for reporting the issue. I have replicated the issues and reviewed possible workarounds and next steps.

Issue 1: Resizing

The UAMI configuration should not need to change when resizing. I am consulting internally to understand how PowerShell and CLI handle this scenario. As this requires review from other teams, it may take some time to provide a response.

Workarounds

1. Option 1: Assign the Landing zone owner the “Managed Identity Operator” role on the UAMI used for AMA. This enables the Landing zone owner to read and assign User Assigned Identities. The assignment duration can be set to limit the period for which these permissions are granted.

2. Option 2: Utilize Azure Policy exemptions and remove the UAMI from the VM. After the exemption period ends and the resizing is complete, verify and remediate the AMA policies to ensure the UAMI is reconfigured. Note that AMA relies on the UAMI, and logging may be affected.

Issue 2: Moving a VM to an availability zone

In this scenario, a new virtual machine (VM) is created with the specified zone configuration, and the source VM is deallocated. As a result of creating a new VM, the User Assigned Managed Identity (UAMI) configuration for that VM is also updated, which necessitates having access to manage the UAMI.

Even if you possess the permissions to manage the UAMI, the tools will display an error message indicating that the UAMI is outside the permissible scope (UserManagedIdentityOutsideSubscriptionScope).

You can opt not to retain the UAMI, but only when the user has the necessary permissions on the UAMI. Removing a UAMI from a VM configuration shouldn't require access to the UAMI. I will consult internal teams to clarify why access might be needed when not retaining the UAMI.

Workarounds

1. Option 1: Assign the Landing zone owner the “Managed Identity Operator” role on the UAMI used for AMA. This will permit the Landing zone owner to read and assign User Assigned Identities. You can set the assignment duration to limit the period for which these permissions are granted. You can then opt not to retain the UAMI when moving the VM to an availability zone. The new VM created in the selected AV will automatically be reconfigured with the UAMI by the Policy.

2. Option 2: Use Azure Policy exemptions and remove the UAMI from the VM. The new VM created in the selected AV will be reconfigured with the UAMI by the Policy automatically.

I'll update when I have more information.

[HeiGeorge]

Thanks @arjenhuitema ! Much appreciated!