Azure ARM deployment - deleting resource group can't delete role assignments cleanly

Problem statement

Once an ARM template is deployed into a resource group, one way to completely delete those deployed resources is to delete that resource group. But it seems roles assignments can't be cleaned up entirely.

In our case, we have an ARM template to deploy Azure AKS cluster in a specified resource group, and two role assignments. One is to assign "Network Contributor" on VNET to the managed Id of the AKS cluster, and the other is to assign "Contributor" role on an already existing Azure Container Registry (ACR) to the managed Id of the AKS cluster.

First we successfully deployed the ARM template. The AKS was setup, and those two role assignments were deployed. Then we deleted that resource group. All resources under that resource group, including the AKS cluster, VNET,  etc. were deleted successfully. The role assignment on VNET was cleaned up as well, but the role assignment on ACR was still there.

Role assignments before and after resource group deletion


Since role assignments are Azure subscription level resources, they won't be listed under a resource group on Azure portal. You can use below command to list all role assignments under an Azure subscription.
az role assignment --all

The below is the role assignment on the ACR before deletion.
  {
    "canDelegate": null,
    "condition": null,
    "conditionVersion": null,
    "description": null,
    "id": "/subscriptions/<subscription Id>/resourcegroups/<resource group name>/providers/Microsoft.ContainerRegistry/registries/<ACR name>/providers/Microsoft.Authorization/roleAssignments/4b6bbdc6-0051-4b09-ae6b-2287c57aa5b2",
    "name": "4b6bbdc6-0051-4b09-ae6b-2287c57aa5b2",
    "principalId": "08706c32-1e0d-4bf5-8537-cf87e5eaf706",
    "principalName": "6a04bf60-6240-4133-b568-e36a363f458f",
    "principalType": "ServicePrincipal",
    "resourceGroup": "<resource group name>",
    "roleDefinitionId": "/subscriptions/<subscription Id>/providers/Microsoft.Authorization/roleDefinitions/b24988ac-6180-42a0-ab88-20f7382dd24c",
    "roleDefinitionName": "Contributor",
    "scope": "/subscriptions/<subscription Id>/resourcegroups/<resource group name>/providers/Microsoft.ContainerRegistry/registries/<ACR name>",
    "type": "Microsoft.Authorization/roleAssignments"
  }

After the deletion of the resource group, we can see the role assignment on the ACR is still there, but the principleName field is cleared.
  {
    "canDelegate": null,
    "condition": null,
    "conditionVersion": null,
    "description": null,
    "id": "/subscriptions/<subscription Id>/resourcegroups/<resource group name>/providers/Microsoft.ContainerRegistry/registries/<ACR name>/providers/Microsoft.Authorization/roleAssignments/4b6bbdc6-0051-4b09-ae6b-2287c57aa5b2",
    "name": "4b6bbdc6-0051-4b09-ae6b-2287c57aa5b2",
    "principalId": "08706c32-1e0d-4bf5-8537-cf87e5eaf706",
    "principalName": "",
    "principalType": "ServicePrincipal",
    "resourceGroup": "<resource group name>",
    "roleDefinitionId": "/subscriptions/<subscription Id>/providers/Microsoft.Authorization/roleDefinitions/b24988ac-6180-42a0-ab88-20f7382dd24c",
    "roleDefinitionName": "Contributor",
    "scope": "/subscriptions/<subscription Id>/resourcegroups/<resource group name>/providers/Microsoft.ContainerRegistry/registries/<ACR name>",
    "type": "Microsoft.Authorization/roleAssignments"
  }

If you are going to re-deploy the ARM template, you must ensure the name of that role assignment on ACR is unique, otherwise you will get an error like below,

{
RoleAssignmentUpdateNotPermitted",
"message": "Tenant ID, application ID, principal ID, and scope are not allowed to be updated."
}

Template to ensure every role assignment name unique

The snippet of the ARM template of creating that role assignment with an unique name is following. 
  "parameters": {
	...
    "guidValue-1": {
      "type": "string",
      "metadata": {
      "description": "The unique id used in the role assignment of the kubernetes service to the container registry service. It is recommended to use the default value."
      },
      "defaultValue": "[newGuid()]"
    },
	...
  }

  "resources": [
    ...  
	{
      "condition": "[or(equals(parameters('newOrExistingACR'), 'new'), equals(parameters('newOrExistingCluster'), 'new'))]",
      "name": "ConnectAKStoACR-RoleAssignment",
      "type": "Microsoft.Resources/deployments",
      "apiVersion": "2017-05-10",
      "resourceGroup": "[parameters('acrResourceGroup')]",      
      "dependsOn": [
        "[concat('Microsoft.ContainerService/managedClusters/', parameters('clusterName'))]"        
      ],
      "properties": {
        "mode": "Incremental",
        "template": {
          "$schema": "https://schema.management.azure.com/schemas/2015-01-01/deploymentTemplate.json#",
          "contentVersion": "1.0.0.0",
          "resources": [
            {
              "apiVersion": "2018-09-01-preview",
              "type": "Microsoft.ContainerRegistry/registries/providers/roleAssignments",
              "name": "[concat(parameters('acrName'), '/Microsoft.Authorization/', parameters('guidValue-1'))]",
              "properties": {
                "principalId": "[reference(parameters('clusterName'), '2020-03-01').identityProfile.kubeletidentity.objectId]",
                "principalType": "ServicePrincipal",
                "roleDefinitionId": "[concat('/subscriptions/', subscription().subscriptionId, '/providers/Microsoft.Authorization/roleDefinitions/', 'b24988ac-6180-42a0-ab88-20f7382dd24c')]",
                "scope": "[resourceId(parameters('acrSubscriptionId'),parameters('acrResourceGroup'),'Microsoft.ContainerRegistry/registries/', parameters('acrName'))]"
              }
            }
          ]
        }
      }
    },
	...
  }

References

Microsoft mentions this kind of 'undeleted role assignments' at Role assignments with identity not found.

Comments

Post a Comment

Popular posts from this blog

Deep-archive an aws S3 bucket with versioning enabled once

Image
It's not a surprise if you want to deep-achieve a S3 bucket, or some objects in it to decrease storage cost. But if versioning has once been enabled on the bucket, how to do the deep archiving needs more consideration.  Problem Statement If you have a S3 bucket, which versioning has never been enabled, then to move objects into Glacier Deep Archive can be achieved by iterating over every object, and  making a copy of an object using the  PUT Object - Copy  API.  You copy an object in the same bucket using the same key name and specify request headers, e.g. here you set  the   x-amz-storage-class   to the storage class,  ' DEEP_ARCHIVE',   that you want to use  [1]. Then the final effect is just like that the storage class of an object is being changed.  But if a bucket has once been versioning enabled, the above method is not straight forward. - If versioning is still enabled, even  you can copy an object with a specific version ID, Amazon S3 gives it a new version ID. The
Read more

Good practices of using Python logging

Problem Statement During the maintenance work of a legacy Python project, it's noticed some issues of logging code events. Outputs not in sequence The legacy code in some places still uses print() function, and traceback.print_exc() like below, try : #my logic goes here print ( "my output" ) except Exception : traceback . print_exc() By default in Python those functions are buffered. print() will send outputs to stdout, and traceback_print_exc() will send output to stderr. If those buffers are not flushed timely or disabled, the output may not in sequence which causes the outputs out of order.  Logging configuration not centralized and externalized It's noticed the logging configuration is added in every file using the Python logging. It means the logging configuration is not centralized and externalized for easy usage. Also when a message is logged, the 'logging' object is used directly instead of using a 'logger' object. The '
Read more

Auto-installing NVIDIA device plug-in on GPU nodes only

Problem Statement On a Kubernetes cluster before the GPUs in the nodes can be used, you need to deploy a DaemonSet for the NVIDIA device plugin. This DaemonSet runs a pod on each node to provide the required drivers for the GPUs. E.g. on Azure AKS cluster, here is the Azure official document regarding how to install NVIDIA device plug-in .  The YMAL manifest is in the nvidia-device-plugin-ds.yaml file. Once this file is applied, it does automatically run a pod to provide the required driver once a GPU node is added, no matter it's manually scaled or auto-scaled. But it may also run the pod on a node which has no GPUs at all, since that yaml manifest doesn't constraint the pods to run only on GPU nodes. This is a waste of resources on non-GPU nodes. Solution This is where " nodeSelector " and " affinity " can help. "nodeSelector" provides a very simple way to constrain pods to nodes with particular labels. The "affinity/anti-affinity" feat
Read more