Problem

While creating a new workspace using Terraform, you receive the following error.

Error: cannot create mws workspaces: Response from server (400 Bad Request) MALFORMED_REQUEST: Failed credentials validation checks: Delete Tags: invalid character 'M' looking for beginning of value

Cause

There is a condition in the Service Control Policy (SCP) on the customer's AWS side that is blocking the DeleteTags action. This action is required for Databricks to operate and manage clusters in an effective manner. 

When a cluster releases an instance, it returns to the pool and is free for another cluster to use. The same instances will be re-used, and when the instance is being re-used by another cluster, existing tags must be deleted to update the new cluster details on the tag.

Solution

1. Check your organization-level service control policies (SCPs) that deny the AssumeRole action or deny EC2/VPC access.
2. Verify if there are any errors from the AWS side for the IAM role events.
3. If you find that the SCP is blocking the DeleteTags action, you will need to allow this action for the assumed role at the Service Control Policy.
4. After allowing the DeleteTags permission, try creating the workspace again.

The policy should look like this. 

{
    "Version": "2012-10-17",
    "Statement": [
      {
        "Sid": "Stmt1403287045000",
        "Effect": "Allow",
        "Action": [
          ....
          ....
          "ec2:DeleteFleets",
          "ec2:DeleteInternetGateway",
          "ec2:DeleteLaunchTemplate",
          "ec2:DeleteLaunchTemplateVersions",
          "ec2:DeleteNatGateway",
          "ec2:DeleteRoute",
          "ec2:DeleteRouteTable",
          "ec2:DeleteSecurityGroup",
          "ec2:DeleteSubnet",
          "ec2:DeleteTags",
          "ec2:DeleteVolume",
          "ec2:DeleteVpc",
          ....
          ....
        ],
        "Resource": [
          "*"
        ]
      },
      {
        "Effect": "Allow",
        "Action": [
          "iam:CreateServiceLinkedRole",
          "iam:PutRolePolicy"
        ],
        "Resource": "arn:aws:iam::*:role/aws-service-role/spot.amazonaws.com/AWSServiceRoleForEC2Spot",
        "Condition": {
          "StringLike": {
            "iam:AWSServiceName": "spot.amazonaws.com"
          }
        }
      }
    ]
  }

For the full policy please refer to the Create an IAM role for workspace deployment documentation.
 

To avoid this problem generally, ensure the settings in steps 1 and 3 persist. 

For more information, please review the Permissions in cross-account IAM roles documentation and the AWS Service control policies (SCPs) documentation.