Master DevOps Tools & Best Practices - DevOps Knowledge Hub Master DevOps Tools & Best Practices - DevOps Knowledge Hub

Troubleshooting 'Access Denied' and Authentication Issues in AWS CLI

Systematically troubleshoot 'Access Denied' and authentication errors when using the AWS CLI. This guide covers essential diagnostic steps, starting with validating credentials using `aws sts get-caller-identity`, examining the complex IAM policy evaluation hierarchy, and resolving issues stemming from temporary credentials or resource-based policies (like S3 bucket policies). Learn to use key commands and the IAM Policy Simulator to swiftly identify and fix authorization failures in your AWS environment.

Troubleshooting 'Access Denied' and Authentication Issues in AWS CLI

Access Denied in the AWS CLI is frustrating because the message often names the failed API call but not the policy line that caused it. These issues are almost always rooted in misconfigured Identity and Access Management (IAM) policies, expired temporary credentials, or incorrect setup of the CLI environment.

The useful habit is to split the problem into two questions: who is the CLI calling as, and what policy boundary is stopping that identity from doing the action?

1. Initial Diagnosis: Identifying the Caller and Credentials

Before diving into complex policy analysis, the first step is to definitively confirm which IAM identity the AWS CLI is using and whether those credentials are still valid.

Check Current Identity: sts get-caller-identity

This is the most critical diagnostic command. It tells you exactly which User ARN, Role ARN, or assumed role session is executing the subsequent AWS commands.

aws sts get-caller-identity

Expected Output:

{
    "UserId": "AIDAIAMUSERID",
    "Account": "123456789012",
    "Arn": "arn:aws:iam::123456789012:user/DevUser"
}

If the ARN returned does not match the user or role you expect, you are operating under the wrong AWS profile or environment variables.

Verify Profile Configuration and Region

Ensure that the correct AWS profile is being used, either specified via the --profile flag or set via the AWS_PROFILE environment variable.

# Check the configured settings for the default profile
aws configure list

# Or check a specific profile
aws configure list --profile prod-admin

If the output shows no region or an incorrect region, operations on global resources or region-specific services (like S3 buckets or EC2 instances) may fail, sometimes resulting in an Access Denied if the resource isn't found where the CLI is looking.

Tip: If the sts get-caller-identity command itself fails with an authentication error, it indicates a fundamental problem with the access keys (they might be deleted, revoked, or incorrect).

Dealing with Temporary Credentials

If you are using an IAM Role (via STS AssumeRole), the CLI operates on temporary credentials that include a SessionToken. These credentials expire after the duration configured for the session, often one hour in default setups. While the AWS CLI usually refreshes them automatically when sourcing from the standard credential provider chain, manual configurations can fail.

Symptom: Commands work initially, but fail after a set period (e.g., 60 minutes) with an authentication error.

Solution: If using a custom script or a manually loaded environment, ensure your method for assuming the role includes a mechanism to refresh the credentials when they expire.

2. Deep Dive into IAM Policies and Authorization

Once you confirm the identity being used, the next step is determining why that identity lacks the necessary permissions. AWS authorization failures are complex because multiple policies are evaluated simultaneously.

The IAM Policy Evaluation Hierarchy

Authorization errors typically occur because of one of the following policy components:

  1. Explicit Deny: An explicit Deny statement in any applicable policy (Identity, Resource, or Boundary) always overrides an Allow statement.
  2. Missing Allow: No policy applicable to the identity or resource grants the specific action.

A. Identity-Based Policies (Users and Roles)

Check the IAM policies attached directly to the user or the role being assumed. Look for:

  • Missing Actions: Does the policy specifically list the necessary API action (e.g., s3:ListBucket, ec2:DescribeInstances)?
  • Resource Constraints: Does the policy restrict the action to specific resources using the Resource element? A common error is setting Resource: "*" when the action requires a specific ARN, or vice versa.
  • Conditions: Are there conditions (e.g., source IP address, time of day, MFA required) that are not being met?

B. Resource-Based Policies (S3, SQS, KMS)

For services like S3 or KMS, the resource itself has a policy that dictates who can access it. For cross-account access, and for many resource-specific designs, the resource policy must also permit the principal. In the same account, the exact interaction depends on the principal type and service, so do not assume an identity policy alone explains every result.

Example: Attempting to access an S3 bucket (Resource Policy) that has an explicit Deny for all users outside a specific VPC Endpoint will result in Access Denied, regardless of the user's IAM policy.

C. Permission Boundaries and SCPs

If your organization uses AWS Organizations, Service Control Policies (SCPs) define the maximum permissions allowed within an account. Similarly, Permission Boundaries limit the maximum permissions an IAM entity can ever possess.

If the SCP or Boundary denies the required action, the CLI operation will fail, even if the Identity Policy grants the permission.

Practical Tool: IAM Policy Simulator

When troubleshooting complex failures, the IAM Policy Simulator in the AWS Management Console is invaluable. You can test a specific action (e.g., s3:GetObject) against a specific resource (e.g., arn:aws:s3:::my-bucket/key) and specify the IAM entity, helping you pinpoint exactly which policy statement is causing the denial.

3. Common 'Access Denied' Scenarios and Solutions

Scenario 1: S3 Access Denied (Resource vs. Identity)

S3 Access Denied is notorious because it can stem from either the User Policy or the Bucket Policy.

Cause Diagnosis Solution
Missing Bucket Policy Allow sts get-caller-identity works, but aws s3 ls fails. Modify the Bucket Policy to explicitly allow the calling ARN to perform the necessary actions (s3:ListBucket, s3:GetObject).
Missing KMS Decrypt Permissions Accessing encrypted objects fails (even if S3 policy allows). Ensure the IAM entity has permissions to use the underlying KMS key (kms:Decrypt) that encrypted the object.
Requester Pays Attempts to download large files fail. If the bucket requires Requester Pays, the CLI command must include the --request-payer requester flag.

Scenario 2: Implicit Deny Due to Condition Failures

Many policies utilize conditions that enforce security best practices, such as requiring Multi-Factor Authentication (MFA).

If your policy includes a condition like:

"Condition": {
    "Bool": {
        "aws:MultiFactorAuthPresent": "true"
    }
}

And you attempt to run a command without authenticated MFA, the action will be implicitly denied.

Solution: For commands requiring MFA, you must first create an STS session authenticated with your MFA device:

# 1. Get temporary credentials using your MFA token
aws sts get-session-token --serial-number arn:aws:iam::123456:mfa/user --token-code 123456

# 2. Export the resulting AccessKeyId, SecretAccessKey, and SessionToken as environment variables for your CLI session.

Scenario 3: Missing or Incorrect Region

While not always an Access Denied error, trying to perform an action on a resource without specifying the correct region often leads to authorization or resource-not-found failures, especially when working with regional services like EC2, DynamoDB, or EKS.

Solution: Explicitly define the region in your command or ensure your profile is configured correctly.

aws ec2 describe-instances --region us-west-2

Service-specific checks that save time

IAM errors are easier to debug when you stop treating AWS as one permission system. Each service adds its own resource model and condition keys.

For S3, separate bucket-level actions from object-level actions. s3:ListBucket uses the bucket ARN, while s3:GetObject and s3:PutObject use object ARNs under the bucket. A policy that grants s3:GetObject on arn:aws:s3:::my-bucket is shaped wrong; object access needs arn:aws:s3:::my-bucket/*. The reverse mistake is just as common for listing.

For KMS, check the key policy as well as the IAM policy. A role can appear to have kms:Decrypt, but if the key policy does not allow that role path, the decrypt still fails. This shows up when downloading encrypted S3 objects, pulling encrypted EBS snapshots, or reading secrets that use a customer managed key.

For ECR, Docker login and image pull require multiple services to line up. The CLI identity may need ecr:GetAuthorizationToken, and the repository policy may need to allow pull actions if the repository is shared across accounts. If the cluster node role is pulling the image, debugging with your personal admin profile proves very little; test as the node role or task execution role.

For STS assume-role workflows, look at both sides of the trust relationship. The caller needs permission to call sts:AssumeRole, and the target role trust policy must trust the caller. If an external ID or MFA condition is present in the trust policy, the assume-role command must satisfy it.

Environment precedence can fool experienced users

The AWS CLI does not only read ~/.aws/credentials. It walks a credential provider chain that can include command-line flags, environment variables, named profiles, SSO cache entries, web identity tokens, EC2 or ECS metadata, and role assumptions configured in the profile. That is why aws configure list is more useful than opening one file.

One common local failure looks like this: you run export AWS_PROFILE=dev, then later paste temporary production credentials into AWS_ACCESS_KEY_ID, AWS_SECRET_ACCESS_KEY, and AWS_SESSION_TOKEN. The environment keys can take over, so commands that look like they use dev are really using the exported session. In a terminal that has been open all day, run:

env | sort | grep '^AWS_'

If you are switching accounts often, use separate terminal tabs, a credential helper, or wrapper scripts that print the caller identity before destructive commands. The extra line in the output is cheaper than deleting from the wrong account.

4. Summary of Diagnostic Commands

Use this checklist of commands to quickly diagnose the authorization failure chain:

Goal Command Purpose
Verify Identity aws sts get-caller-identity Confirms the ARN executing the command.
Verify Configuration aws configure list Checks profile settings, region, and output format.
Test Policy Validity (Use IAM Policy Simulator) Checks if an IAM identity can perform a specific API action on a resource.
Identify Policy Denials aws cloudtrail lookup-events ... Use CloudTrail to see the exact event record where the policy evaluation failed.

A real debugging path

A good first pass looks like this. Run the failing command again with the profile and region written explicitly, even if you think your shell is already set:

AWS_PROFILE=prod-readonly aws s3 ls s3://example-audit-logs --region us-east-1
aws sts get-caller-identity --profile prod-readonly
aws configure list --profile prod-readonly

If the identity is wrong, stop there. Check AWS_ACCESS_KEY_ID, AWS_SECRET_ACCESS_KEY, AWS_SESSION_TOKEN, AWS_PROFILE, and AWS_REGION in the environment. Environment credentials can take precedence over a profile and make the CLI act as a role you forgot you exported earlier. In CI, print aws sts get-caller-identity near the failing step so the build log proves the role used at runtime.

If the identity is right, write down the exact API action. High-level commands can hide this. aws s3 cp may need s3:GetObject, s3:PutObject, s3:ListBucket, kms:Decrypt, or kms:GenerateDataKey depending on direction, encryption, and whether the CLI must inspect the bucket. When the error message includes AccessDenied but not the action, CloudTrail usually gives you the event name and resource.

For S3, check bucket policy, object ownership, block public access settings, VPC endpoint policy, and KMS key policy. For KMS-encrypted objects, an S3 allow is not enough; the caller also needs the relevant KMS permission and the key policy must allow the principal path. For organizations, check SCPs before editing identity policies. An SCP cannot grant access, but it can cap what any principal in the account may do.

Prevention is mostly boring hygiene: short-lived role credentials, clearly named profiles, least-privilege policies tested against the real ARN, and CloudTrail enabled where engineers can actually query it. The best fix is not a broader Action: "*"; it is finding the one missing action or the one deny condition that matches the request.