DevOps Knowledge Hub
DevOps Knowledge Hub

How to Configure and Manage Multiple AWS CLI Profiles Efficiently

Learn to efficiently manage multiple AWS accounts and environments using AWS CLI named profiles. This guide provides step-by-step instructions on configuring, switching between, and securing different sets of AWS credentials and settings. Optimize your cloud workflow by mastering profile management for enhanced productivity and security.

34 views

Understanding AWS CLI Profiles

The AWS Command Line Interface (AWS CLI) is a powerful tool for interacting with Amazon Web Services. For users managing multiple AWS accounts or different roles within an account, configuring and switching between profiles is crucial for efficient workflow management. Named profiles allow you to store different sets of credentials, default regions, and output formats, enabling seamless transitions between various AWS environments without manually specifying credentials each time.

This guide will walk you through the process of setting up, managing, and utilizing multiple AWS CLI profiles. By the end, you'll be able to optimize your command-line interactions across different AWS accounts, enhancing security and productivity.

Setting Up Your AWS CLI Configuration File

The AWS CLI stores configuration information in a file named config and credentials in a file named credentials. By default, these files are located in the .aws directory within your user's home directory (~/.aws/ on Linux/macOS, %USERPROFILE%\.aws\ on Windows).

Default Profile

When you first configure the AWS CLI using aws configure, it creates a default profile named [default]. This profile is used if no other profile is specified.

# Example of a default profile in ~/.aws/credentials
[default]
aws_access_key_id = YOUR_DEFAULT_ACCESS_KEY
aws_secret_access_key = YOUR_DEFAULT_SECRET_KEY

# Example of a default profile in ~/.aws/config
[default]
region = us-east-1
output = json

Creating Named Profiles

To create a named profile, you simply add a new section to your credentials and config files, using a descriptive name for the profile. For example, to create a profile for a production account and another for a development account:

1. Editing ~/.aws/credentials:

Add a new section with a profile name (e.g., [prod], [dev]).

# ~/.aws/credentials

[default]
aws_access_key_id = YOUR_DEFAULT_ACCESS_KEY
aws_secret_access_key = YOUR_DEFAULT_SECRET_KEY

[prod]
aws_access_key_id = YOUR_PROD_ACCESS_KEY
aws_secret_access_key = YOUR_PROD_SECRET_KEY

[dev]
aws_access_key_id = YOUR_DEV_ACCESS_KEY
aws_secret_access_key = YOUR_DEV_SECRET_KEY

2. Editing ~/.aws/config:

Similarly, add corresponding sections to the config file. You can specify region and output format per profile.

# ~/.aws/config

[default]
region = us-east-1
output = json

[profile prod]
region = us-west-2
output = text

[profile dev]
region = eu-central-1
output = json

Important Notes:
* When defining profiles in ~/.aws/config, you must prefix the profile name with profile (e.g., [profile prod]). This is different from ~/.aws/credentials, where you use just the profile name (e.g., [prod]).
* If a setting is defined in both the [default] profile and a named profile, the named profile's setting takes precedence.

Switching Between Profiles

Once your named profiles are set up, you can use them by specifying the --profile option with your AWS CLI commands.

Example: To list S3 buckets in your production account:

aws s3 ls --profile prod

Example: To describe EC2 instances in your development account:

aws ec2 describe-instances --profile dev

If you omit the --profile option, the CLI will use the [default] profile.

Setting a Default Profile for the Environment

Constantly typing --profile can be tedious. You can set an environment variable to specify which profile the AWS CLI should use by default for the current session.

Using AWS_PROFILE environment variable:

On Linux/macOS:

export AWS_PROFILE=prod

# Now, commands will use the 'prod' profile by default
aws s3 ls
aws ec2 describe-instances

To unset it:

unset AWS_PROFILE

On Windows (Command Prompt):

set AWS_PROFILE=prod

# Now, commands will use the 'prod' profile by default
aws s3 ls

To unset it:

set AWS_PROFILE=

On Windows (PowerShell):

$env:AWS_PROFILE = "prod"

# Now, commands will use the 'prod' profile by default
aws s3 ls

To unset it:

Remove-Item Env:\AWS_PROFILE

Managing Multiple AWS Accounts with Profiles

This is a common use case for named profiles. Each profile can be configured with the IAM user's access keys for a different AWS account. This is particularly useful for:

  • Development vs. Production: Keeping your development and production environments separate for safety.
  • Different Teams/Projects: Isolating resources and permissions for distinct teams or projects.
  • Cross-Account Access: If you are an administrator or developer who needs to manage resources in multiple accounts.

Best Practice: Using IAM Roles

Instead of storing long-lived access keys for each account directly in your credentials file, it's a more secure and recommended practice to use IAM roles. This involves:

  1. Creating an IAM role in the target account that your IAM user (in your primary account) can assume.
  2. Configuring your local AWS CLI to assume this role when using a specific profile.

To configure role assumption in ~/.aws/config:

# ~/.aws/config

[profile dev-role]
role_arn = arn:aws:iam::ACCOUNT_ID:role/ROLE_NAME
source_profile = default  # Or another profile that has permissions to assume the role
region = us-east-1
output = json

When you use aws ec2 describe-instances --profile dev-role, the CLI will automatically attempt to assume the specified IAM role and use the temporary credentials obtained.

Advanced Configuration Options

Beyond credentials and region, profiles can specify other AWS CLI settings:

  • output: json, text, table
  • region: e.g., us-east-1
  • s3.max_concurrent_requests: Number of parallel requests for S3 operations.
  • s3.max_queue_size: Size of the queue for S3 multipart uploads.
  • cli_binary_url: Specify a custom URL for downloading CLI binaries (less common).

Example: Configuring S3 settings for a specific profile

# ~/.aws/config

[profile s3-optimized]
region = us-east-1
output = json
s3.max_concurrent_requests = 50
s3.max_queue_size = 10000

Tips and Best Practices

  • Use Descriptive Profile Names: Make your profile names clear and indicative of the account or environment they represent (e.g., prod-admin, dev-web, sandbox-research).
  • Secure Your Credentials: Never commit your ~/.aws/credentials file to version control. Use IAM roles for cross-account access whenever possible to avoid storing long-lived access keys.
  • Regularly Review Access Keys: If you must use access keys, rotate them periodically and disable old ones.
  • Leverage Environment Variables: Use AWS_PROFILE for temporary switching or for CI/CD pipelines where you need to target specific accounts.
  • Combine Profile and Environment Variables: If an AWS_PROFILE environment variable is set, it will override any profile specified with --profile.

Conclusion

Mastering AWS CLI profiles is a fundamental skill for anyone working extensively with Amazon Web Services. By correctly configuring and utilizing named profiles, you can significantly enhance your productivity, maintain better security practices, and manage multiple AWS environments with ease. Whether you're switching between dev, staging, and production accounts or managing resources for different clients, profiles provide the flexibility and control needed for efficient cloud operations.