4 AWS CLI features to master in 2018
Most AWS admins and developers are familiar with the AWS SDKs and CLI found on the AWS Github account. Fewer people are familiar with the many experiments AWS has open sourced on Amazon Web Services – Labs. These tools don’t get the same public relations love that the rest of AWS services receive and documentation of advanced features falls on the responsibility of the feature creator and project maintainer.
Because of their lack of PR love, many features get released every month that may not be well documented, and may only get a few lines in the Changelog. So, here are four that may help improve the way you utilize the AWS cli tool.
1. Command Aliases
Since version 1.11.24 (Dec. 1, 2016), the AWS CLI has supported the ability to create command aliases. If you’re familiar with Git aliases, you understand the premise: Commands that are short for commands or for sections of commands, and are executed . For Git, an example alias may be expressed as:
# in the ~/.gitconfig file [alias] exec = “!exec “
and invoked as git exec $cmd
. This example runs the $cmd
in the root of the repository using exec
.
AWS CLI aliases work the same way. The AWS cli looks into the ~/.aws/cli/alias
file for any matching aliases and then will run the command.
# in the ~/.aws/cli/alias file<code> [toplevel] whoami = sts get-caller-identity </code>
Is invoked as aws whoami
, which runs aws sts get-caller-identity
and returns the result.
Neat. Commonly used aws commands can be stored in the alias file and utilized in future scripting. Even if you don’t use the commands, or can’t remember what the alias was called, it provides a single location for common commands or portions of commands to be stored for future reference. No more history | grep 'aws'
to try and find the exact text you needed to run a command.
Here’s an example: If you frequently are trying to max out your bandwidth to move large directories of files, you can define an alias to do that:
[toplevel] # from https://aws.amazon.com/blogs/apn/getting-the-most-out-of-the-amazon-s3-cli/ s3-sync-high-throughput = !f() { find $1 -mindepth 1 -maxdepth 1 -type d -print0 | xargs -n1 -0 -P30 -I {} aws s3 cp --recursive --quiet {}/ $2/{}/ }; f
This alias takes a source directory and copies everything to a destination bucket by creating up to 30 concurrent upload processes to upload to S3 (aws s3-sync-high-throughput $src s3://$dstBucket
).
Note that in this example we start the alias with !
to tell AWS that it needs to invoke an external function rather than expand the alias directly in the command. We immediately follow with a function definition and then call the definition afterwards.
One command I run frequently is to mark an instance as unhealthy to force an autoscaling group to replace it. However, there is an error in the documentation on AWS for the command that causes the command to fail every. single. time. A hidden en-dash snuck it’s way in looking like a hyphen.
aws autoscaling set-instance-health --instance-id i-123abc45d –-health-status Unhealthy
Spot the error? Probably not. So, to make sure I don’t run the command ever with the en-dash, I create an AWS alias with the command and the en-dash removed.
[toplevel] # from https://aws.amazon.com/blogs/apn/getting-the-most-out-of-the-amazon-s3-cli/ mark-instance-unhealthy = autoscaling set-instance-health --health-status Unhealthy --instance-id
Now, running aws mark-instance-unhealthy i-123abc45d
works as expected, marking instance i-123abc45d
as unhealthy and replacing it within my autoscaling group.
2. AWS CLI History aws history
One of my favorite features to be released in 2017 was the include of aws history
. Gone are the times of trying to figure out why an aws
command in a bootstrap script failed by cranking up the log level and hoping you can capture the response. With aws history
, you can view a list of commands run and even inspect the request and command arguments, as well as the response generated.
You can turn on aws history
by installing aws-cli >1.13.0, and adding a line to your configuration file or by running aws configure set cli_history enabled
.
aws history
has two subcommands: list
and show
. You use aws history list
to view the commands you’ve run and aws history show <\code> to show the results of a command.
When you run aws history list
the result is piped to your $PAGER
(usually less
), where you see something like this:
91a71c0a-0001-4a4c-ae78-231fc200d1d5 2018-01-07 10:44:44 PM configure set null
From there you can grab the ID (in this case 91a71c0a-0001-4a4c-ae78-231fc200d1d5
), and run aws history show 91a71c0a-0001-4a4c-ae78-231fc200d1d5
to view the command. And what you get will look something like this:
AWS CLI command entered at time: 2018-01-07 22:44:44.413 with AWS CLI version: aws-cli/1.14.10 Python/3.6.4 Darwin/17.3.0 botocore/1.8.14 with arguments: ['configure', 'set', 'cli_history', 'enabled'] AWS CLI command exited at time: 2018-01-07 22:44:44.418 with return code: None
You can see that we ran aws configure set cli_history enabled
, what version of the cli, python, kernel, and boto that we were running.
Go check it out. Enable it, run a few commands, and inspect the responses. This is going to become one of your favorite utilities for figuring out why CLI commands are failing.
3. --query
Flag
The query flag is a well-documented feature, but maybe not a well-mentioned feature. The query flag filters the results of a command using the tool jp
. Most people are familiar with using jq
to parse JSON output, so you can always --output json
and pipe your output through jq
to get a similar effect.
Now, there’s some benefit to just using jq
over --query
; in particular, it is a small, single purpose tool that can be reused with any JSON output, which makes it handy for interacting with RESTful APIs with other commands or curl
. However, if you’re running a command on machine where you don’t want to install additional dependencies or including it in a script where you don’t have control over installing jq
, then learning how to use query can make it possible to achieve a similar result without having to introduce additional dependencies.
--query
takes a string containing markup formatted using jmespath. --query
can be tacked on to any command to filter the output to just the required info. The query result matches the format specified in the AWS ClI (either the default, AWS_DEFAULT_OUTPUT
, config file ~/.aws/config
, or by using --output
) So if you want to maintain a table-format for your results, You can --query
the command view the results using --output table
Say for example, you want to view all instances in us-east-1d
as a ASCII-formatted table. This would look like:
aws ec2 describe-instances --output table --query 'Reservations[].Instances[] | [?Placement.AvailabilityZone==`us-east-1d`] | [].{ID: InstanceId, IP: PrivateIpAddress}'
The result would be the standard AWS ASCII table format, showing only the instances that reside in us-east-1d (example shows result with fake ids/ips)
~$ aws ec2 describe-instances --output table --query 'Reservations[].Instances[] | [?Placement.AvailabilityZone==`us-east-1d`] | [].{ID: InstanceId, IP: PrivateIpAddress}' ------------------------------------- | DescribeInstances | +----------------------+------------+ | ID | IP | +----------------------+------------+ | i-08e5920ab829753a8 | 10.0.0.1 | | i-0160b26e2193ce427 | 10.0.0.2 | | i-037da2cc88552f7ca | 10.0.0.3 | | i-0424d17d59091ecd2 | 10.0.0.4 | | i-0dc61acdb90f8469b | 10.0.0.5 | | i-0f64c12fbeae29bfe | 10.0.0.6 | | i-001499a3c5cd9fd7b | 10.0.0.7 | | i-0142719653944908f | 10.0.0.8 | +----------------------+------------+
The table output is to make it human readable. You can also output to text or json to read with other programs.
4. S3 API Settings
In the first feature above, I showed an example of an alias command that will multithread s3 uploads across 30 processes (s3-sync-high-throughput
). However, this isn’t needed, as the AWS CLI provides a way to configure the s3
and s3api
commands to work on 30 or more processes, adding finer grain of control over the s3 command.
Each CLI profile can contain a top-level s3
key that can configure features of the s3 command like the maximum number of threads for the s3 command to use while uploading.
# From http://docs.aws.amazon.com/cli/latest/topic/s3-config.html # in the ~/.aws/config file [profile development] aws_access_key_id=foo aws_secret_access_key=bar s3 = max_concurrent_requests = 20 max_queue_size = 10000 multipart_threshold = 64MB multipart_chunksize = 16MB use_accelerate_endpoint = true addressing_style = path max_bandwidth = 1MB/s
The max_concurrent_requests
can be adjusted up or down (default 10) to set how many files should be uploaded to S3 by a single command. Included in this example is the full list of available configuration, and you can find out more about each key and what they do in the S3 CLI documentation.