Jan 12, 2018

4 AWS CLI features to master in 2018

Most AWS admins and developers are familiar with the AWS SDKs and CLI found on the AWS Github account. Fewer people are familiar with the many experiments AWS has open sourced on Amazon Web Services – Labs. These tools don’t get the same public relations love that the rest of AWS services receive and documentation of advanced features falls on the responsibility of the feature creator and project maintainer.

Because of their lack of PR love, many features get released every month that may not be well documented, and may only get a few lines in the Changelog. So, here are four that may help improve the way you utilize the AWS cli tool.

1. Command Aliases

Since version 1.11.24 (Dec. 1, 2016), the AWS CLI has supported the ability to create command aliases. If you’re familiar with Git aliases, you understand the premise: Commands that are short for commands or for sections of commands, and are executed . For Git, an example alias may be expressed as:

# in the ~/.gitconfig file
 
[alias]
exec = “!exec “

and invoked as git exec $cmd. This example runs the $cmd in the root of the repository using exec.

AWS CLI aliases work the same way. The AWS cli looks into the ~/.aws/cli/alias file for any matching aliases and then will run the command.

# in the ~/.aws/cli/alias file<code>
 
[toplevel]
whoami = sts get-caller-identity
</code>

Is invoked as aws whoami, which runs aws sts get-caller-identity and returns the result.

Neat. Commonly used aws commands can be stored in the alias file and utilized in future scripting. Even if you don’t use the commands, or can’t remember what the alias was called, it provides a single location for common commands or portions of commands to be stored for future reference. No more history | grep 'aws' to try and find the exact text you needed to run a command.

Here’s an example: If you frequently are trying to max out your bandwidth to move large directories of files, you can define an alias to do that:

[toplevel]
# from https://aws.amazon.com/blogs/apn/getting-the-most-out-of-the-amazon-s3-cli/
s3-sync-high-throughput =
!f() {
find $1 -mindepth 1 -maxdepth 1 -type d -print0 | xargs -n1 -0 -P30 -I {} aws s3 cp --recursive --quiet {}/ $2/{}/
}; f

This alias takes a source directory and copies everything to a destination bucket by creating up to 30 concurrent upload processes to upload to S3 (aws s3-sync-high-throughput $src s3://$dstBucket).

Note that in this example we start the alias with ! to tell AWS that it needs to invoke an external function rather than expand the alias directly in the command. We immediately follow with a function definition and then call the definition afterwards.

One command I run frequently is to mark an instance as unhealthy to force an autoscaling group to replace it. However, there is an error in the documentation on AWS for the command that causes the command to fail every. single. time. A hidden en-dash snuck it’s way in looking like a hyphen.

aws autoscaling set-instance-health --instance-id i-123abc45d –-health-status Unhealthy

Spot the error? Probably not. So, to make sure I don’t run the command ever with the en-dash, I create an AWS alias with the command and the en-dash removed.

[toplevel]
# from https://aws.amazon.com/blogs/apn/getting-the-most-out-of-the-amazon-s3-cli/
mark-instance-unhealthy = autoscaling set-instance-health --health-status Unhealthy --instance-id

Now, running aws mark-instance-unhealthy i-123abc45d works as expected, marking instance i-123abc45d as unhealthy and replacing it within my autoscaling group.

2. AWS CLI History aws history

One of my favorite features to be released in 2017 was the include of aws history. Gone are the times of trying to figure out why an aws command in a bootstrap script failed by cranking up the log level and hoping you can capture the response. With aws history, you can view a list of commands run and even inspect the request and command arguments, as well as the response generated.

You can turn on aws history by installing aws-cli >1.13.0, and adding a line to your configuration file or by running aws configure set cli_history enabled.

aws history has two subcommands: list and show. You use aws history list to view the commands you’ve run and aws history show &lt;\code&gt; to show the results of a command.

When you run aws history list the result is piped to your $PAGER (usually less), where you see something like this:

91a71c0a-0001-4a4c-ae78-231fc200d1d5  2018-01-07 10:44:44 PM  configure set     null

From there you can grab the ID (in this case 91a71c0a-0001-4a4c-ae78-231fc200d1d5), and run aws history show 91a71c0a-0001-4a4c-ae78-231fc200d1d5 to view the command. And what you get will look something like this:

AWS CLI command entered
at time: 2018-01-07 22:44:44.413
with AWS CLI version: aws-cli/1.14.10 Python/3.6.4 Darwin/17.3.0 botocore/1.8.14
with arguments: ['configure', 'set', 'cli_history', 'enabled']
 
AWS CLI command exited
at time: 2018-01-07 22:44:44.418
with return code: None

You can see that we ran aws configure set cli_history enabled, what version of the cli, python, kernel, and boto that we were running.

Go check it out. Enable it, run a few commands, and inspect the responses. This is going to become one of your favorite utilities for figuring out why CLI commands are failing.

3. --query Flag

The query flag is a well-documented feature, but maybe not a well-mentioned feature. The query flag filters the results of a command using the tool jp. Most people are familiar with using jq to parse JSON output, so you can always --output json and pipe your output through jq to get a similar effect.

Now, there’s some benefit to just using jq over --query; in particular, it is a small, single purpose tool that can be reused with any JSON output, which makes it handy for interacting with RESTful APIs with other commands or curl. However, if you’re running a command on machine where you don’t want to install additional dependencies or including it in a script where you don’t have control over installing jq, then learning how to use query can make it possible to achieve a similar result without having to introduce additional dependencies.

--query takes a string containing markup formatted using jmespath. --query can be tacked on to any command to filter the output to just the required info. The query result matches the format specified in the AWS ClI (either the default, AWS_DEFAULT_OUTPUT, config file ~/.aws/config, or by using --output) So if you want to maintain a table-format for your results, You can --query the command view the results using --output table

Say for example, you want to view all instances in us-east-1d as a ASCII-formatted table. This would look like:

aws ec2 describe-instances --output table --query 'Reservations[].Instances[] | [?Placement.AvailabilityZone==`us-east-1d`] | [].{ID: InstanceId, IP: PrivateIpAddress}'

The result would be the standard AWS ASCII table format, showing only the instances that reside in us-east-1d (example shows result with fake ids/ips)

~$ aws ec2 describe-instances --output table --query 'Reservations[].Instances[] | [?Placement.AvailabilityZone==`us-east-1d`] | [].{ID: InstanceId, IP: PrivateIpAddress}'
-------------------------------------
|         DescribeInstances         |
+----------------------+------------+
|          ID          |    IP      |
+----------------------+------------+
|  i-08e5920ab829753a8 |  10.0.0.1  |
|  i-0160b26e2193ce427 |  10.0.0.2  |
|  i-037da2cc88552f7ca |  10.0.0.3  |
|  i-0424d17d59091ecd2 |  10.0.0.4  |
|  i-0dc61acdb90f8469b |  10.0.0.5  |
|  i-0f64c12fbeae29bfe |  10.0.0.6  |
|  i-001499a3c5cd9fd7b |  10.0.0.7  |
|  i-0142719653944908f |  10.0.0.8  |
+----------------------+------------+

The table output is to make it human readable. You can also output to text or json to read with other programs.

4. S3 API Settings

In the first feature above, I showed an example of an alias command that will multithread s3 uploads across 30 processes (s3-sync-high-throughput). However, this isn’t needed, as the AWS CLI provides a way to configure the s3 and s3api commands to work on 30 or more processes, adding finer grain of control over the s3 command.

Each CLI profile can contain a top-level s3 key that can configure features of the s3 command like the maximum number of threads for the s3 command to use while uploading.

# From http://docs.aws.amazon.com/cli/latest/topic/s3-config.html
# in the ~/.aws/config file
[profile development]
aws_access_key_id=foo
aws_secret_access_key=bar
s3 =
  max_concurrent_requests = 20
  max_queue_size = 10000
  multipart_threshold = 64MB
  multipart_chunksize = 16MB
  use_accelerate_endpoint = true
  addressing_style = path
  max_bandwidth = 1MB/s

The max_concurrent_requests can be adjusted up or down (default 10) to set how many files should be uploaded to S3 by a single command. Included in this example is the full list of available configuration, and you can find out more about each key and what they do in the S3 CLI documentation.

About the Author

Object Partners profile.
Leave a Reply

Your email address will not be published.

Related Blog Posts
Natively Compiled Java on Google App Engine
Google App Engine is a platform-as-a-service product that is marketed as a way to get your applications into the cloud without necessarily knowing all of the infrastructure bits and pieces to do so. Google App […]
Building Better Data Visualization Experiences: Part 2 of 2
If you don't have a Ph.D. in data science, the raw data might be difficult to comprehend. This is where data visualization comes in.
Unleashing Feature Flags onto Kafka Consumers
Feature flags are a tool to strategically enable or disable functionality at runtime. They are often used to drive different user experiences but can also be useful in real-time data systems. In this post, we’ll […]
A security model for developers
Software security is more important than ever, but developing secure applications is more confusing than ever. TLS, mTLS, RBAC, SAML, OAUTH, OWASP, GDPR, SASL, RSA, JWT, cookie, attack vector, DDoS, firewall, VPN, security groups, exploit, […]