Sep 24, 2019

Some Command Line Tools for AWS

I’ve been working a lot in AWS lately and while I like the tech stack, going through the console to do stuff is really annoying to a command-line junkie such as myself. I could cook up my own scripts via the aws-cli or using boto3 and Python. But luckily others have already done that for me. Here are a couple of tools I use everyday.

awslogs

If I want to check the logs for my lambda function, I used to log in to the console, go to CloudWatch, find the my lambda (which is made difficult because the search bar doesn’t do substrings), click on my lambda group, and then sort by time, because it doesn’t necessarily sort by time, and then click on the link for the logs. That is exhausting.

Now I use awslogs, which does a lot of the work for me. awslogs groups will list all the existing CloudWatch groups and then awslogs get <group name> will display the latest entries. Of course you pipe it to grep or ag and search for messages. Or you can use --search in the awslogs command. You can also set a start time and/or an end time for the log messages.

This command doesn’t seem to honor the standard AWS_PROFILE environment variable, but you can use --profile <name> at the end of the awslogs command (it doesn’t seem to work anywhere else in the command, unfortunately).

s3cmd

I think of s3cmd as a little Swiss Army knife for all things S3. It does syncing, copying, tree-traversals. It even knows when to use multi-part uploads. Most of my work is in S3 lately, and I use this all the time.

My biggest problem with s3cmd is it doesn’t use AWS’s standard configs but it’s own config file. That makes switching between profiles (something I do a lot) tricky. I ended up making a wrapper script that reads ~/.aws/credentials and takes the given profile name and executes the S3 call. You can see the script here. Usage is:

s3ctl.py <profile_name> <s3cmd command>

so if you are putting fancy-file into s3://my-fancy-bucket/some-folder using your development AWS profile, it’s:

s3ctl.py development put fancy-file  s3://my-fancy-bucket/some-folder 

About the Author

Mike Hostetler profile.

Mike Hostetler

Principal Technologist

Mike has almost 20 years of experience in technology. He started in networking and Unix administration, and grew into technical support and QA testing. But he has always done some development on the side and decided a few years ago to pursue it full-time. His history of working with users gives Mike a unique perspective on writing software.

Leave a Reply

Your email address will not be published. Required fields are marked *

Related Blog Posts
Setup Spring DataSource from values stored in AWS Secret Manager
I was tasked migrating an application to AWS recently. The company wanted the application to store the database credentials in AWS’s Secret Manager. The issue that I had is that if I setup a configuration […]
Up to Spec: JavaScript Numeric Separators
Let's take a look at the proposal to add Numeric Separators to the JavaScript specification.
Using Conftest to Validate Configuration Files
Conftest is a utility within the Open Policy Agent ecosystem that helps simplify writing validation tests against configuration files. In a previous blog post, I wrote about using the Open Policy Agent utility directly to […]
SwiftGen with Image & Color Asset Catalogs
You might remember back in 2015 when iOS 9 was introduced, and we were finally given a way to manage all of our assets in one place with Asset Catalogs. A few years later, support […]