Sep 24, 2019

Some Command Line Tools for AWS

I’ve been working a lot in AWS lately and while I like the tech stack, going through the console to do stuff is really annoying to a command-line junkie such as myself. I could cook up my own scripts via the aws-cli or using boto3 and Python. But luckily others have already done that for me. Here are a couple of tools I use everyday.

awslogs

If I want to check the logs for my lambda function, I used to log in to the console, go to CloudWatch, find the my lambda (which is made difficult because the search bar doesn’t do substrings), click on my lambda group, and then sort by time, because it doesn’t necessarily sort by time, and then click on the link for the logs. That is exhausting.

Now I use awslogs, which does a lot of the work for me. awslogs groups will list all the existing CloudWatch groups and then awslogs get <group name> will display the latest entries. Of course you pipe it to grep or ag and search for messages. Or you can use --search in the awslogs command. You can also set a start time and/or an end time for the log messages.

This command doesn’t seem to honor the standard AWS_PROFILE environment variable, but you can use --profile <name> at the end of the awslogs command (it doesn’t seem to work anywhere else in the command, unfortunately).

s3cmd

I think of s3cmd as a little Swiss Army knife for all things S3. It does syncing, copying, tree-traversals. It even knows when to use multi-part uploads. Most of my work is in S3 lately, and I use this all the time.

My biggest problem with s3cmd is it doesn’t use AWS’s standard configs but it’s own config file. That makes switching between profiles (something I do a lot) tricky. I ended up making a wrapper script that reads ~/.aws/credentials and takes the given profile name and executes the S3 call. You can see the script here. Usage is:

s3ctl.py <profile_name> <s3cmd command>

so if you are putting fancy-file into s3://my-fancy-bucket/some-folder using your development AWS profile, it’s:

s3ctl.py development put fancy-file  s3://my-fancy-bucket/some-folder 

About the Author

Mike Hostetler profile.

Mike Hostetler

Principal Technologist

Mike has almost 20 years of experience in technology. He started in networking and Unix administration, and grew into technical support and QA testing. But he has always done some development on the side and decided a few years ago to pursue it full-time. His history of working with users gives Mike a unique perspective on writing software.

Leave a Reply

Your email address will not be published. Required fields are marked *

Related Blog Posts
Interpreting Spatial Data in the Age of COVID-19
As 2020 has come to an end, many are eager to leave the mess of COVID-19 behind with the new year and gain a fresh start. Unfortunately, new cases are still soaring across the United […]
Building a Better Mousetrap
Recently, my daughter (age 6) was into building “mousetraps” out of shoe boxes. These were more or less comfortable cardboard mouse houses complete with beds, rooms, everything a mouse could want or need and not […]
ARM Wrestling Its Way Into Mainstream Software Development
Nearly all smart phones have been running ARM-based processors for years. They provide superior power for the amount of power consumed, and thus extend battery life. With Apple’s recent release of the Apple Silicon M1 […]
Video capture using WebRTC
Capturing the feed of a users camera can be a daunting undertaking involving one or multiple third party packages. Often we opt for prompting the user to upload the video or image we need, causing […]