Sep 24, 2019

Some Command Line Tools for AWS

I’ve been working a lot in AWS lately and while I like the tech stack, going through the console to do stuff is really annoying to a command-line junkie such as myself. I could cook up my own scripts via the aws-cli or using boto3 and Python. But luckily others have already done that for me. Here are a couple of tools I use everyday.

awslogs

If I want to check the logs for my lambda function, I used to log in to the console, go to CloudWatch, find the my lambda (which is made difficult because the search bar doesn’t do substrings), click on my lambda group, and then sort by time, because it doesn’t necessarily sort by time, and then click on the link for the logs. That is exhausting.

Now I use awslogs, which does a lot of the work for me. awslogs groups will list all the existing CloudWatch groups and then awslogs get <group name> will display the latest entries. Of course you pipe it to grep or ag and search for messages. Or you can use --search in the awslogs command. You can also set a start time and/or an end time for the log messages.

This command doesn’t seem to honor the standard AWS_PROFILE environment variable, but you can use --profile <name> at the end of the awslogs command (it doesn’t seem to work anywhere else in the command, unfortunately).

s3cmd

I think of s3cmd as a little Swiss Army knife for all things S3. It does syncing, copying, tree-traversals. It even knows when to use multi-part uploads. Most of my work is in S3 lately, and I use this all the time.

My biggest problem with s3cmd is it doesn’t use AWS’s standard configs but it’s own config file. That makes switching between profiles (something I do a lot) tricky. I ended up making a wrapper script that reads ~/.aws/credentials and takes the given profile name and executes the S3 call. You can see the script here. Usage is:

s3ctl.py <profile_name> <s3cmd command>

so if you are putting fancy-file into s3://my-fancy-bucket/some-folder using your development AWS profile, it’s:

s3ctl.py development put fancy-file  s3://my-fancy-bucket/some-folder 

About the Author

Object Partners profile.
Leave a Reply

Your email address will not be published.

Related Blog Posts
Natively Compiled Java on Google App Engine
Google App Engine is a platform-as-a-service product that is marketed as a way to get your applications into the cloud without necessarily knowing all of the infrastructure bits and pieces to do so. Google App […]
Building Better Data Visualization Experiences: Part 2 of 2
If you don't have a Ph.D. in data science, the raw data might be difficult to comprehend. This is where data visualization comes in.
Unleashing Feature Flags onto Kafka Consumers
Feature flags are a tool to strategically enable or disable functionality at runtime. They are often used to drive different user experiences but can also be useful in real-time data systems. In this post, we’ll […]
A security model for developers
Software security is more important than ever, but developing secure applications is more confusing than ever. TLS, mTLS, RBAC, SAML, OAUTH, OWASP, GDPR, SASL, RSA, JWT, cookie, attack vector, DDoS, firewall, VPN, security groups, exploit, […]