Dec 19, 2018

A Useful Tip or Two, Part 1

Introduction

I’ve lately had the opportunity to do a lot of teaching to some up and coming DevOps personnel. I take a lot of pride in teaching because good talent, as we all know too well, is hard to find. Helping a person find their inner engineer can be a daunting task, though. What’s worse, many of our learning tools are pretty obtuse to a newbie. Telling a normal person to (or being told to) ‘man ip’ is part of what makes us into the shocked-out remnants of our former selves.

If you happen to be one of my students, or just a student in general, then this series of posts is for you. I’m sure you’ve read the bash programming manual, or are about to, so this post is less about for-loops and if-else blocks, and more about the little tricks we pull on a daily basis. More to the point, it’s about how I can flit about from problem to problem without worrying about breaking anything (too badly). It’s about how I can screw things up, but rely on my own safe habits and the tools available to me in order to remain (mostly) safe.

To that effect, I try to keep some useful examples of everyday work that can inspire questions and show the tools we have at our disposal at full use.

One of my favorite tools is the Bash shell. It makes me feel heroic, like I have a cape. When time is tight, I can construct archaic one-liners to save the day. Which can also be an excellent way to wreck your day. See the following example:

Example: BASH subtleties

In the following example, I cat out a json file I have. Then I change it in a potentially dangerous manner. Finally, I’ll do the same thing, but in a much safer manner. I do this by taking advantage of standard bash tools.

Consider this:

$ cat ugly.json
{
"version": "old",
"name": "doug",
"potentially": "lots more data"
}

If I wanted to alter this output, I’d normally reach directly for jq (‘man jq’ … lol). Try this:

$ jq '.version = "old"' ugly.json
{
"version": "old",
"name": "doug",
"potentially": "lots more data"
}

It’s not a permanent change, as jq merely outputs to STDOUT. Just look at the contents of ugly.json now:

$ cat ugly.json
{
"version": "current",
"name": "doug",
"potentially": "lots more data"
}

I guess that means we have to jump through hoops to make this a permanent change. Here’s one way:

$ jq '.version = "old"' ugly.json > .my.ugly.json
$ cp .my.ugly.json ugly.json
$ cat ugly.json
{
"version": "old",
"name": "doug",
"potentially": "lots more data"
}

In my scenario, I have to do this kind of modification often. I could, of course, just script it. Or I could do a one liner because I wish I was an ninja-astronaut, and that’s what they would do.

$ jq '.version = "old"' ugly.json > .my.ugly.json ; mv .my.ugly.json ugly.json ; cat ugly.json
{
"version": "old",
"name": "doug",
"potentially": "lots more data"
}

( And this is where all those command-line-heroics get me in trouble… )

Looks like it works, right? So… now I can do this several times, fearlessly hitting every time?

Unfortunately, not really. In my one-liner, I use the ‘;’ operator. Like:

$ cmd1 ; cmd2 ; cmd3

If I had done that and cmd1 fails, cmd2 and cmd3 will run anyway. So looking back at my example, you can see that if jq fails to run properly, then I stand the chance to copy over ugly.json with bad, incomplete, or no data at all. Which means I lost data!

The following method, using the ‘&&’ operator, is a much safer means of executing this sort of sequential, dependent command sequence:

$ cmd1 && cmd2 && cmd3

In that example, cmd3 only runs if cmd2 exited with success. In turn, cmd2 only runs if cmd1 runs successfully. I should take advantage of this in my own command sequence. This would be the proper way for me to run my command line:

$ jq '.version = "old"' ugly.json > .my.ugly.json && mv .my.ugly.json ugly.json && cat ugly.json
{
"version": "old",
"name": "doug",
"potentially": "lots more data"
}

Now I can run this, or similar, command without fear of losing data.

Conclusion

If you’ve made it this far, you’ve seen that even old professionals can walk their way into awkward traps by getting too loose on the CLI. However, as I’ve also attempted to demonstrate, cultivating and practicing safe habits can save your whole day (and data).

Thanks for reading, and happy CLI’ing.

About the Author

Doug Hernandez profile.

Doug Hernandez

Sr. Consultant
Leave a Reply

Your email address will not be published.

Related Blog Posts
Building Better Data Visualization Experiences: Part 1 of 2
Through direct experience with data scientists, business analysts, lab technicians, as well as other UX professionals, I have found that we need a better understanding of the people who will be using our data visualization products in order to build them. Creating a product utilizing data with the goal of providing insight is fundamentally different from a typical user-centric web experience, although traditional UX process methods can help.
Kafka Schema Evolution With Java Spring Boot and Protobuf
In this blog I will be demonstrating Kafka schema evolution with Java, Spring Boot and Protobuf.  This app is for tutorial purposes, so there will be instances where a refactor could happen. I tried to […]
Redis Bitmaps: Storing state in small places
Redis is a popular open source in-memory data store that supports all kinds of abstract data structures. In this post and in an accompanying example Java project, I am going to explore two great use […]
Let’s build a WordPress & Kernel updated AMI with Packer
First, let’s start with What is an AMI? An Amazon Machine Image (AMI) is a master image for the creation of virtual servers in an AWS environment. The machine images are like templates that are configured with […]