Blog posts about topics that are not related to the Java JVM in someway are few and far between on the Object Partners blog, but since I’ve spent the last 2 years working about 75% outside of the JVM doing DevOps work, I thought it would be worth while to start a series of posts with some of the lessons I’ve learned.

This first post is a retrospective about building out a production environment that is managed using Chef. A little background to start. When we first moved to using Chef for our server configuration management, there was very little experience with the tool among the team. Many of these lessons learned are a 2 year historical perspective of a lot of code that was written to “get something that works”.

Lesson 1 – Understand the difference between Attributes, Databags, Roles, Environments, and Solo Nodes

This is the number 1 lesson in my list because it was the root of a lot of other trouble to follow. Chef ships with a fairly complex system for managing variables that are available to recipes and resources. It’s complex because there is a hierarchy applied to how defining variables in different spots overrides each other (precedence system). Before you doing anything, read the Chef document on Attributes. Then because it’s Chef documentation, read it again, and again, and again.

Here’s a good rule of thumb that works for me: 1) Only put values that don’t change in Databags. If you even remotely think that value will change depending on the server or environment you are deploying, put it in an attribute. 2) Use recipe attributes for default values. 3) Use roles to provide attributes overrides.

Lesson 2 – Use Databags as Key-Value store

When dealing with databags, I’ve found that its best to treat the databag contents as a Key-Value store. Don’t be tempted to store hashes or lists in them. Remember, databags are for immutable content. This allows for more readability into your databags. If you’re storing a a Hash in there, it can be difficult to decipher what is happening later. Especially when…

Often you’ll need to provide an encrypted value to Chef. Luckily, Chef ships with support for encrypted databags. When creating an encrypted databag, you end up with a JSON file that contains the encrypted value in a String form. If you were storing a Hash or Array in this value, when it’s encrypted, you no longer have the insight that it’s a Hash, Array, or a plain String value.

Lesson 3 – Don’t Loop

This is another lesson that resulted from being a developer but not understanding how Chef does it’s work. If you find yourself writing a loop, for-each, or while block, take a step back and reconsider (as always there are some uses cases for looping in Chef).

This really amounts to understanding the flow of the Chef run and how resources are used. Initially you might write a recipe that loops of an array of services to install and call the necessary resources to install each service. That’s fine, but what happens when you change the array of services based on your environment? Maybe you want to run 2 machines with each hosting a different set of services. The issue here is that you’ll likely end up using an attributes that stores an Array and when you execute Chef you lose visibility into what is installed.

Instead, if you define individual install recipes for each service, then you can use a role to composite them together in whatever manner you like. The biggest benefits is that when Chef runs and expands the run_list (list of all recipes to execute), you will see exactly which recipes will be execute (and thus which services are installed).

Lesson 4 – Learn how to write a LWRP and do it

This lesson is a spin off of Lesson 3 in that once you start writing a recipe for each service to install, you might find yourself duplicating the same code over and over. In Chef you can write your own resource providers using a couple methods. First, you can write a “heavy-weight” provider by writing Ruby classes that extend the Chef::Resource and Chef::Provider classes. Alternatively, you can write a “light-weight” resource provider (LWRP). An LWRP is simply a short form resource provider using a Chef DSL. It takes some getting used to, but once you write a couple, you’ll get the hang of it. Start simple and small. Then you can encapsulate your custom logic (that calls other Chef resources) in a custom LWRP which you can call from your recipes reducing their size and complexity.

Lesson 5 – Use Berkshelf to manage cookbooks

Berkshelf is the de-facto standard for managing cookbooks dependencies in Chef. Start with it on Day 1. Do not copy cookbooks to your own repository. You’ll lose any insight into if someone modified its contents which makes updating much harder. It will also force you to isolate your custom cookbooks from upstream public cookbooks you are using. Remember, composition is good, so treat public cookbooks as immutable items that you utilize.

If you aren’t using a Chef Server but are instead running Chef in solo-mode, get in the habit on Day 1 of producing an output artifact that contains all your Chef code (cookbooks, roles, solo-nodes, etc) and deploy that to your target machines. From this repo you can provide custom cookbooks and have Berkshelf vendor them into the standard cookbooks directory (or configure Chef to have multiple cookbook directories).