Berkshelf and Chef Cookbook Dependencies

Greetings, readers!

I continue delving into the specifics of management automation and configuration, while also trying to share my community experience.

In this article, I will continue talking  about the automation tool for solving Chef cookbook dependencies, namely Berkshelf.

What does Berkshelf have to do with this?

Chef has a significantly large and actively developing community that constantly contributes to creating and updating cookbooks. They are all stored on the community website, and we often use many of them.

But, there is a catch in our company regarding editing community cookbooks. The correct way of applying any changes related to the specifics of the corporate infrastructure is creating wrappers with updates (for example, reassigned attributes, switched recipes and so on). In brief, below is my description of how to create a wrapper. 

However, the fact that a correct path exists yet does not mean that everyone will follow it. For this reason, at some point our corporate cookbook repository gathered a large number of non-conventionally edited community cookbooks with misplaced edits that should be rather located in the wrapper than in cookbooks.. So the time has come to clean up the repository.

The plan was as follows:
— Select community cookbooks in our repository.
— Identify the quantity of local updates (use diff to find the differences between clean community cookbooks and the spoiled versions).
— Collect the updates in the wrapper and add a request into it to call the community cookbook or one of its recipes.
— Remove the community cookbook and add it to the wrapper’s dependencies.

But what do we do with the “cleaned up” community cookbooks? After the removal, how do the cleaned up cookbooks appear on the Chef Server?

This is where Berkshelf can help.

What is Berkshelf and how does it work?

Berkshelf is a dependency manager for Chef cookbooks. It is written in Ruby and has the methods and the API for interacting with the Chef Server.
Berkshelf is installed either from Ruby Gems or using Chef DK.
We use Berkshelf  2.0, but the recently released version 3.0 introduced several changes and “goodies”.
If you choose Ruby Gem, I suggest installing it within the framework of Ruby installed by the Chef Server (executable files are located here: /opt/chef/embedded/bin/). You must configure Berkshelf for it to be able to interact with the Chef Server, define the address of our server and the authorization certificates. To configure Berkshelf, run the following command:

berks configure

Configuration files are stored in one of the following locations:

$PWD/.berkshelf/config.json
$PWD/berkshelf/config.json
$PWD/berkshelf-config.json
$PWD/config.json
~/.berkshelf/config.json

After Berkshelf is correctly installed, you can move on to resolving cookbook dependencies. For this purpose, Berkshelf first copies the cookbooks and their dependencies to its shelves (it is a local Berkshelf storage/repository, by default it is the ~/.berkshelf/cookbooks/ directory), and then loads them to the Chef Server.

At this point, you are probably wondering about many things: “How does Berkshelf know about the dependencies? What does it do to resolve them?”
All instructions for Berkshelf are stored in the Berksfile located in the cookbook’s root directory. Create this file by running the following command within the root directory:

berks init

The contents of this file describe the dependencies and the source for resolving them. An example of a Berksfile:

site :opscode
metadata
cookbook 'my-cookbook', (:path | :git | :github)
cookbook 'my-book-2', ('> 1.0.0')

This file is to do the following:
— Recognize the dependencies from the metadata.rb file of our cookbook and load them from the Opscode Community website.
— Load the my-cookbook folder from the location defined in brackets (it can be a local path or a link to Git).
— Load the latest version (higher than 1.0.0) of the my-book-2 file  from the Opscode Community website.

Afterwards, you can run the following command

berks install 

and wait for the dependencies to be successfully copied to the Berkshelf file’s local storage. As a result, the storage directory should contain all cookbooks mentioned in the depends field of the metadata.rb file, their dependencies (the dependencies’ dependencies) and two cookbooks mentioned in the Berksfile. Verify the result with the following command:

berks shelf list

Next, run the following command:

berks upload

It will load everything from the local storage to the Chef Server. As a result (given that the Chef Server is available and we can authorize it for using the certificate files), all the dependencies should be resolved. Verify the result of the upload with the following command:

knife cookbook list

Its output should contain new cookbooks.

Essentially, this is the basic process of using Berkshelf. Of course, that’s not the entire functionality, as we don’t touch the interaction between Berkshelf and Vagrant, Chef Solo, Chef Client, as well as some of the improvements introduced in version 3.0. However, I consider this amount of information to be sufficient for the majority of users.

Our experience with Berkshelf

It all would be totally great if not for one thing — Berkshelf cannot work cyclically. I was very surprised, because I was unable to find an option that would make it “natively” consider the nested behavior of directories and cookbooks. What am I talking about? For example, imagine the most typical situation where you have the chef-repo directory, and the cookbooks directory with our cookbooks (./chef-repo/cookbooks/) is nested into it. In the current version of Berkshelf, you must go to each of the cookbook’s directories to run commands aimed at Berkshelf. Namely — write the bash script that would do it manually, seriously? It is a solution, but not the best one, to say the least – I would go as far as calling it a “crutch”.
On the Internet, I found an article about how to solve this issue using Ruby.
The following is the code of our “root” Berksfile stored in the ./chef-repo/ directory:
site :opscode
metadata

def dependencies(path)
berks = "#{path}/Berksfile"
instance_eval(File.read(berks)) if File.exists?(berks)
end

dir.glob('./cookbooks/*').each do |path|
dependencies path
cookbook File.basename(path), :path => path
end

cookbook 'gecode', '= 2.1.0'

Essentially, this chunk of code gathers the contents of all cookbook directories (their metadata.rb files and Berksfiles) into the “root” Berksfile.

We applied the solution successfully and voila – all cookbooks are delivered to our Chef Server.
However, the article I just mentioned  doesn’t say anything about one significant peculiarity — the format of Berksfiles in our cookbooks.

By trial and error, I discovered that they should look as follows:

group :name_of_cookbook do
cookbook 'my-cookbook', (:path | :git | :github)
cookbook 'my-book-2', ('> 1.0.0')
end

In other words, they gather the cookbooks that correspond to certain conditions, for example, versions or location, into a group (so that matching dependencies of different cookbooks do not cause conflicts, such as multiple entries). All other dependencies are obtained from the metadata.rb file. The following is a specific example to make my point more graphic:

group :database do
cookbook 'postgresql', '= 3.3.4'
cookbook 'aws', :path => './cookbooks/aws'
cookbook 'xfs', :path => './cookbooks/xfs'
cookbook 'mysql', '= 4.1.2'
end

Eventually, we will upload four cookbooks for the database cookbook. These cookbooks meet a set of certain conditions. Besides them, we upload all other dependencies mentioned in metadata.rb, including the aforementioned dependencies’ dependencies (pardon the tautology).
Last thing we need to do after editing the Berksfiles of our cookbooks is to run the following command in the ./chef-repo directory:

berks install && berks upload

So that’s the solution. It may not be the most reliable and convenient one, but it is a solution and it worked successfully on our repositories several times.
If anyone faced this issue before and has some valuable experience, please share it in the comments or contact me directly.

Thank you and so long until we meet again in another article!

P.S. A colleague just told me that Berkshelf v.3.0 is broken, so we advise against it!