Blogging in Lisp


Chef Dev Newbie

Submitted by Bill St. Clair on Sat, 03 Oct 2015 12:28:48 GMT

I discovered that Chef has an Erlang-based server and is looking for developers. Chef is the king of DevOps for huge installations, though it can manage, for free, smaller installations as well. My introduction was the Chef Style DevOps Kungfu speech video by Chef CTO Adam Jacob at ChefConf 2015. On configuration workstations and target nodes, Chef is largely Ruby scripts, nice simple ones that are amenable to building by people who don't write software for a living. There's a Chef Supermarket filled with Cookbooks full of Recipes submitted by the Chef community. A Supermarket Cookbook is an indexed github archive that you can modify, or use as is, and run on your nodes. A Recipe is a Ruby script. Yeah, the food metaphors are a bit much, but it's light-hearted.


I went through the first two tutorials, learned how to install and run the command line tools on an Ubuntu VM on my Mac and to use them to configure a node on a CloudShare VM that the Chef tutorial spins up for you for free. All straightforward and rock solid. And very well documented. Shiny!

Were I doing DevOps for a company's servers, I would have continued doing tutorials about using Chef. Since I'm a developer considering applying to work for them, I decided I should give a try at building the server. I would guess that most of their customers just use Chef's server, free for small installations or paid for larger ones. But some companies need to run their own. There's a tutorial for installing the server, which looks involved, but not really hard, but I don't just want to run a server, I want to be able to change its code. Chef's code is open source and on their GitHub site. The server source is in the chef-server repository. The Erlang code is in src/oc_erchef.

Being a glutton for punishment, I decided to ignore the system requirements and try the installation on a Scaleway ARM machine. It's hard to beat €3/month for 4 cores, 2 gigs of RAM, and 50 gigs of HD, with lots of bandwidth. I got pretty far into installing the required Ruby gems, when it errored saying it needed an x8664 (to install Sun's Java). Rather than attempt to make it work, I switched over to a $10/month DigitalOcean VM (1 core, 1 gig of RAM, 30 gigs of HD). Still not as much RAM as Chef recommends, but I figured it should at least build. And it mostly did.

After spinning up an Ubuntu 14.04 VM, and adding a swap file, I installed Emacs:

sudo apt-get install emacs24-nox emacs24-el info

Then GCC & Erlang dependencies:

sudo apt-get install build-essential autoconf git libncurses5-dev
sudo apt-get install libssl-dev libssh-dev

And Erlang 17.5:

mkdir -p ~/erlang/release
cd ~/erlang
git clone
ln -s ~/erlang/kerl/kerl ~/bin/
kerl build 17.5 erlang-17.5
kerl install erlang-17.5 ~/erlang/release/17.5
kerl_activate 17.5    # My own simple script

Chef likes at least Ruby 2.0, with Bundler:

sudo apt-get install ruby2.0

# The Ubuntu 14.04 ruby2.0 package installs 1.9.1 and 2.0, with 1.9.1 selected.
# Fix that.
sudo ln -sf ruby2.0 /usr/bin/ruby
sudo ln -sf gem2.0 /usr/bin/gem

sudo apt-get install ruby2.0-dev bundler

Ready to clone the server code:

mkdir -p ~/chef
cd ~/chef
git clone

The installation appears to be intended to be done by root, but I decided to do it from a regular account. It worked fine, but I had to change a few normally root-only directories to world-writeable. I suppose I could have just changed them to be owned by my account. Maybe I would have seen fewer complaints about them in the logs.

sudo chmod 777 /var/lib/gems/2.0.0
sudo mkdir -p /var/cache/omnibus
sudo chmod 777 /var/cache/omnibus
sudo mkdir -p /opt/opscode
sudo chmod 777 /opt/opscode

There's a whole bunch of Ruby code to download and build:

cd chef-server/omnibus
bundle install --binstubs

I watched it happen from Emacs, showing the build shell in one pane and dired in the other, sorted by time descending.

$ ls -w 90 /var/lib/gems/2.0.0/gems
addressable-2.3.8             hitimes-1.2.2                rake-10.4.2
berkshelf-3.3.0               httpclient-           retryable-2.0.2
berkshelf-api-client-1.3.0    ipaddress-0.8.0              ridley-4.2.0
buff-config-1.0.1             json-1.8.3                   rspec-3.3.0
buff-extensions-1.0.0         libyajl2-1.2.0               rspec-core-3.3.2
buff-ignore-1.1.1             method_source-0.8.2          rspec-expectations-3.3.1
buff-ruby_engine-0.1.0        mime-types-2.6.1             rspec-its-1.2.0
buff-shell_out-0.2.0          minitar-0.5.4                rspec_junit_formatter-0.2.3
builder-3.2.2                 mixlib-authentication-1.3.0  rspec-mocks-3.3.2
celluloid-0.16.0              mixlib-cli-1.5.0             rspec-support-3.3.0
celluloid-io-0.16.2           mixlib-config-2.2.1          ruby-progressbar-1.7.5
chef-12.4.1                   mixlib-log-1.6.0             sawyer-0.6.0
chef-config-12.4.1            mixlib-shellout-2.1.0        semverse-1.2.1
chefspec-4.3.0                mixlib-versioning-1.1.0      serverspec-2.21.0
chef-sugar-3.1.1              multi_json-1.11.2            sfl-2.2
chef-zero-4.2.3               multipart-post-2.0.0         slop-3.6.0
cleanroom-1.0.0               net-http-persistent-2.9.4    solve-1.2.1
coderay-1.1.0                 net-scp-1.2.1                specinfra-2.41.1
dep_selector-1.0.3            net-ssh-2.9.2                syslog-logger-1.6.8
dep-selector-libgecode-1.0.2  net-ssh-gateway-1.2.0        systemu-2.6.5
diff-lcs-1.2.5                net-ssh-multi-1.2.1          thor-0.19.1
erubis-2.7.0                  net-telnet-0.1.1             timers-4.0.1
faraday-0.9.1                 nio4r-1.1.1                  uber-s3-0.1.1
fauxhai-2.3.0                 octokit-3.8.0                uuidtools-2.1.5
ffi-1.9.10                    ohai-8.5.1                   varia_model-0.4.0
ffi-yajl-2.2.2                plist-3.1.0                  wmi-lite-1.0.0
hashie-2.1.2                  pry-0.10.1
highline-1.7.3                rack-1.6.4

Buildig the server requires your git identity:

git config --global ""
git config --global "Your Name"

Finally I was ready to build the server proper:

cd ~/chef/chef-server/omnibus
bin/omnibus build chef-server

This took a very long time. Reminiscent of Gentoo Linux builds (which took overnight to build Open Office many years ago).

$ ls -w 90 /opt/opscode/embedded/bin
2to3            find2perl                pg_basebackup     resty
a2p             fz                       pgbench           ri
appbundler      gem                      pg_config         rspec
berks           generate_field_data      pg_controldata    ruby
bundle          genhash                  pg_ctl            run_erl
bundler         h2ph                     pg_dump           runit
bunzip2         h2xs                     pg_dumpall        runit-init
bzcat           htmldiff                 pg_receivexlog    runsv
bzcmp           httpclient               pg_resetxlog      runsvchdir
bzdiff          idle                     pg_restore        runsvdir
bzegrep         infocmp                  pg_standby        runsvdir-start
bzfgrep         infotocap                pg_test_fsync     s2p
bzgrep          initdb                   pg_test_timing    sequel
bzip2           instmodsh                pg_upgrade        serverspec-init
bzip2recover    irb                      piconv            shasum
bzless          json_pp                  pkg-config        shell-quote
bzmore          knife                    pl2pm   
c2ph            ldiff                    pod2html          splain
captoinfo       libnetcfg                pod2latex         sqitch
chef-apply      libtool                  pod2man           sv
chef-client     libtoolize               pod2text          svlogd
chef-shell      lzcat                    pod2usage         tabs
chef-solo       lzcmp                    podchecker        testgdbm
chef-zero       lzdiff                   podselect         testrb
chpst           lzegrep                  postgres          thor
clear           lzfgrep                  postmaster        tic
clusterdb       lzgrep                   prove             toe
coderay         lzless                   pry               to_erl
config_data     lzma                     psed              tput
corelist        lzmadec                  psql              tset
cpan            lzmainfo                 pstruct           typer
cpan2dist       lzmore                   ptar              unlzma
cpanm           makedepend               ptardiff          unxz
cpanp           minitar                  ptargrep          utmpset
cpanp-run-perl  moose-outdated           pydoc             uuid
createdb        mzn-gecode               python            uuid-config
createlang      ncurses5-config          python2           vacuumdb
createuser      ncursesw5-config         python2.7         vacuumlo
c_rehash        node                     python2.7-config  xml2-config
ct_run          nokogiri                 python2-config    xmlcatalog
dbilogstrip     npm                      python-config     xmllint
dbiprof         ohai                     rabbitmqctl       xslt-config
dbiproxy        oid2name                 rabbitmq-env      xsltproc
dialyzer        omnibus-ctl              rabbitmq-server   xsubpp
dropdb          openssl                  rackup            xz
droplang        package-stash-conflicts  rake              xzcat
dropuser        pcre-config              rdoc              xzcmp
ecpg            pcregrep                 rebar             xzdec
enc2xs          pcretest                 redis-benchmark   xzdiff
epmd            perl                     redis-check-aof   xzegrep
erb             perl5.18.1               redis-check-dump  xzfgrep
erl             perlbug                  redis-cli         xzgrep
erlc            perldoc                  redis-sentinel    xzless
erubis          perlivp                  redis-server      xzmore
escript         perlthanks               reindexdb         zipdetails
ffi-yajl-bench  pg_archivecleanup        reset

It finally started compiling the Erlang code, which it retried three times, with increasingly long pauses in between

[Builder: oc_erchef] W | [1/3] Failed to execute command. Retrying in 10 seconds...
[Builder: oc_erchef] W | [2/3] Failed to execute command. Retrying in 20 seconds...
[Builder: oc_erchef] W | [3/3] Failed to execute command. Retrying in 40 seconds...
===> Compiling oc_chef_authz                                                                                                                                  
===> Compiling .../oc_erchef/.../oc_chef_authz/src/oc_chef_policy.erl failed                                                
.../oc_erchef/.../oc_chef_authz/src/oc_chef_policy.erl:26: behaviour chef_object undefined

Apparently, chef-server/src/oc_erchef/apps/oc_chef_authz needs to be built in an environment with the chef-server/src/oc_erchef/apps/chef_objects/src/chef_object.erl module available, to provide the chef_object behaviour. I may figure this out sometime soon, but I won't be working much this weekend.

Add comment   Edit post   Add post

Comments (4)

Building as root makes no difference

Submitted by Bill St. Clair on Mon, 05 Oct 2015 12:15:40 GMT

At Oliver Ferrigni's request, I reran the install as root, in a fresh VM. Same error. Fixed a couple of things I forgot in the install instructions above.

I'll work on this more this afternoon. Something came up for the morning.

My guess is that it's either an install path baked in, which is very unlikely, or an Erlang library path environment setting, which I'll try first.

Edit comment

Script to build just oc_erchef

Submitted by Bill St. Clair on Mon, 05 Oct 2015 22:08:11 GMT

I figured out how to separate the Erlang build from omnibus, making it much easier to repetively build just the Erlang code. Also, omnibus deletes all the build output, making it even harder to figure out what's going on. This script allowed me to reproduce my error in a few minutes, and left the _build directory around:

# How to build oc_erchef without all the omnibus nonsense.                                                                                                    
# This is done by ~/chef/chef-server/omnibus/config/software/oc_erchef.rb                                                                                     

cd ~wws/chef/chef-server/src/oc_erchef
export PATH=/opt/opscode/embedded/bin/:$PATH
# From chef-server/omnibus/config/projects/chef-server.rb                                                                                                     
export REL_VERSION=12.2.0
make omnibus

Interestingly, this gave me the folliowing warning, shortly before erroring out due to the missing chef_object behaviour:

bundling up depselector, This might take a while...
Don't run Bundler as root. Bundler can ask for sudo if it is needed, and
installing your bundle as root will break this application for all non-root
users on this machine.

Once I get this to work, I'll try again building it from a user account, with the output directories set world writeable.

Late in the day realization: it would probably work to just have an Erlang 17.5 in the PATH and do "make". Will try that tomorrow. Oh well, I learned a lot about how Omnibus works.

Edit comment

More progress on oc_erchef build

Submitted by Bill St. Clair on Tue, 06 Oct 2015 10:51:58 GMT

I tried adding erl_first_files to rebar.config to compile the two files that define behaviours first. No change. Rebar3 is supposed to do that automatically.

Commenting out warnings_as_errors in the erl_opts setting made the compile at least finish, with a bunch of warnings about the missing behaviour. What I don't understand is why apps/oc_chef_authz is getting compiled before apps/oc_objects.

I also updated my oc_erchef_build script to be able to just compile (there really should be a compile target in the Makefile):

# How to build oc_erchef without all the omnibus nonsense.                                                                                                    
# This is done by ~/chef/chef-server/omnibus/config/software/oc_erchef.rb                                                                                     

cd ~wws/chef/chef-server/src/oc_erchef
export PATH=/opt/opscode/embedded/bin/:$PATH
# From chef-server/omnibus/config/projects/chef-server.rb                                                                                                     
export REL_VERSION=12.2.0
if [[ $MAKE_TARGET == 'compile' ]]; then
  ./rebar3 compile

Here's the output of the build:

# ~wws/bin/oc_erchef-build compile
===> Verifying dependencies...
bundling up depselector, This might take a while...
Don't run Bundler as root. Bundler can ask for sudo if it is needed, and
installing your bundle as root will break this application for all non-root
users on this machine.
Using dep-selector-libgecode 1.0.2
Using ffi 1.9.10
Using dep_selector 1.0.3 from git:// (at master)
Using erlectricity 1.1.1 from git:// (at master)
Using uuidtools 2.1.5
Using bundler 1.10.6
Bundle complete! 3 Gemfile dependencies, 6 gems now installed.
Bundled gems are installed into ./.bundle.
===> Compiling oc_chef_authz
  Warning: behaviour chef_object undefined
===> Compiling chef_objects
===> Compiling chef_index
===> Running neotoma...
===> Compiling oc_chef_wm
===> Compiling depsolver
===> Compiling chef_db
===> Compiling oc_erchef
===> Compiling chef_test

Checking out the 12.1-stable branch of chef-server gets a version that uses rebar instead of rebar3 and builds without warning. Updating the bundled rebar3 in the master branch to the latest release at does the depselector stuff after the rest of the compiles, but still compiles oc_chef_authz before chef_objects, hence gets the missing behaviour warnings.

Edit comment

Well, I got it to build

Submitted by Bill St. Clair on Wed, 07 Oct 2015 21:27:58 GMT

Well, I got it to build without error, even with warnings_as_errors in erl_opts. All I had to do was add chef_objects to the applications in

I made a pull request here. For some reason it fails tests. I'll be investigating that, but first I have to build everything on a 4-core DigitalOcean VM, since the eunit tests fail:

enif_send: env==NULL on non-SMP VM
make: *** [all] Aborted (core dumped)

Edit comment

Add comment   Edit post   Add post

Previous Posts:

Kerl Rocks!!
SSL Certificate Added
Mac OS X El Capitan Startup Transients
Ready... Set... Blog...
Seeking Work
Feeds List for RSS Aggregator
Lisp News Aggregator
RSS Aggregator Goes Live
RSS Aggregator
I'm Not Dead yet. Yes you are. Shut up.