Chef Dev Newbie
Submitted by Bill St. Clair on Sat, 03 Oct 2015 12:28:48 GMT
I discovered that Chef has an Erlang-based server and is looking for developers. Chef is the king of DevOps for huge installations, though it can manage, for free, smaller installations as well. My introduction was the Chef Style DevOps Kungfu speech video by Chef CTO Adam Jacob at ChefConf 2015. On configuration workstations and target nodes, Chef is largely Ruby scripts, nice simple ones that are amenable to building by people who don't write software for a living. There's a Chef Supermarket filled with Cookbooks full of Recipes submitted by the Chef community. A Supermarket Cookbook is an indexed github archive that you can modify, or use as is, and run on your nodes. A Recipe is a Ruby script. Yeah, the food metaphors are a bit much, but it's light-hearted.
I went through the first two tutorials, learned how to install and run the command line tools on an Ubuntu VM on my Mac and to use them to configure a node on a CloudShare VM that the Chef tutorial spins up for you for free. All straightforward and rock solid. And very well documented. Shiny!
Were I doing DevOps for a company's servers, I would have continued doing tutorials about using Chef. Since I'm a developer considering applying to work for them, I decided I should give a try at building the server. I would guess that most of their customers just use Chef's server, free for small installations or paid for larger ones. But some companies need to run their own. There's a tutorial for installing the server, which looks involved, but not really hard, but I don't just want to run a server, I want to be able to change its code. Chef's code is open source and on their GitHub site. The server source is in the chef-server repository. The Erlang code is in src/oc_erchef.
Being a glutton for punishment, I decided to ignore the system requirements and try the installation on a Scaleway ARM machine. It's hard to beat €3/month for 4 cores, 2 gigs of RAM, and 50 gigs of HD, with lots of bandwidth. I got pretty far into installing the required Ruby gems, when it errored saying it needed an x8664 (to install Sun's Java). Rather than attempt to make it work, I switched over to a $10/month DigitalOcean VM (1 core, 1 gig of RAM, 30 gigs of HD). Still not as much RAM as Chef recommends, but I figured it should at least build. And it mostly did.
After spinning up an Ubuntu 14.04 VM, and adding a swap file, I installed Emacs:
sudo apt-get install emacs24-nox emacs24-el info
Then GCC & Erlang dependencies:
sudo apt-get install build-essential autoconf git libncurses5-dev sudo apt-get install libssl-dev libssh-dev
And Erlang 17.5:
mkdir -p ~/erlang/release cd ~/erlang git clone git@github.com:yrashk/kerl.git ln -s ~/erlang/kerl/kerl ~/bin/ kerl build 17.5 erlang-17.5 kerl install erlang-17.5 ~/erlang/release/17.5 kerl_activate 17.5 # My own simple script
Chef likes at least Ruby 2.0, with Bundler:
sudo apt-get install ruby2.0 # The Ubuntu 14.04 ruby2.0 package installs 1.9.1 and 2.0, with 1.9.1 selected. # Fix that. sudo ln -sf ruby2.0 /usr/bin/ruby sudo ln -sf gem2.0 /usr/bin/gem sudo apt-get install ruby2.0-dev bundler
Ready to clone the server code:
mkdir -p ~/chef cd ~/chef git clone git@github.com:chef/chef-server.git
The installation appears to be intended to be done by root, but I decided to do it from a regular account. It worked fine, but I had to change a few normally root-only directories to world-writeable. I suppose I could have just changed them to be owned by my account. Maybe I would have seen fewer complaints about them in the logs.
sudo chmod 777 /var/lib/gems/2.0.0 sudo mkdir -p /var/cache/omnibus sudo chmod 777 /var/cache/omnibus sudo mkdir -p /opt/opscode sudo chmod 777 /opt/opscode
There's a whole bunch of Ruby code to download and build:
cd chef-server/omnibus bundle install --binstubs
I watched it happen from Emacs, showing the build shell in one pane and dired in the other, sorted by time descending.
$ ls -w 90 /var/lib/gems/2.0.0/gems addressable-2.3.8 hitimes-1.2.2 rake-10.4.2 berkshelf-3.3.0 httpclient-2.6.0.1 retryable-2.0.2 berkshelf-api-client-1.3.0 ipaddress-0.8.0 ridley-4.2.0 buff-config-1.0.1 json-1.8.3 rspec-3.3.0 buff-extensions-1.0.0 libyajl2-1.2.0 rspec-core-3.3.2 buff-ignore-1.1.1 method_source-0.8.2 rspec-expectations-3.3.1 buff-ruby_engine-0.1.0 mime-types-2.6.1 rspec-its-1.2.0 buff-shell_out-0.2.0 minitar-0.5.4 rspec_junit_formatter-0.2.3 builder-3.2.2 mixlib-authentication-1.3.0 rspec-mocks-3.3.2 celluloid-0.16.0 mixlib-cli-1.5.0 rspec-support-3.3.0 celluloid-io-0.16.2 mixlib-config-2.2.1 ruby-progressbar-1.7.5 chef-12.4.1 mixlib-log-1.6.0 sawyer-0.6.0 chef-config-12.4.1 mixlib-shellout-2.1.0 semverse-1.2.1 chefspec-4.3.0 mixlib-versioning-1.1.0 serverspec-2.21.0 chef-sugar-3.1.1 multi_json-1.11.2 sfl-2.2 chef-zero-4.2.3 multipart-post-2.0.0 slop-3.6.0 cleanroom-1.0.0 net-http-persistent-2.9.4 solve-1.2.1 coderay-1.1.0 net-scp-1.2.1 specinfra-2.41.1 dep_selector-1.0.3 net-ssh-2.9.2 syslog-logger-1.6.8 dep-selector-libgecode-1.0.2 net-ssh-gateway-1.2.0 systemu-2.6.5 diff-lcs-1.2.5 net-ssh-multi-1.2.1 thor-0.19.1 erubis-2.7.0 net-telnet-0.1.1 timers-4.0.1 faraday-0.9.1 nio4r-1.1.1 uber-s3-0.1.1 fauxhai-2.3.0 octokit-3.8.0 uuidtools-2.1.5 ffi-1.9.10 ohai-8.5.1 varia_model-0.4.0 ffi-yajl-2.2.2 plist-3.1.0 wmi-lite-1.0.0 hashie-2.1.2 pry-0.10.1 highline-1.7.3 rack-1.6.4
Buildig the server requires your git identity:
git config --global user.email "you@example.com" git config --global user.name "Your Name"
Finally I was ready to build the server proper:
cd ~/chef/chef-server/omnibus bin/omnibus build chef-server
This took a very long time. Reminiscent of Gentoo Linux builds (which took overnight to build Open Office many years ago).
$ ls -w 90 /opt/opscode/embedded/bin 2to3 find2perl pg_basebackup resty a2p fz pgbench ri appbundler gem pg_config rspec berks generate_field_data pg_controldata ruby bundle genhash pg_ctl run_erl bundler h2ph pg_dump runit bunzip2 h2xs pg_dumpall runit-init bzcat htmldiff pg_receivexlog runsv bzcmp httpclient pg_resetxlog runsvchdir bzdiff idle pg_restore runsvdir bzegrep infocmp pg_standby runsvdir-start bzfgrep infotocap pg_test_fsync s2p bzgrep initdb pg_test_timing sequel bzip2 instmodsh pg_upgrade serverspec-init bzip2recover irb piconv shasum bzless json_pp pkg-config shell-quote bzmore knife pl2pm smtpd.py c2ph ldiff pod2html splain captoinfo libnetcfg pod2latex sqitch chef-apply libtool pod2man sv chef-client libtoolize pod2text svlogd chef-shell lzcat pod2usage tabs chef-solo lzcmp podchecker testgdbm chef-zero lzdiff podselect testrb chpst lzegrep postgres thor clear lzfgrep postmaster tic clusterdb lzgrep prove toe coderay lzless pry to_erl config_data lzma psed tput corelist lzmadec psql tset cpan lzmainfo pstruct typer cpan2dist lzmore ptar unlzma cpanm makedepend ptardiff unxz cpanp minitar ptargrep utmpset cpanp-run-perl moose-outdated pydoc uuid createdb mzn-gecode python uuid-config createlang ncurses5-config python2 vacuumdb createuser ncursesw5-config python2.7 vacuumlo c_rehash node python2.7-config xml2-config ct_run nokogiri python2-config xmlcatalog dbilogstrip npm python-config xmllint dbiprof ohai rabbitmqctl xslt-config dbiproxy oid2name rabbitmq-env xsltproc dialyzer omnibus-ctl rabbitmq-server xsubpp dropdb openssl rackup xz droplang package-stash-conflicts rake xzcat dropuser pcre-config rdoc xzcmp ecpg pcregrep rebar xzdec enc2xs pcretest redis-benchmark xzdiff epmd perl redis-check-aof xzegrep erb perl5.18.1 redis-check-dump xzfgrep erl perlbug redis-cli xzgrep erlc perldoc redis-sentinel xzless erubis perlivp redis-server xzmore escript perlthanks reindexdb zipdetails ffi-yajl-bench pg_archivecleanup reset
It finally started compiling the Erlang code, which it retried three times, with increasingly long pauses in between
[Builder: oc_erchef] W | [1/3] Failed to execute command. Retrying in 10 seconds... ... [Builder: oc_erchef] W | [2/3] Failed to execute command. Retrying in 20 seconds... ... [Builder: oc_erchef] W | [3/3] Failed to execute command. Retrying in 40 seconds... ... ===> Compiling oc_chef_authz ===> Compiling .../oc_erchef/.../oc_chef_authz/src/oc_chef_policy.erl failed .../oc_erchef/.../oc_chef_authz/src/oc_chef_policy.erl:26: behaviour chef_object undefined
Apparently, chef-server/src/oc_erchef/apps/oc_chef_authz needs to be built in an environment with the chef-server/src/oc_erchef/apps/chef_objects/src/chef_object.erl module available, to provide the chef_object behaviour. I may figure this out sometime soon, but I won't be working much this weekend.
Comments (4)
Building as root makes no difference
Submitted by Bill St. Clair on Mon, 05 Oct 2015 12:15:40 GMT
At Oliver Ferrigni's request, I reran the install as root, in a fresh VM. Same error. Fixed a couple of things I forgot in the install instructions above.
I'll work on this more this afternoon. Something came up for the morning.
My guess is that it's either an install path baked in, which is very unlikely, or an Erlang library path environment setting, which I'll try first.
Edit comment
Script to build just oc_erchef
Submitted by Bill St. Clair on Mon, 05 Oct 2015 22:08:11 GMT
I figured out how to separate the Erlang build from omnibus, making it much easier to repetively build just the Erlang code. Also, omnibus deletes all the build output, making it even harder to figure out what's going on. This script allowed me to reproduce my error in a few minutes, and left the
_builddirectory around:Interestingly, this gave me the folliowing warning, shortly before erroring out due to the missing
chef_objectbehaviour:Once I get this to work, I'll try again building it from a user account, with the output directories set world writeable.
Late in the day realization: it would probably work to just have an Erlang 17.5 in the PATH and do "make". Will try that tomorrow. Oh well, I learned a lot about how Omnibus works.
Edit comment
More progress on oc_erchef build
Submitted by Bill St. Clair on Tue, 06 Oct 2015 10:51:58 GMT
I tried adding
erl_first_filestorebar.configto compile the two files that define behaviours first. No change. Rebar3 is supposed to do that automatically.Commenting out
warnings_as_errorsin theerl_optssetting made the compile at least finish, with a bunch of warnings about the missing behaviour. What I don't understand is whyapps/oc_chef_authzis getting compiled beforeapps/oc_objects.I also updated my
oc_erchef_buildscript to be able to just compile (there really should be acompiletarget in theMakefile):Here's the output of the build:
Checking out the
12.1-stablebranch ofchef-servergets a version that usesrebarinstead ofrebar3and builds without warning. Updating the bundledrebar3in themasterbranch to the latest release at s3.amazonaws.com/rebar3/rebar3 does the depselector stuff after the rest of the compiles, but still compilesoc_chef_authzbeforechef_objects, hence gets the missing behaviour warnings.Edit comment
Well, I got it to build
Submitted by Bill St. Clair on Wed, 07 Oct 2015 21:27:58 GMT
Well, I got it to build without error, even with
warnings_as_errorsinerl_opts. All I had to do was addchef_objectsto the applications inoc_chef_authz.app.src.I made a pull request here. For some reason it fails tests. I'll be investigating that, but first I have to build everything on a 4-core DigitalOcean VM, since the eunit tests fail:
Edit comment