Submitted by Bill St. Clair on Sat, 03 Oct 2015 12:28:48 GMT
I discovered that Chef has an Erlang-based server and is looking for developers. Chef is the king of DevOps for huge installations, though it can manage, for free, smaller installations as well. My introduction was the Chef Style DevOps Kungfu speech video by Chef CTO Adam Jacob at ChefConf 2015. On configuration workstations and target nodes, Chef is largely Ruby scripts, nice simple ones that are amenable to building by people who don't write software for a living. There's a Chef Supermarket filled with Cookbooks full of Recipes submitted by the Chef community. A Supermarket Cookbook is an indexed github archive that you can modify, or use as is, and run on your nodes. A Recipe is a Ruby script. Yeah, the food metaphors are a bit much, but it's light-hearted.
I went through the first two tutorials, learned how to install and run the command line tools on an Ubuntu VM on my Mac and to use them to configure a node on a CloudShare VM that the Chef tutorial spins up for you for free. All straightforward and rock solid. And very well documented. Shiny!
Were I doing DevOps for a company's servers, I would have continued doing tutorials about using Chef. Since I'm a developer considering applying to work for them, I decided I should give a try at building the server. I would guess that most of their customers just use Chef's server, free for small installations or paid for larger ones. But some companies need to run their own. There's a tutorial for installing the server, which looks involved, but not really hard, but I don't just want to run a server, I want to be able to change its code. Chef's code is open source and on their GitHub site. The server source is in the chef-server repository. The Erlang code is in
Being a glutton for punishment, I decided to ignore the system requirements and try the installation on a Scaleway ARM machine. It's hard to beat €3/month for 4 cores, 2 gigs of RAM, and 50 gigs of HD, with lots of bandwidth. I got pretty far into installing the required Ruby gems, when it errored saying it needed an x8664 (to install Sun's Java). Rather than attempt to make it work, I switched over to a $10/month DigitalOcean VM (1 core, 1 gig of RAM, 30 gigs of HD). Still not as much RAM as Chef recommends, but I figured it should at least build. And it mostly did.
After spinning up an Ubuntu 14.04 VM, and adding a swap file, I installed Emacs:
sudo apt-get install emacs24-nox emacs24-el info
Then GCC & Erlang dependencies:
sudo apt-get install build-essential autoconf git libncurses5-dev sudo apt-get install libssl-dev libssh-dev
And Erlang 17.5:
mkdir -p ~/erlang/release cd ~/erlang git clone firstname.lastname@example.org:yrashk/kerl.git ln -s ~/erlang/kerl/kerl ~/bin/ kerl build 17.5 erlang-17.5 kerl install erlang-17.5 ~/erlang/release/17.5 kerl_activate 17.5 # My own simple script
Chef likes at least Ruby 2.0, with Bundler:
sudo apt-get install ruby2.0 # The Ubuntu 14.04 ruby2.0 package installs 1.9.1 and 2.0, with 1.9.1 selected. # Fix that. sudo ln -sf ruby2.0 /usr/bin/ruby sudo ln -sf gem2.0 /usr/bin/gem sudo apt-get install ruby2.0-dev bundler
Ready to clone the server code:
mkdir -p ~/chef cd ~/chef git clone email@example.com:chef/chef-server.git
The installation appears to be intended to be done by root, but I decided to do it from a regular account. It worked fine, but I had to change a few normally root-only directories to world-writeable. I suppose I could have just changed them to be owned by my account. Maybe I would have seen fewer complaints about them in the logs.
sudo chmod 777 /var/lib/gems/2.0.0 sudo mkdir -p /var/cache/omnibus sudo chmod 777 /var/cache/omnibus sudo mkdir -p /opt/opscode sudo chmod 777 /opt/opscode
There's a whole bunch of Ruby code to download and build:
cd chef-server/omnibus bundle install --binstubs
I watched it happen from Emacs, showing the build shell in one pane and dired in the other, sorted by time descending.
$ ls -w 90 /var/lib/gems/2.0.0/gems addressable-2.3.8 hitimes-1.2.2 rake-10.4.2 berkshelf-3.3.0 httpclient-220.127.116.11 retryable-2.0.2 berkshelf-api-client-1.3.0 ipaddress-0.8.0 ridley-4.2.0 buff-config-1.0.1 json-1.8.3 rspec-3.3.0 buff-extensions-1.0.0 libyajl2-1.2.0 rspec-core-3.3.2 buff-ignore-1.1.1 method_source-0.8.2 rspec-expectations-3.3.1 buff-ruby_engine-0.1.0 mime-types-2.6.1 rspec-its-1.2.0 buff-shell_out-0.2.0 minitar-0.5.4 rspec_junit_formatter-0.2.3 builder-3.2.2 mixlib-authentication-1.3.0 rspec-mocks-3.3.2 celluloid-0.16.0 mixlib-cli-1.5.0 rspec-support-3.3.0 celluloid-io-0.16.2 mixlib-config-2.2.1 ruby-progressbar-1.7.5 chef-12.4.1 mixlib-log-1.6.0 sawyer-0.6.0 chef-config-12.4.1 mixlib-shellout-2.1.0 semverse-1.2.1 chefspec-4.3.0 mixlib-versioning-1.1.0 serverspec-2.21.0 chef-sugar-3.1.1 multi_json-1.11.2 sfl-2.2 chef-zero-4.2.3 multipart-post-2.0.0 slop-3.6.0 cleanroom-1.0.0 net-http-persistent-2.9.4 solve-1.2.1 coderay-1.1.0 net-scp-1.2.1 specinfra-2.41.1 dep_selector-1.0.3 net-ssh-2.9.2 syslog-logger-1.6.8 dep-selector-libgecode-1.0.2 net-ssh-gateway-1.2.0 systemu-2.6.5 diff-lcs-1.2.5 net-ssh-multi-1.2.1 thor-0.19.1 erubis-2.7.0 net-telnet-0.1.1 timers-4.0.1 faraday-0.9.1 nio4r-1.1.1 uber-s3-0.1.1 fauxhai-2.3.0 octokit-3.8.0 uuidtools-2.1.5 ffi-1.9.10 ohai-8.5.1 varia_model-0.4.0 ffi-yajl-2.2.2 plist-3.1.0 wmi-lite-1.0.0 hashie-2.1.2 pry-0.10.1 highline-1.7.3 rack-1.6.4
Buildig the server requires your git identity:
git config --global user.email "firstname.lastname@example.org" git config --global user.name "Your Name"
Finally I was ready to build the server proper:
cd ~/chef/chef-server/omnibus bin/omnibus build chef-server
This took a very long time. Reminiscent of Gentoo Linux builds (which took overnight to build Open Office many years ago).
$ ls -w 90 /opt/opscode/embedded/bin 2to3 find2perl pg_basebackup resty a2p fz pgbench ri appbundler gem pg_config rspec berks generate_field_data pg_controldata ruby bundle genhash pg_ctl run_erl bundler h2ph pg_dump runit bunzip2 h2xs pg_dumpall runit-init bzcat htmldiff pg_receivexlog runsv bzcmp httpclient pg_resetxlog runsvchdir bzdiff idle pg_restore runsvdir bzegrep infocmp pg_standby runsvdir-start bzfgrep infotocap pg_test_fsync s2p bzgrep initdb pg_test_timing sequel bzip2 instmodsh pg_upgrade serverspec-init bzip2recover irb piconv shasum bzless json_pp pkg-config shell-quote bzmore knife pl2pm smtpd.py c2ph ldiff pod2html splain captoinfo libnetcfg pod2latex sqitch chef-apply libtool pod2man sv chef-client libtoolize pod2text svlogd chef-shell lzcat pod2usage tabs chef-solo lzcmp podchecker testgdbm chef-zero lzdiff podselect testrb chpst lzegrep postgres thor clear lzfgrep postmaster tic clusterdb lzgrep prove toe coderay lzless pry to_erl config_data lzma psed tput corelist lzmadec psql tset cpan lzmainfo pstruct typer cpan2dist lzmore ptar unlzma cpanm makedepend ptardiff unxz cpanp minitar ptargrep utmpset cpanp-run-perl moose-outdated pydoc uuid createdb mzn-gecode python uuid-config createlang ncurses5-config python2 vacuumdb createuser ncursesw5-config python2.7 vacuumlo c_rehash node python2.7-config xml2-config ct_run nokogiri python2-config xmlcatalog dbilogstrip npm python-config xmllint dbiprof ohai rabbitmqctl xslt-config dbiproxy oid2name rabbitmq-env xsltproc dialyzer omnibus-ctl rabbitmq-server xsubpp dropdb openssl rackup xz droplang package-stash-conflicts rake xzcat dropuser pcre-config rdoc xzcmp ecpg pcregrep rebar xzdec enc2xs pcretest redis-benchmark xzdiff epmd perl redis-check-aof xzegrep erb perl5.18.1 redis-check-dump xzfgrep erl perlbug redis-cli xzgrep erlc perldoc redis-sentinel xzless erubis perlivp redis-server xzmore escript perlthanks reindexdb zipdetails ffi-yajl-bench pg_archivecleanup reset
It finally started compiling the Erlang code, which it retried three times, with increasingly long pauses in between
[Builder: oc_erchef] W | [1/3] Failed to execute command. Retrying in 10 seconds... ... [Builder: oc_erchef] W | [2/3] Failed to execute command. Retrying in 20 seconds... ... [Builder: oc_erchef] W | [3/3] Failed to execute command. Retrying in 40 seconds... ... ===> Compiling oc_chef_authz ===> Compiling .../oc_erchef/.../oc_chef_authz/src/oc_chef_policy.erl failed .../oc_erchef/.../oc_chef_authz/src/oc_chef_policy.erl:26: behaviour chef_object undefined
chef-server/src/oc_erchef/apps/oc_chef_authz needs to be built in an environment with the
chef-server/src/oc_erchef/apps/chef_objects/src/chef_object.erl module available, to provide the
chef_object behaviour. I may figure this out sometime soon, but I won't be working much this weekend.