Submitted by Bill St. Clair on Sat, 03 Oct 2015 17:50:23 GMT
github.com/billstclair/wws-scripts is a small collection of simple shell scripts that I've written over the years.
Documented by its README.
Lisplog is a templating system that blends Apache and Hunchentoot to aid in the maintenance of a blog-like web site.
It is open source, written in Common Lisp, and the code is at github.com/billstclair/Lisplog
Submitted by Bill St. Clair on Sat, 03 Oct 2015 17:50:23 GMT
github.com/billstclair/wws-scripts is a small collection of simple shell scripts that I've written over the years.
Documented by its README.
Submitted by Bill St. Clair on Sat, 03 Oct 2015 12:28:48 GMT
I discovered that Chef has an Erlang-based server and is looking for developers. Chef is the king of DevOps for huge installations, though it can manage, for free, smaller installations as well. My introduction was the Chef Style DevOps Kungfu speech video by Chef CTO Adam Jacob at ChefConf 2015. On configuration workstations and target nodes, Chef is largely Ruby scripts, nice simple ones that are amenable to building by people who don't write software for a living. There's a Chef Supermarket filled with Cookbooks full of Recipes submitted by the Chef community. A Supermarket Cookbook is an indexed github archive that you can modify, or use as is, and run on your nodes. A Recipe is a Ruby script. Yeah, the food metaphors are a bit much, but it's light-hearted.
I went through the first two tutorials, learned how to install and run the command line tools on an Ubuntu VM on my Mac and to use them to configure a node on a CloudShare VM that the Chef tutorial spins up for you for free. All straightforward and rock solid. And very well documented. Shiny!
Were I doing DevOps for a company's servers, I would have continued doing tutorials about using Chef. Since I'm a developer considering applying to work for them, I decided I should give a try at building the server. I would guess that most of their customers just use Chef's server, free for small installations or paid for larger ones. But some companies need to run their own. There's a tutorial for installing the server, which looks involved, but not really hard, but I don't just want to run a server, I want to be able to change its code. Chef's code is open source and on their GitHub site. The server source is in the chef-server repository. The Erlang code is in
Being a glutton for punishment, I decided to ignore the system requirements and try the installation on a Scaleway ARM machine. It's hard to beat €3/month for 4 cores, 2 gigs of RAM, and 50 gigs of HD, with lots of bandwidth. I got pretty far into installing the required Ruby gems, when it errored saying it needed an x8664 (to install Sun's Java). Rather than attempt to make it work, I switched over to a $10/month DigitalOcean VM (1 core, 1 gig of RAM, 30 gigs of HD). Still not as much RAM as Chef recommends, but I figured it should at least build. And it mostly did.
After spinning up an Ubuntu 14.04 VM, and adding a swap file, I installed Emacs:
$ sudo apt-get install emacs24-nox emacs24-el info
Then GCC & Erlang dependencies:
$ sudo apt-get install build-essential autoconf git libncurses5-dev $ sudo apt-get install libssl-dev libssh-dev
And Erlang 17.5:
mkdir -p ~/erlang/release cd ~/erlang git clone email@example.com:yrashk/kerl.git ln -s ~/erlang/kerl/kerl ~/bin/ kerl build 17.5 erlang-17.5 kerl install erlang-17.5 ~/erlang/release/17.5 kerl_activate 17.5 # My own simple script
Chef likes at least Ruby 2.0, with Bundler:
sudo apt-get install ruby2.0 # The Ubuntu 14.04 ruby2.0 package installs 1.9.1 and 2.0, with 1.9.1 selected. # Fix that. sudo ln -sf ruby2.0 /usr/bin/ruby sudo ln -sf gem2.0 /usr/bin/gem sudo apt-get install ruby2.0-dev bundler
Ready to clone the server code:
mkdir -p ~/chef cd ~/chef git clone firstname.lastname@example.org:chef/chef-server.git
The installation appears to be intended to be done by root, but I decided to do it from a regular account. It worked fine, but I had to change a few normally root-only directories to world-writeable. I suppose I could have just changed them to be owned by my account. Maybe I would have seen fewer complaints about them in the logs.
sudo chmod 777 /var/lib/gems/2.0.0 sudo mkdir -p /var/cache/omnibus sudo chmod 777 /var/cache/omnibus sudo mkdir -p /opt/opscode sudo chmod 777 /opt/opscode
There's a whole bunch of Ruby code to download and build:
cd chef-server/omnibus bundle install --binstubs
I watched it happen from Emacs, showing the build shell in one pane and dired in the other, sorted by time descending.
$ ls -w 90 /var/lib/gems/2.0.0/gems addressable-2.3.8 hitimes-1.2.2 rake-10.4.2 berkshelf-3.3.0 httpclient-126.96.36.199 retryable-2.0.2 berkshelf-api-client-1.3.0 ipaddress-0.8.0 ridley-4.2.0 buff-config-1.0.1 json-1.8.3 rspec-3.3.0 buff-extensions-1.0.0 libyajl2-1.2.0 rspec-core-3.3.2 buff-ignore-1.1.1 method_source-0.8.2 rspec-expectations-3.3.1 buff-ruby_engine-0.1.0 mime-types-2.6.1 rspec-its-1.2.0 buff-shell_out-0.2.0 minitar-0.5.4 rspec_junit_formatter-0.2.3 builder-3.2.2 mixlib-authentication-1.3.0 rspec-mocks-3.3.2 celluloid-0.16.0 mixlib-cli-1.5.0 rspec-support-3.3.0 celluloid-io-0.16.2 mixlib-config-2.2.1 ruby-progressbar-1.7.5 chef-12.4.1 mixlib-log-1.6.0 sawyer-0.6.0 chef-config-12.4.1 mixlib-shellout-2.1.0 semverse-1.2.1 chefspec-4.3.0 mixlib-versioning-1.1.0 serverspec-2.21.0 chef-sugar-3.1.1 multi_json-1.11.2 sfl-2.2 chef-zero-4.2.3 multipart-post-2.0.0 slop-3.6.0 cleanroom-1.0.0 net-http-persistent-2.9.4 solve-1.2.1 coderay-1.1.0 net-scp-1.2.1 specinfra-2.41.1 dep_selector-1.0.3 net-ssh-2.9.2 syslog-logger-1.6.8 dep-selector-libgecode-1.0.2 net-ssh-gateway-1.2.0 systemu-2.6.5 diff-lcs-1.2.5 net-ssh-multi-1.2.1 thor-0.19.1 erubis-2.7.0 net-telnet-0.1.1 timers-4.0.1 faraday-0.9.1 nio4r-1.1.1 uber-s3-0.1.1 fauxhai-2.3.0 octokit-3.8.0 uuidtools-2.1.5 ffi-1.9.10 ohai-8.5.1 varia_model-0.4.0 ffi-yajl-2.2.2 plist-3.1.0 wmi-lite-1.0.0 hashie-2.1.2 pry-0.10.1 highline-1.7.3 rack-1.6.4
Buildig the server requires your git identity:
git config --global user.email "email@example.com" git config --global user.name "Your Name"
Finally I was ready to build the server proper:
cd ~/chef/chef-server/omnibus bin/omnibus build chef-server
This took a very long time. Reminiscent of Gentoo Linux builds (which took overnight to build Open Office many years ago).
$ ls -w 90 /opt/opscode/embedded/bin 2to3 find2perl pg_basebackup resty a2p fz pgbench ri appbundler gem pg_config rspec berks generate_field_data pg_controldata ruby bundle genhash pg_ctl run_erl bundler h2ph pg_dump runit bunzip2 h2xs pg_dumpall runit-init bzcat htmldiff pg_receivexlog runsv bzcmp httpclient pg_resetxlog runsvchdir bzdiff idle pg_restore runsvdir bzegrep infocmp pg_standby runsvdir-start bzfgrep infotocap pg_test_fsync s2p bzgrep initdb pg_test_timing sequel bzip2 instmodsh pg_upgrade serverspec-init bzip2recover irb piconv shasum bzless json_pp pkg-config shell-quote bzmore knife pl2pm smtpd.py c2ph ldiff pod2html splain captoinfo libnetcfg pod2latex sqitch chef-apply libtool pod2man sv chef-client libtoolize pod2text svlogd chef-shell lzcat pod2usage tabs chef-solo lzcmp podchecker testgdbm chef-zero lzdiff podselect testrb chpst lzegrep postgres thor clear lzfgrep postmaster tic clusterdb lzgrep prove toe coderay lzless pry to_erl config_data lzma psed tput corelist lzmadec psql tset cpan lzmainfo pstruct typer cpan2dist lzmore ptar unlzma cpanm makedepend ptardiff unxz cpanp minitar ptargrep utmpset cpanp-run-perl moose-outdated pydoc uuid createdb mzn-gecode python uuid-config createlang ncurses5-config python2 vacuumdb createuser ncursesw5-config python2.7 vacuumlo c_rehash node python2.7-config xml2-config ct_run nokogiri python2-config xmlcatalog dbilogstrip npm python-config xmllint dbiprof ohai rabbitmqctl xslt-config dbiproxy oid2name rabbitmq-env xsltproc dialyzer omnibus-ctl rabbitmq-server xsubpp dropdb openssl rackup xz droplang package-stash-conflicts rake xzcat dropuser pcre-config rdoc xzcmp ecpg pcregrep rebar xzdec enc2xs pcretest redis-benchmark xzdiff epmd perl redis-check-aof xzegrep erb perl5.18.1 redis-check-dump xzfgrep erl perlbug redis-cli xzgrep erlc perldoc redis-sentinel xzless erubis perlivp redis-server xzmore escript perlthanks reindexdb zipdetails ffi-yajl-bench pg_archivecleanup reset
It finally started compiling the Erlang code, which it retried three times, with increasingly long pauses in between
[Builder: oc_erchef] W | [1/3] Failed to execute command. Retrying in 10 seconds... ... [Builder: oc_erchef] W | [2/3] Failed to execute command. Retrying in 20 seconds... ... [Builder: oc_erchef] W | [3/3] Failed to execute command. Retrying in 40 seconds... ... ===> Compiling oc_chef_authz ===> Compiling .../oc_erchef/.../oc_chef_authz/src/oc_chef_policy.erl failed .../oc_erchef/.../oc_chef_authz/src/oc_chef_policy.erl:26: behaviour chef_object undefined
chef-server/src/oc_erchef/apps/oc_chef_authz needs to be built in an environment with the
chef-server/src/oc_erchef/apps/chef_objects/src/chef_object.erl module available, to provide the
chef_object behaviour. I may figure this out sometime soon, but I won't be working much this weekend.
Submitted by Bill St. Clair on Fri, 02 Oct 2015 10:41:59 GMT
I interviewed with Basho last week, hoping for a job writing Erlang for their Riak distributed database. They didn't hire me, at least not yet, but their system is open source, so I decided one way to convince them I'm capable, and to ground the Erlang study I've been doing for the last couple of months, is to submit some pull requests. They ship binaries built against their fork of version R16B03 of Erlang, and Riak builds in 17.5, but you get build errors for 18.x, so I decided to fix those build errors.
The build errors are mostly due to
erlang:now() being deprecated and most of the Riak source set to error on warnings, so the fixes are easy. You either change to
os:now(), which returns the same form of result, but doesn't guarantee a unique value each time, or you call
erlang:timestamp(), falling back to
erlang:now() if it isn't defined (which it isn't before 18.x).
time_compat:timestamp() from time_compat.erl does exactly that. Easy fixes, but testing needed in the Basho versions of Erlang 16, 17, and 18 before a pull request is ready for submission.
And that's where Yurii Rashkovskii's
kerl comes in. It makes it very easy to download source, build, install, and choose from a bunch of different versions of Erlang. Now would be a good time to click Yurii's photo below and read the README for
kerl. I'll go over some of the details, but only what I've used so far.
On a new Ubuntu 14.04 machine, I had to install some packages to get Erlang to build:
$ sudo apt-get install build-essential git libncurses5-dev # not strictly necessary, but enables those Erlang applications $ sudo apt-get install libssl-dev libssh-dev
You'll also need to install Sun's Java SDK, if you need the
jinterface application and
java_src from the
ic application (necessary, or at least desireable for Riak).
$ mkdir -p erlang $ cd ~/erlang $ git clone firstname.lastname@example.org:yrashk/kerl.git $ ln -s ~/erlang/kerl/kerl ~/bin/
And here's how I know it worked:
$ kerl kerl: build and install Erlang/OTP usage: .../bin/kerl <command> [options ...]
Command to be executed Valid commands are: build Build specified release or git repository install Install the specified release at the given location ...
To find which releases
kerl knows about:
$ kerl list usage: .../bin/kerl list
$ kerl list releases R10B-0 ... R16B03 ... 17.4 17.5 18.0 18.1 Run ".../bin/kerl update releases" to update this list from erlang.org
Download and install one of them:
$ kerl build R16B03 erlang-r16b03 Downloading otp_src_R16B03.tar.gz to .../.kerl/archives ... Verifying archive checksum... Checksum verified (c330150913556a0fe73e57a441cb6375) Extracting source code Building Erlang/OTP R16B03 (erlang-r16b03), please wait... Erlang/OTP R16B03 (erlang-r16b03) has been successfully built
kerl doesn't print out much while building. To see progress, you can look at the log file (from another shell window, the log file is deleted when the build completes successfully):
$ cd ~/.kerl/builds/erlang-r16b03/ $ tail -f otp_build_R16B03.log
The Erlang configure script omits applications for which you have no library support on your system. Here's how to find which ones the build will skip:
$ cd ~/.kerl/builds/erlang-r16b03/ $ find . -name SKIP ./otp_src_R16B03/lib/odbc/SKIP
Install the build you just made in a directory of your choosing:
$ mkdir -p ~/erlang/release $ kerl install erlang-r16b03 ~/erlang/release/r16b03 Installing Erlang/OTP R16B03 (erlang-r16b03) in .../erlang/release/r16b03... You can activate this installation running the following command: . .../erlang/release/activate Later on, you can leave the installation typing: kerl_deactivate
Start using the new installation:
$ kerl list installations erlang-17.5 .../erlang/release/17.5 erlang-18.1 .../erlang/release/18.1 erlang-r16b03 .../erlang/release/r16b03 $ KERL_ENABLE_PROMPT=y . ~/erlang/release/r16b03/activate (erlang-r16b03)$ which erl .../erlang/release/r16b03/bin/erl (erlang-r16b03)$ erl Eshell V5.10.4 (abort with ^G) 1>halt(). (erlang-r16b03)$ kerl_deactivate $ which erl $
I rarely use
kerl_deactivate. Instead I just activate a different version.
Riak builds require Basho's special patched Erlang versions. You can use
kerl for them, too. Below I build the
basho-otp-16 branch of Basho's OTP fork:
$ kerl build git usage: .../bin/kerl build git <git_url> <git_version> <build_name> $ kerl build git email@example.com:basho/otp.git basho-otp-16 basho-r16 Checking Erlang/OTP git repository from firstname.lastname@example.org:basho/otp.git... Building Erlang/OTP basho-r16 from git, please wait... Erlang/OTP basho-r16 from git has been successfully built $ kerl install basho-r16 ~/erlang/release/basho-r16 Installing Erlang/OTP git (basho-r16) in .../erlang/release/basho-r16... ... $ KERL_ENABLE_PROMPT=y . ~/erlang/release/basho-r16/activate (basho-r16)billstclair@Gabriel:~/erlang/release$ erl Eshell V5.10.3 (abort with ^G) 1> halt(). (basho-r16)$ kerl_deactivate $
KERL_ENABLE_PROMPT=y is responsible for the
(basho-r16) prefix on the shell prompt. If you don't like that, don't use it. I have a shell script that automates most of the activation. I'll document it soon, along with some of my other useful scripts, with code on GitHub.
$ . kerl_activate basho-r16 (basho-r16)$ kerl_deactivate $
kerl makes it easy to test my Riak patches in the three most recent Erlang versions. It has more features, but you already clicked on Yurii's photo, right? So you know about them.
Submitted by Bill St. Clair on Fri, 02 Oct 2015 09:10:46 GMT
I got an SSL certificate from startssl.com. I use them because they're free and recognized by major browsers. I'm looking forward to the advent of Let's Encrypt, which is due in Q4 of 2015. Finally, free, automated SSL certificates, recognized by all major browsers.
Strangely, StartSSL denied my initial request for a certificate. Why? I had a donation button prominently visible at the top of the right-hand column. So I removed it, re-applied, and they approved. Go figure. I hadn't gotten any donations anyway. But if you're so moved, PayPal to the email at the bottom of the page.
The bottom line is that you can now reach this site via https://lisplog.org/.
Submitted by Bill St. Clair on Thu, 01 Oct 2015 15:46:56 GMT
If you have a Mac, and lots of lispers do, you probably know that yesterday Apple released Mac OS X El Capitan, version 10.11. I'm usually more eager than is healthy to jump on new releases. Not so eager that I feel I have to get the developer releases, or even the public betas, but when a golden master comes out the door, for my Mac or my iPhone, I want it.
But I know to be at least a little cautious. So I followed MacWorld's advice about How to Make a Bootable OS X 10.0 El Capitan Installer Drive, and re-partitioned one of my mostly-empty external drives into three, the existing stuff, a partition to install El Capitan, and a tiny 10 gigabyte partition for the installer itself.
cd /Applications/Install\ OS\ X\ El\ Capitan.app/Contents/Resources
sudo ./createinstallmedia --volume /Volumes/installer \
--applicationpath /Applications/Install\ OS\ X\ El\ Capitan.app
I tried to set the new install partition as my startup drive in System Preferences, but it wasn't a choice. So I shut down, restarted, and held down the <option> key. It WAS one of the choices there. My external drive is a slow USB-2 drive, so I didn't expect lighting, but it said it would take 8 minutes. Yow! Well, that 8 minutes was actually 15, the last 7 of which at it saying "About a second left". Then it took another 24 minutes, pretty accurately predicted, to finish the install.
I moved Emacs, XCode, CCL, and Erlang, from my 10.10 drive to 10.11 and downloaded Karabiner from the web (so that I could use the <fn> key on my Apple bluetooth keyboard as <control>). I ran XCode, so it would do its initialization stuff. Emacs & Erlang just worked. CCL started, and recompiled its lisp code, but failed at rebuilding its kernel. Something about a missing include file. I reinstalled the command-line tools, and it worked.
At this point I'd had enough. I could have tried other things, but I decided to install on my main system. That took just a few minutes to complete the setup, then an hour and a quarter after the reboot to do the upgrade, estimated at 31 minutes, which rose quickly to 37 and fell slowly. I really wish Apple's installer gave a little more feedback about what it's doing. It sometimes seems to be stuck. WIth the old installer, you could open a progress window and see somewhat frightening messages, but at least some progress.
The new system came up. I had to re-enter some passwords for Mail, and I had to reinstall the command-line tools, but nearly everything I use works: Emacs, CCL, Erlang (and rebuilding Erlang with kerl), Mailplane, Firefox, VMWare Fusion (6.0.6), X11 & Gnumeric, Skype, Ignite (midi keyboard player), Adium (client for IRC, Jabber, and just about every other known chat system), Letterspace (notes app that syncs with iPhone), Nimbus (client for IRCCloud.com) and of Course Apple's own Messages, Safari, Contacts, and Calendars.
Also Mail now organizes conversations by actual conversations, not just by matching subject lines. I depended on the former behavior for my Fail2Ban notifications on a web site I adminster. It let me easily tell if a particular IP was causing enough problems to ban it permanently instead of just Fail2Ban's ten minutes at a time. I requested the old behavior as a preference here.
So what's new? Nothing much that I've noticed so far. Maybe it's a little snappier.
Submitted by Bill St. Clair on Thu, 01 Oct 2015 14:49:28 GMT
I haven't posted much to this site in a long time. I envisioned it as a place to document and announce new features for my Lisp blogging software: Lisplog. But I haven't made any changes to it in a long time, and I never did the promised documentation. It still needs to automate the setup of the settings and .htaccess files. No telling when or even if I'll get that done. Nobody has been beating down my door for it.
So, I'm going to change the way I think about this site. I'll now use it for blogging about Lisp and functional languages in general, including a lot about my new interest: Erlang. I already have at least three posts in mind: "OS X El Capitan Startup Transients", "Useful Shell Scripts", and "Kerl: Erlang Source Builds and Configuration Made Easy".
Watch this space. note the "XML" link on the top of the right column, and read the "news" feed, which already aggregates some lisp-oriented sites, and to which I'll soon be adding Erlang-oriented sites. If you have recommendations for sites to add to the list, either comment on this post, or send email to the address at the bottom of the page.
Submitted by Bill St. Clair on Mon, 31 Aug 2015 14:21:50 GMT
I have been laid off from my dream telecommuting job with Clozure Associates, so I'm looking for work.
My resumé is here.
Submitted by Bill St. Clair on Sat, 04 May 2013 15:03:52 GMT
Today I went live with a new feature for the RSS aggregator, a feed list. It's accessible via the feeds link on the right side of the news index page (in the "Navigation" section of the right column). I'll add OPML and a lisp list of feed URLs soon.
Submitted by Bill St. Clair on Sat, 06 Apr 2013 14:23:32 GMT
I compiled a list of a few Lisp-related RSS feeds to aggregate, and pointed the new code at them. Accessible via the "news" link in the "Navigation" section of the right column, which links to lisplog.org/aggregator. Here's what I found in a quick search:
I'll add more as I come across them. Comment on this post to recommend your favorites.
Submitted by Bill St. Clair on Sat, 06 Apr 2013 07:35:01 GMT
After running it on my iMac for a couple of nights, I made Lisplog's new RSS aggregator live at http://billstclair.com/blog/aggregator/ . There are still lots of features I could add, but it's now functional. I'll put together a Lisp blog aggregator here real soon now.