Lisplog

Blogging in Lisp

Search

Feed Aggregator

Rendered on Mon, 27 Feb 2017 13:32:12 GMT  newer latest older 
Next udpate: Mon, 27 Feb 2017 14:00:00 GMT feeds

Nicolas Hafner: Portacle - Adventures in Cross-Platform Deployment - Confession 72

via Planet Lisp by on Wed, 22 Feb 2017 18:58:29 GMT

header
As announced in my previous post, I've decided to do a write-up that illustrates my adventures in developing Portacle, a cross-platform, portable, installer-less, self-contained Common Lisp development environment.

The reason why I started with portacle in the first place is that, some day, probably in the distant future, I want to write a long series of long articles that should eventually accumulate into a book that introduces complete novices and interested people to programming in Common Lisp. As a prerequisite for that, I deem it absolutely necessary that the readers have an easy way to try out examples. Forcing them to perform a strange setup process that has absolutely nothing to do with the rest of the book and is likely to be error-prone in multiple ways is completely out of the question to me. They should be able to download an archive, extract it, and then just click on an icon to get a fully-featured IDE with which they can start out and play around with.

So, right off the bat this sets a few constraints that are rather hard to deal with.

  • It needs to be cross-platform. I can't constrain the users to a certain operating system. Requiring them to set up a virtual machine would be madness.
  • It needs to be cross-distribution. I can't know which distribution of Linux, or which version of it users are going to have. Requiring a specific one or a specific version would, too, be madness.
  • It needs to be self-contained and should not poison the rest of the environment unless necessary or intended. This is necessary for it to be truly portable and work from a single folder.
  • It needs to be reproducible and upgradable in order to be future-proof. This means I cannot simply assemble a package by hand once and call it a day once it works. It needs to be automatically buildable.
  • It needs to be able to run on all platforms simultaneously as a single distribution. Otherwise, you would not be able to put it onto a USB stick and use it from any machine.
  • It needs to be fully-featured enough to provide for both quick experiments and larger projects. As such, it will need to package a few different components and make them all work alongside, with the above constraints on top.

Since I want the eventual article series to be for absolute beginners as well, I also had some serious concerns about the usability aspects. It shouldn't be too confusing or hindering to use. Despite this, I still settled for Emacs+SLIME for the IDE, simply because there isn't really any alternative out there that is free and supports Lisp to a sufficient degree. I'm still not entirely set on the way the Emacs customisation is currently laid out, though. I might have to make more or less severe changes to make it easier for people to get started with. But, that's something for another time.

Now, considering the last constraint listed above, I decided on the following packages to be included in Portacle:

  • Emacs, for the primary editor.
  • Emacs customisation files, since the default is horrid.
  • SBCL, for a capable Lisp implementation.
  • ASDF, for building and system management.
  • Quicklisp, for package management and library access.
  • Git, for version control and project sharing.

ASDF was included specifically so that I would not have to rely on the implementation updating its internally shipped version and could instead deliver one that is possibly ahead of it, and thus less buggy/more feature-rich.

The first goal was to figure out how to build everything on OS X, Linux, and Windows. This alone took its fair share of experimentation and scouring as I had to look for the right combination of feature flags and build flags to make things work minimally.

In order to make this process work, I've developed a series of bash scripts that take care of downloading the source files of the various parts, configuring them, building them, and installing them in the proper locations in the resulting Portacle distribution tree. I settled on the following layout:

portacle      -- Base Portacle directory
  build       -- Build scripts and artefacts
    $package  -- Build artefacts for the package
  config      -- Configuration files and user files
  projects    -- User projects directory
  all         -- Cross-platform packages and resources
    emacsd    -- Emacs configuration package
    quicklisp -- Quicklisp installation
  $platform   -- Platform-specific packages and resources
    $package  -- Installed package files
    lib       -- Shared library files
    bin       -- Shared executable files

The layout used to be different at first, namely the $platform and $package directories used to be flipped around in the hierarchy. That proved to be less than ideal, however. I won't go into the details as to why. Suffice to say that complications that cropped up along the way ended up poisoning the hierarchy. Having a platform directory for all the specific files means you can create a Portacle distribution that works on all platforms simultaneously too.

Now, in order to properly illustrate the problems that cropped up, I'm going to talk about the development process for each platform separately. Pretty much every one of them had unique problems and complications that lead to a lot of wasted time and questions about the universe.

Linux

Linux was the first platform I started developing for, and ironically enough the last one I finished for good. The first phase of just getting everything built was comparatively painless. Things just worked.

However, as soon as I tried to run a complete installation on another Linux setup, things just segfaulted left and right. No wonder, too. After all, every component has shared library dependencies. Those are going to have different versions, and possibly even different names on different distributions. The first idea to deal with this mess was to simply copy every shared library the applications touched to a directory and set LD_LIBRARY_PATH in a wrapper script before the respective application was launched.

In order to do this, I extended the build scripts with tools that would statically analyse all the executables that were shipped and recursively trace a full list of shared library dependencies. It would then copy them all over to the shared library files directory.

After doing this, things worked a bit better. But only a tiny bit. SBCL worked now, at least. Everything else still crashed. Unfortunately however, SBCL also only worked on my particular system. Much later I would find out that on others it would still fail. The root of the problem lay in the fact that LD_LIBRARY_PATH does not cover everything. The system will still pick some shared libraries from elsewhere. Particularly, libc and so forth.

So I scoured the web in search of a solution that would somehow let me constrain where linux looked for shared libraries. After a long while, I finally found something: ld-linux.so. This file, which lives on every Linux system, is responsible for setting up the shared libraries an application depends on, and then running the application. It can be called directly, and it accepts a library-path argument, with which I can force it to look in a particular place first. So I changed the wrappers around to also start things with ld-linux.so. Doing this allowed me to get Emacs working.

However, now SBCL didn't work anymore. For some reason that is still completely beyond me, if I try to run SBCL under ld-linux.so, it acts as if the argument that is the SBCL binary path was simply not there. You can try it for yourself: /lib64/ld-linux.so $(which sbcl) will show you the help text, as if that argument was just not there. If you pass it the path twice it does something, but it won't work right either. Fortunately, since SBCL doesn't have many dependencies to speak of, I've been able to get by without wrapping it in ld-linux.so. I can't wait for the day for that to break as well, though.

Anyway, back to where I was. Emacs now worked. Kind of. Running other binaries with it did not, because the LD_LIBRARY_PATH would poison their loading. Regular binaries on the system somewhere would not execute. So I had to do away with that. Thanks to the ld-linux.so trick though, it wasn't really needed, at least for Emacs. For Git, the situation was worse. Git does some weird stuff where it has a multitude of git-* binaries that each perform certain tasks. Some of those tasks call other Git binaries. Some of them call bash or things like that. I couldn't set LD_LIBRARY_PATH because that would crash bash and others, and I couldn't use ld-linux.so because that doesn't work when a new process is created. Back to Google, then.

After even more endless searching I finally found a solution. Using LD_PRELOAD I could inject a shared library to be loaded before any other. This library could then replace the exec* family of libc system functions and ensure that the process you wanted to call was actually called under ld-linux.so. You can do this kind of symbol replacement by using libdl, fetching the function locations of the functions you want to replace using dlsym in the library's init function, and then defining your own versions that call out to the original functions. You can find a few interesting blog articles out there that use LD_PRELOAD and this kind of technique for malicious purposes. This culminated in something I called ld-wrap.so. Getting all the exec* calls to work again was its own adventure. On that path I discovered that execv will actually call out to a shell if it finds a shell file, and, even worse than that, excecvp* will actually call a shell any time if they can't seem to execute the file regularly. I think it's pretty insane for a "low-level system call" to potentially invoke a shell to run a script file, rather than "just" executing an ELF binary. In fact, there is no way to "just" execute a binary without implementing your own system calls directly. That's bonkers.

Anyway, getting ld-wrap.so to work right involved multiple iterations, the last of which finally got Git working proper. I had to incorporate tests to check whether the binary was in the Portacle distribution, as well as checks to see whether the binary was static or not. Apparently launching a static binary under ld-linux.so just results in a segfault. How very C-like. Anyway, it has not been the least bit of fun to go through this and pretty much every step of getting all of this worked out cost me weeks.

Aside from the SBCL problem mentioned above, there's an outstanding issue regarding the usage of fonts in Emacs. Since there's absolutely no guarantee for any particular font to exist on the system, and since I'd like to provide for some nice-looking default, I need to ship and install fonts automatically. I haven't gotten that to work quite yet, but I've been very disappointed to find that Emacs apparently has no support for loading a font from a File, and that Linux and OS X don't really support "temporarily" loading a font either. You have to install it into a user directory for it to be visible, so I have to touch things outside of the Portacle distribution folder.

Finally, apparently if you try to compile Emacs on some Linux systems, it will fail to run under ld-linux.so and just crashes immediately with a "Memory Exhausted" warning. I have no clue what that is about and haven't received any feedback at all from the Emacs mailing lists either. So far this is not a problem, since the virtual system I build on works, but it is a major hassle because it means that building a distribution is out of the question for a lot of people. You can find out more about this bug on the issue tracker.

Windows

Windows has been an interesting system to work with. On one hand it has given me a lot of problems, but on the other it has also avoided a lot of them. The biggest pleasure was that shared library deployment "just worked". No need for complicated ld-linux shenanigans, Windows is just consistent and faithful to loading the first library that it can find on its PATH. Given that most of the libraries I need are already either provided by the Windows system or from the self-contained MSYS installation, there really haven't been any issues with getting things running on a deployed system.

However, it makes up for this in other areas.

While Emacs and SBCL have been very easy to get compiled and running, Git has once again proven to be a problem child. First, Git insists on keeping around git-* prefixed hard-links to its binary because apparently a lot of things both internal and external still directly reference those. Hard links are difficult to ship because most archiving systems don't support them, making the resulting archive humongous. The current size of ~80 megabytes is already a lot by my measures, but the hard link issues exploded it to well over 100. Thus I had to go hunt for an archiver that allowed both self-extracting SFX executables -after all I couldn't ask people to install an archiver first- and was able to compress well and handle hard links. 7zip was up to the task, but it required complicating the deployment quite a bit in order to get the SFX working.

Next, Git is primarily a system written for Linux. As such, it has some "interesting" ideas about what the filesystem really looks like. I'm still not entirely sure how the path translation happens, but apparently Git interprets the root of the filesystem somewhere relative to its application location. This is fortunate for me, but took a long while to figure out. It is fortunate, because Git needs CURL to work, and CURL needs a certificate file to be able to do HTTPS. Naturally, Windows does not provide this by itself, so I have to do it. Git does allow you to set the path of the certificate file through a configuration variable, but it was one hell of a journey to get the entire setup working right. I'll try to explain why.

Because of - or rather thanks to Git's weird interpretation of the filesystem root, I can create a "fake" Git system configuration. Usually, Git looks up /etc/gitconfig as part of the configuration files. Now, since the root is sort of relative to the Git binary, I can create an etc subdirectory with the configuration file in the Git platform directory. This allows me to specify the certificate file path without having to disturb any of the other platforms. Then, thanks again to this root interpretation I can specify the path to the certificate file as an absolute path within the configuration file. Since the Git interpreted root moves with the Git binary, it becomes effectively relative. Naturally this would not work on Linux or OS X, but thankfully there I don't need to resort to such tricks.

Finally, if I try to compile Git without gettext on Windows, it fails to run properly and just exits with vsnprintf is broken. I did find some issues related to vsnprintf on Google, but nothing conclusive. Either way, if I do compile with gettext it seems to work fine. It doesn't work with gettext on OS X though, so I can't just enable it everywhere either.

Last but not least, Windows is primarily responsible for me not shipping a spell checker system. I really wanted to include that, as a spell checker is a really great thing to have when you're writing lots of comments and documentation, or just using Emacs for prose. However, I was simply not able to get ispell, aspell, or hunspell to compile and run no matter how much I tried. I was also unable to find any other compatible alternatives to those three that could be used as a replacement. If anyone else knows of a solution that I can compile successfully, and for which I can generate working dictionaries, I'd be very grateful.

Actually, I suppose it's also worth mentioning that for a while, before I wrote the launcher, I was trying to use batch scripts to launch things. Please, don't ever try to do that. Batch files are unbelievably cryptic to read and a huge pain to work with. There's no easy way to pass the arglist to another program for example, as it'll just reinterpret spaces within an argument as an argument separator. If you can, use PowerShell, which is supposedly much better, or just write a proper launcher application that does that kind of logic.

Mac OS X

Finally, OS X. This is a bit of a problematic system for me because Apple has, for reasons beyond me, decided to make it as difficult as possible to test for. Since I needed Portacle to work on multiple versions of OS X if possible, and even beyond that just ensure that it works outside of a development environment, I had to get my hands on virtual machines somehow. I first bought a MacBook Air just for this purpose -that's a thousand dollars wasted in the wind- only to realise that trying to run any kind of Virtual Machine on it is futile because it's just unbearably slow. Fortunately there are ways to get VMWare Workstation on Linux to run OS X as well if you use specially prepared images and some nefarious tools to unlock OS X as an option. However, only versions 10.11+ seem to run at any usable speed. 10.10 is so slow that you can pretty much forget trying to use it for anything.

Anyway, even just setting up a suitable build and test environment proved to be a major hassle. Thanks to Homebrew and MacPorts the building aspect wasn't too bad, though there too I've found weird inconsistencies like the aforementioned gettext problem. Aside from the compilation though, it really bothers me a lot that the OS X coreutils lack so many useful features. The build scripts have several special cases just for OS X because some Unix utility lacks some option that I need to get it done succinctly or efficiently.

Aside from the virtualisation thing, Apple also seems to try their hardest to prevent anyone from developing software for their system without also paying them a substantial fee every year. If you want to launch Portacle as a normal user, you just get a "security" error message that blocks the application from running. To get it to launch, you have to start the systems settings application, navigate to the security tab, unlock it, then click on "run this application" in order to finally run it. They completely removed the option to allow foreign apps by default, too, so there's no to me visible option to make it shut up. Windows has a security popup similar to that too, but at least you can immediately tell it to launch it, rather than having to waste multiple minutes in menus to do so.

In addition to the "security" popup, Apple has also recently decided to activate a feature that makes it virtually impossible for me to properly ship additional libraries, or versions of the libraries that Portacle requires, thus forever constraining the possible versions it can run on. OS X will now automatically clear out DYLD_LIBRARY_PATH whenever you execute a new application. You can only disable this lunatic behaviour by booting into recovery mode and changing a security option- definitely not something I can ask people to do just to use Portacle. Thus, it seems it is impossible for me to, in the long term, cover a wider version scheme than a single one. This is very annoying, especially given that lots of people seem to stop upgrading now that Apple is screwing up ever more with every new version.

Ah well.

Future Outlook

That about sums up most of the major issues that I can remember. You can find out more stories of me going borderline insane if you browse around in the issues or the commit history.

Now that most of the platform problems are finished, Portacle is almost done. There are a few more things left to do, however. The Emacs configuration as it is right now is somewhat passable, but I'd like to add and change a few more things to make it more comfortable, especially for beginners. I don't want to change too much either though, as that would make it hard for beginners to find help to a specific issue online.

Otherwise, I'd also like to work a bunch more on the included documentation and help file. As it stands it gives an overview over almost everything one needs to know to get started, but I haven't had it run by anyone yet, so I can't really give any estimates as to whether it's comprehensible, readable, or even useful in the first place.

I might write about Portacle again another time, hopefully when it has finally reached something I can call "version 1.0". Until then, I'll try to write some more articles about other subjects.

Common Lisp Tips: The return value of read-sequence

via Planet Lisp by on Thu, 16 Feb 2017 13:59:06 GMT

I’ve seen this idiom a few times when working with a binary stream:

(let ((elements-read (read-sequence buffer stream)))
  ...)

While the return value is the number of elements read in that situation, it’s a useful coincidence in a common case. The actual return value of read-sequence is:

the index of the first element of sequence that was not updated

Therefore, treating the return value as the number of elements read is completely wrong if you use the start option to read-sequence.

For example, if stream has one or more elements available to read:

(read-sequence buffer stream :start 42 :end 43)
=> 43

The actual number of elements read is 1, not 43.

The return value of read-sequence

via Common Lisp Tips by on Thu, 16 Feb 2017 13:59:06 GMT

I’ve seen this idiom a few times when working with a binary stream:

(let ((elements-read (read-sequence buffer stream)))
  ...)

While the return value is the number of elements read in that situation, it’s a useful coincidence in a common case. The actual return value of read-sequence is:

the index of the first element of sequence that was not updated

Therefore, treating the return value as the number of elements read is completely wrong if you use the start option to read-sequence.

For example, if stream has one or more elements available to read:

(read-sequence buffer stream :start 42 :end 43)
=> 43

The actual number of elements read is 1, not 43.

Elm Digital Ocean Interface

via Lisplog by on Sat, 11 Feb 2017 01:01:19 GMT

I have built an Elm interface to a subset of the Digital Ocean HTTP API.

http://package.elm-lang.org/packages/billstclair/elm-digital-ocean/latest
https://www.digitalocean.com
https://lisplog.org/elm-do

Nicolas Hafner: It's Been So Long - Confession 71

via Planet Lisp by on Thu, 09 Feb 2017 23:56:12 GMT

header
It's been a good while since I last wrote an entry. I didn't write anything all January for a multitude of reasons, at the forefront being that it was exam season again. That's over with now, fortunately, so I do have some more time on my hands to goof off. Unfortunately though, I only have one more week left of this precious "free" time before university strikes me in the back again. I best use it wisely.

The first week of my holidays now I've spent on a heavy reworking of Portacle, a portable, multiplatform, upgradable, install-less Common Lisp development environment project. I've only just now finally got it to work properly on all three platforms and differing versions thereof. It has been an unbelievable struggle, as every platform posed unique problems and nasty edge-cases that were hard to catch without testing everything on a multitude of virtual machines all the time. If you look through the commit history you can find plentyful examples of the borderline insanity that I reached at points. I might write an in-depth article about my adventures another time.

During the exam session itself I was mostly occupied with working on Lichat, my own light-weight chat protocol. It has the nice properties that it is very uniform and rather simple to implement a client for. Among the nice aspects of the protocol are that everything goes over a channel, even private messages, that each update is associated with an ID and failures or responses can be tracked even if they happen to be out of order, and that connection multiplexing is provided directly by the protocol, so you can log into the same user account from multiple machines or instances simultaneously. I've written both a TCP and a WebSockets server, and a CL and JS client for it as well, so you can easily set up your own distributions of it. These CL and JS clients also have respective user interfaces as well, namely LionChat and Lichat-JS. Lionchat might turn into a generic chat client at some point in the future.

I've also continued work on Maiden, and finished a Lichat client for it as well. It's now running as a bot alongside the previous Colleen and I'm intending on switching over fully soon. The only thing that I haven't switched over yet is the chat logging, as I haven't had time to test that extensively yet. I've also been using its Text To Speech module to provide some fun interactivity for my video game streams. Before I can unleash Maiden onto Quicklisp I'll need to heavily document everything though, as it is quite a lot more complex -and capable- than Colleen was.

Then, I've also taken the time to switch over TyNET to the new Radiance 1.0. Radiance is now complete and ready for release as far as I'm aware. I've been holding off on releasing it for good as I've been waiting for some feedback on its tutorial from a couple of friends of mine. Once they get ahead with that I'll type up a good, long entry to announce everything. If my paper for ELS 2017 gets accepted, I will even be presenting it at the symposium in April. Speaking of the symposium- I've been working on a new website for it on and off, to which it will be switching in a while. You can get a preview peek at it here.

For Christmas I gave my father a Raspberry Pi 3 and installed Emacs and CCL onto it. He's been using it, and a DAQ board, to interface with a Korg synthesizer, since he's interested in making music on a more low, analog level. To do this I've been trying to teach Lisp to him since Christmas. While it's been a hard road so far, he's slowly getting into it and making advances. Given that he's been a Fortran programmer for almost all his life, with probably close to forty years of experience or more in it, I'm actually surprised at how well he's been doing with something that probably seems very different and alien to what he's used to. As part of this, though, I've had to write a few libraries, namely cl-gpio to interface with the Pi's GPIO pins, cl-spidev to use the serial port interface, cl-k8055 for the DAQ kit he's been using, and pi-plates for the pi-plates extension boards. The last library isn't done yet as I haven't had the time to test it yet, but the other three should work just fine.

So, I've been slowly, but surely, working off my long stack of projects that need to be completed someday. With Maiden out of the way soon too, I'll finally be able to get back to two of the remaining major projects: Trial, my game engine, and Parasol, my painting application. I'm still undecided as to which I'll tackle first, though. If you have a preference, let me know!

Finally, aside from programming work, I've gotten back into the swing of drawing. Before I was constantly feeling down and unmotivated about it. Everything just seemed to suck and I could never come up with ideas for things that I actually wanted to draw. I'm still not exploding with ideas now, but at least I've found something that inspires me a lot. Specifically, I was reading through Yokohama Shopping Trip and it was filled with exactly the kind of scenery-heavy, romantic drawings that I always wanted to make. "Romantic" referring to the Romanticism art movement. For the past few days then I've been trying to get a scenery drawing done every evening. You can see them on my tumblr. I still have a very long way to go and I'm painfully aware of all the things I do badly at or can't figure out how to draw properly, but I just have to keep on keeping on. It's too late to throw it all to the wind now.

In the near future I'd like to focus some more on writing things again, and perhaps also on actually reading some of the books that I've started accumulating over the years. But then, I'd also like to continue streaming video games, as I've been enjoying that quite a lot. Ah, there's just so much I'd like to do. If only someone were to figure out the age-old problem of finite capacity. Or in the very least: if only I didn't have to worry about earning any money to make a living, I could just merrily focus on being as productive as I can be instead of having to focus on what's marketable. Ah well, you can't have everything, eh?

So, in the more immediate future you can, if you want to, look forward to a full release announcement of Portacle, Radiance, and Maiden, more drawings, and hopefully more articles and stories as well. No promises, though. I'm not very good at keeping to deadlines for projects.

Zach Beane: Gary Byers is stepping back from CCL and Clozure

via Planet Lisp by on Wed, 08 Feb 2017 14:07:00 GMT

Here’s a message from Andrew Shalit to openmcl-devel:

To the CCL Community -

As many of you know, Gary Byers has been on a leave of absence from his Clozure CL work to attend to his health. I'm writing to let you know that, unfortunately, this leave will be permanent. Gary has resigned from his position at Clozure and will no longer be the lead developer of Clozure CL.

Over the last 30+ years Gary has made a tremendous contribution to the Common Lisp programming community, first through his work on Macintosh Common Lisp, and then through the creation of CCL. Gary has always been ready to share his deep knowledge of the Common Lisp standard and the issues attending its implementation. I don't know anyone whose understanding of the details of Common Lisp could match Gary's. His many contributions will be missed.

The work on CCL will continue, though. In Gary's absence, Matthew Emerson will be stepping up as the primary maintainer of CCL. Matthew is moving CCL to GitHub, and he hopes that others will pitch in to continue the development of this great software platform. Clozure Associates will also continue to support CCL, through direct contributions and through our consulting work. I'll be supporting those efforts, and I hope you'll consider supporting them as well.

Yours,

Andrew Shalit
Clozure Associates

Gary Byers is stepping back from CCL and Clozure

via Zach Beane Common Lisp by on Wed, 08 Feb 2017 14:07:00 GMT

Here’s a message from Andrew Shalit to openmcl-devel:

To the CCL Community -

As many of you know, Gary Byers has been on a leave of absence from his Clozure CL work to attend to his health. I’m writing to let you know that, unfortunately, this leave will be permanent. Gary has resigned from his position at Clozure and will no longer be the lead developer of Clozure CL.

Over the last 30+ years Gary has made a tremendous contribution to the Common Lisp programming community, first through his work on Macintosh Common Lisp, and then through the creation of CCL. Gary has always been ready to share his deep knowledge of the Common Lisp standard and the issues attending its implementation. I don’t know anyone whose understanding of the details of Common Lisp could match Gary’s. His many contributions will be missed.

The work on CCL will continue, though. In Gary’s absence, Matthew Emerson will be stepping up as the primary maintainer of CCL. Matthew is moving CCL to GitHub, and he hopes that others will pitch in to continue the development of this great software platform. Clozure Associates will also continue to support CCL, through direct contributions and through our consulting work. I’ll be supporting those efforts, and I hope you’ll consider supporting them as well.

Yours,

Andrew Shalit
Clozure Associates

Eugene Zaikonnikov: Announcing CL-VIDEO

via Planet Lisp by on Mon, 06 Feb 2017 15:00:00 GMT

I'm happy to announce CL-VIDEO, a basic AVI/MJPEG video decoder written in Common Lisp. The library leverages CL-JPEG for frame processing and CL-RIFF for container format handling.

The code has only been lightly tested on SBCL 13.x/Linux x86-64. Some sample files can be found here (the toy plane AVI) and here.

To run it, CL-JPEG version 2.7 is required. Since it's not in Quicklisp yet as of time of writing, it must be cloned into your local-projects. Then you can load cl-video-player system and run (cl-video::play <your filename.avi>).

The implementation is still very much naive, as expected from a weekend project. There is no support for indexing, and a number of edge cases of legit AVI encodings might fail. The library will decode video frames in the order they occur. No parsing of BITMAPINFOHEADER structures; the assumption is they are MJPG DIB. No audio playback, although the stream is being read and adding at least PCM/WAV playback shouldn't be too big an effort.The decoder is factored out into independent implementation in cl-video.lisp. A primitive CLX media player is included in player.lisp. Each AVI instance contains one or more (audio or video) stream record objects, which hold ring buffers of frames with read and write cursors. The interaction between the decoder and player frontend runnning in separate threads is done via mutexed access to this ring buffer. The player thread can be kicked off a callback supplied to decode method: it will be called once the header part is parsed.

Since CL-JPEG doesn't use any SIMD extensions, the performance is modest. On my 6 year old 3GHz i5 (running one core of course) it decodes 480p 25fps file without hiccups, but going beyond that would require implementing multicore support in decode. Still it might be useful as is in applications not requiring real-time decoding performance.

Fixed: Slow Macintosh File Dialogs

via Lisplog by on Tue, 07 Feb 2017 11:05:34 GMT

I have been plagued for quite a while now with a two or three second wait every time I invoke the open or save file dialog on my Macintosh (running El Sierra). I finally did a Google search and found a solution. Less than a second now! Yay!

http://osxdaily.com/2015/04/17/fix-slow-folder-populating-cloudkit-macosx/

In a shell:

cd ~/Library/Caches/CloudKit
rm CloudKitMetadata
rm CloudKitMetadata-shm
rm CloudKitMetadata-wal
killall cloudd

Quicklisp news: January 2017 Quicklisp dist update now available

via Planet Lisp by on Wed, 25 Jan 2017 13:27:00 GMT

New projects:
  • ahungry-fleece — A general utility library of convenience functions and features. — GPLv3
  • asd-generator — Automatic directory scanner/generator for .asd project files. — GPLv3
  • cl-cache-tables — A wrapper around native hash-tables to facilitate in-process caching of common lisp data structures. — MIT
  • cl-feedparser — Common Lisp universal feed parser — LLGPL
  • cl-gpio — A library for the Linux GPIO kernel module as used on hobby kits such as the Raspberry Pi — Artistic
  • cl-k8055 — Bindings to the k8055 DAQ hobby board. — Artistic
  • cl-online-learning — Online Machine Learning for Common Lisp — MIT Licence
  • cl-spidev — A library for the Linux SPIDEV kernel module as used on hobby kits such as the Raspberry Pi — Artistic
  • cl-xdg — freedesktop.org standards handling — GNU General Public License
  • cl4l — esoteric CL essentials — MIT
  • glisph — Glyph rendering engine using OpenGL shading language — MIT
  • lichat-protocol — The independent protocol part of Lichat. — Artistic
  • lichat-serverlib — Tools to help build a server using the lichat protocol. — Artistic
  • lichat-tcp-client — A simple TCP client implementation for lichat — Artistic
  • lichat-tcp-server — A simple TCP server implementation for lichat. — Artistic
  • lichat-ws-server — A simple WebSocket server implementation for lichat. — Artistic
  • lionchat — A GUI client for the Lichat protocol — Artistic
  • lisp-binary — Declare binary formats as structs and then read and write them. — GPLv3
  • omer-count — A script to assist in counting the time period between Pesach and Shavuot. — GPL V3
  • ook — A CL compiler and enviroment for literate Orangutans. — Public Domain
  • opticl-core — A library for representing and processing images — BSD
  • rate-monotonic — A periodic thread scheduler inspired by RTEMS. — GPL-v3
  • timer-wheel — A timer wheel implementation with BORDEAUX-THREADS backend. — MIT
  • translate-client — A client to online web-server translators, currently only google translate — MIT
  • trivial-escapes — C-style escape directives for Common Lisp. — Public Domain
Updated projectsacclimationalexaanaphoraarchitecture.service-providerasdf-system-connectionsasdf-vizbeastbinfixcaveman2-widgetscaveman2-widgets-bootstrapcl-anacl-ansi-termcl-arxiv-apicl-association-rulescl-autowrapcl-csvcl-cudacl-custom-hash-tablecl-dbicl-digraphcl-enumerationcl-freetype2cl-glfw3cl-gobject-introspectioncl-hamtcl-jpegcl-liballegrocl-libyamlcl-marshalcl-openglcl-opsresearchcl-permutationcl-qrencodecl-readlinecl-routescl-statsdcl-syslogcl-typesettingcl-unificationcl-waylandcl-yamlclazyclinchclmlclobbercloser-mopclssclxcroatoancurry-compose-reader-macrosdeedsdefenumdexadoresrapfare-quasiquotefare-scriptsfare-utilsfast-iofemlispfixedformat-string-buildergsllinquisitorjonathanlambda-readerlet-pluslisp-unitlocal-timelog4clmcclimmetabang-bindmitomk-string-metricsmodularizeneo4clnew-opningleoclclopticlparachuteparse-floatparser.common-rulespipingplumppostmodernpostmodernityprovepsychiqqtools-uiqueuesrestasretrospectiffrfc2388-binaryserapeumspinneretstatic-vectorsstumpwmsxqltriviatrivia.balland2006trivial-ldaptrivial-rfc-1123trivial-updateubiquitousuiopunix-optsvarjovgplotweblocks-utilsxhtmlgenzlib.

Removed projects: cl-ledger, weblocks-examples.

To get this update, use: (ql:update-dist "quicklisp")

Enjoy!

Eugene Zaikonnikov: New in CL-JPEG

via Planet Lisp by on Wed, 18 Jan 2017 15:00:00 GMT

In course of last few months there were numerous small changes introduced to CL-JPEG. None substantial enough to warrant own announcement, but taken together perhaps it's due for an update. So here we go, as of version 2.6:

  • Pre-allocated buffers in DECODE-IMAGE and DECODE-STREAM are now supported. This should help reduce consing in bulk-processing applications. The buffer can be pre-allocated based on dimensions via JPEG:ALLOCATE-BUFFER.
  • CMYK JPEG support. YCCK to CMYK conversion is now performed by decoder. To convert into 3-component RGB, use JPEG:CONVERT-CMYK-TO-RGB convenience function.
  • An option to disable colorspace conversion via :COLORSPACE-CONVERSION NIL supplied to decode functions has been added. Can be useful e.g. if one needs the luminance component. Support of the corresponding option for encoder to improve performance in transcoding applications is in the plans.
  • Small bugfixes and performance tweaks.

Zach Beane: Common Lisp in the Wild - Deploying Common Lisp Applications.

via Planet Lisp by on Wed, 18 Jan 2017 11:54:42 GMT

Common Lisp in the Wild - Deploying Common Lisp Applications.

Zach Beane: MKCL 1.1.10 is now available

via Planet Lisp by on Wed, 18 Jan 2017 11:43:15 GMT

MKCL 1.1.10 is now available

 newer latest older