Lisplog

#CommonLisp #Lisp

Usually you are not supposed to know whether a piece of Lisp code has been compiled or is being interpreted without compilation. In particular, compiling a Lisp program shouldn't change its semantics in any way, right?

See, there is one thing that surely is an exception to that rule. It is specified in a way that allows it to differ in terms of how it behaves between compiled and non-compiled code: LOAD-TIME-VALUE, which I recently used to implement STATIC-LET.

And it's not a nice thing to discover.

Rationale

CLHS 3.2.2.2 Minimal Compilation states (emphasis mine):

The first argument in a load-time-value form in source code processed by compile is evaluated at compile time; in source code processed by compile-file, the compiler arranges for it to be evaluated at load time. In either case, the result of the evaluation is remembered and used later as the value of the load-time-value form at execution time.

More, CLHS Special Operator LOAD-TIME-VALUE states (emphasis mine):

If a load-time-value expression is processed by compile-file, the compiler performs its normal semantic processing (such as macro expansion and translation into machine code) on form, but arranges for the execution of form to occur at load time in a null lexical environment, with the result of this evaluation then being treated as a literal object at run time. It is guaranteed that the evaluation of form will take place only once when the file is loaded, but the order of evaluation with respect to the evaluation of top level forms in the file is implementation-dependent.

If a load-time-value expression appears within a function compiled with compile, the form is evaluated at compile time in a null lexical environment. The result of this compile-time evaluation is treated as a literal object in the compiled code.

If a load-time-value expression is processed by eval, form is evaluated in a null lexical environment, and one value is returned. Implementations that implicitly compile (or partially compile) expressions processed by evalmight evaluate form only once, at the time this compilation is performed.

COMPILE-FILE and COMPILE have a "must", where EVAL only has a "might". This means that functions defined using EVAL, without compilation, can cause a new object to be instantiated every time, which will both call the initialization form every time the body is entered and break all code that depends on the static binding values to stay static.

So, in order to get the behavior we want (in which an object is only allocated once), the code containing the STATIC-LET form must be compiled, at which point the load-time values will either be instantiated (in case of COMPILE) or stored in the resulting FASLs to be instantiated at load time (in case of COMPILE-FILE).

Test

We can make a quick test to figure out how different Lisp implementations handle this.

            ;;; the test goes like this:

;;; let's grab two functions with the same body
;;; which uses the LOAD-TIME-VALUE trick
(defun test-function ()
  (let ((counter-var (load-time-value (cons 0 nil))))
    (symbol-macrolet ((counter (car counter-var)))
      (incf counter))))

(defun test-function-2 ()
  (let ((counter-var (load-time-value (cons 0 nil))))
    (symbol-macrolet ((counter (car counter-var)))
      (incf counter))))

;;; let's compile only the second one
;;; and leave the first one possibly uncompiled
(compile 'test-function-2)

;;; let's call each one a few times
(format t "Possibly not compiled code: ~D ~D ~D~%"
        (test-function) (test-function) (test-function))

(format t "Compiled code: ~D ~D ~D~%"
        (test-function-2) (test-function-2) (test-function-2))

          

The results divide the CL world pretty much in half, speaking numerically.

• SBCL 2.1.11

  • Possibly not compiled code: 1 2 3
  • Compiled code: 1 2 3

• CCL 1.12

  • Possibly not compiled code: 1 2 3
  • Compiled code: 1 2 3

• ECL 21.2.1

  • Possibly not compiled code: 1 2 3
  • Compiled code: 1 2 3

• Clasp current

  • Possibly not compiled code: 1 2 3
  • Compiled code: 1 2 3

• ABCL 1.8.0:

  • Possibly not compiled code: 1 1 1
  • Compiled code: 1 2 3

• CLISP 2.49.93+:

  • Possibly not compiled code: 1 1 1
  • Compiled code: 1 2 3

• ACL 10.1 Express:

  • Possibly not compiled code: 1 1 1
  • Compiled code: 1 2 3

• LW 7.1.2 Personal:

  • Possibly not compiled code: 1 1 1
  • Compiled code: 1 2 3

So, it's time to update the STATIC-LET article I wrote yesterday and add a warning to it that it's only safe to use STATIC-LET in compiled code, or on implementations which are compiler-only.

Detection

But alas! It's also possible to use this behavior to check if a piece of code has been minimally compiled. The following example signals an error only on the latter four implementations and only in code that has not been compiled.

            ;; I used the stones to destroy the stones
;; CLISP 2.49.93+

[1]> (defun foo ()
       (flet ((fn () (let ((x (load-time-value (list 0)))) (incf (car x)))))
         (declare (notinline fn))
         (when (= (fn) (fn))
           (error "STATIC-LET will not work in uncompiled code."))))
FOO

[2]> (foo)
*** - STATIC-LET will not work in uncompiled code.
The following restarts are available:
ABORT          :R1      Abort main loop

Break 1 [3]> :r1

[4]> (compile 'foo)
FOO ;
NIL ;
NIL

[5]> (foo)
NIL

          

On non-compiler-only implementations, it's possible to splice such a piece of code into a macroexpansion in order to have a check which will signal an error (or possibly do something else) on code which was not minimally compiled.

Or, in other words, I just realized that (flet ((fn () (let ((x (load-time-value (list 0)))) (incf (car x))))) (= (fn) (fn))) is a non-portable but luckily working poor man's "was this code minimally compiled" predicate.

I bet $3 that it's useless other than for the above.

So I have been working on Common Lisp Recipes, on a recipe for using global static bindings via global-vars. I was wondering if there was anything for local bindings though, something like in C:

            // Simple test in C

int test_function() {
  static int counter = 0;
  return counter++;
}

test_function(); // -> 0
test_function(); // -> 1
test_function(); // -> 2

          

And then, oh, I remembered. There was an article exploring the topic and showing a technique that had some pretty nice syntax.

            ;;; Simple test in Lisp

(defun test-function ()
  (static-let ((counter 0))
    (incf counter)))

(test-function) ; -> 1
(test-function) ; -> 2
(test-function) ; -> 3

          

Oh, and it should come in a LET* flavor too!

            ;;; Simple sequential test in Lisp

(defun test-function-2 ()
  (static-let* ((counter 0)
                (big-counter (+ counter 100)))
    (list (incf counter) (incf big-counter))))

(test-function-2) ; -> (1 101)
(test-function-2) ; -> (2 102)
(test-function-2) ; -> (3 103)

          

The only thing that was missing was a usable and somewhat tested implementation of that technique that I could link people to from the recipe. There wasn't one, though... So, d'oh, it needed to be written and uploaded somewhere.

Where? The library of Serapeum accepted the idea and I was able to come up with an implementation. It satisfied the maintainer who also provided a few fixes, all of which are implemented in the article.

But, that's the boring stuff. Come! Let us indulge in a little bit of literate programming and figure out how exactly that code works.

Read the full article on GitHub.

#CommonLisp #Lisp

I just released In Nomine - a utility for creating, accessing, and managing custom namespaces in Common Lisp.

It's a backwards incompatible fork of lisp-namespace with more customization and tests.

Grab it from GitHub.

In hindsight, it's a very good way of shaving yaks while working on CLR2. It's not the first nor surely the last yak shaven that way. I have a namespacing utility that has enough functionality for me to like it and - more importantly - enough testware for me to trust it.

Maybe I can eventually port some code back into lisp-namespace so existing users can benefit from it.

Anyway! The difference list:

• Removed macros NAMESPACE-LET and NSLET,
• Removed the DEFINE-NAMESPACE option to automatically generate NAMESPACE-LET-based binding macros,
• Added boundp and makunbound functions for namespace names,
• Moved hash tables for bindings and documentation from being implicitly created as variables to slots of the namespaces objects themselves,
• Added long form of DEFINE-NAMESPACE to customize behavior of generated namespaces,
• Exported namespace objects and their readers.