A road to Lisp: Why Lisp

The question that most programmers face when seeing some Lisp code for the first time is, without doubt, “what the hell is this?”. I asked myself the same thing when I first read its unconventional syntax: all those parentheses, the weird indentation, and who thought to use the first argument of format to print to stdout? (defun flip-coin-for-real () (<= (random 100) 80))(defun hello-lisp () (write-line "What is your name?") (let ((name (read-line)) (learn-it (if (flip-coin-for-real) "should" "should not"))) (format t "Hello, ~A.~%" name) (format t "The Oracle said... you ~A learn Lisp!~%" learn-it))) After getting comfortable reading code with so many parentheses, I had to learn how to use packages and symbols, how to create new projects and import libraries, how to use the REPL, and how to use conditions and restarts. Most importantly, I had to switch to a new way of thinking when constructing algorithms. The Lisp journey has a steep learning curve compared to most common languages. But it can also unlock skills that those languages never will. Why? Because in Lisp you can do things that are not possible in other languages. Lisp enables new possibilities and allows algorithms to take different shapes, giving the programmer more power and flexibility. Paul Graham coined the term Blub paradox to explain why it is difficult for programmers who have only used less powerful languages to understand the power of Lisp. In short, it’s because they are missing the concepts needed to perceive what is lacking in those languages and the advantage Lisp has over them. Louis Armstrong said something similar about jazz:

If you have to ask what jazz is, you’ll never know. — Louis Armstrong.

In this article, I’ll explain a few features that show why this special language is worth learning. It’s not an easy path, and to truly grasp its power you’ll need to use Lisp yourself. And even if you don’t end up using Lisp, you will gain a different perspective on what programming languages can do. Lisp is going to make you a better programmer because it changes the way you think about problems using code. To be more specific, it will teach you an approach that is not possible with other programming languages.

You will be able to program Lisp itself and thus adapt the language you use to the problems you are solving. Using Lisp, you will learn to grow the language toward your problem, then write the program in that language. Extensibility Lisp is extensible within itself. Experts — called lispers — commonly refer to it as the programmable programming language1. Not only will you write code for your programs, but you can write code that extends Lisp itself. This is possible thanks to the macro operator. If you’ve used macros before in languages like C, Rust, or Swift, don’t expect the same thing. Those macros are primarily a way to eliminate boilerplate or generate repetitive code. Beyond that, macros in Lisp allow you to create new constructs that become part of the language itself. They are one of the hardest features to master for Lisp programmers, so I won’t try to teach you how to use them or how they work here. I just want to give you a taste of what they do. For example, C programmers might want to use the while operator to define a simple loop. Common Lisp does not provide it, so the programmer can write a macro and add it to the language. (defmacro while (condition &body body) `(loop while ,condition do (progn ,@body))) The new while macro executes a series of instructions (the body) for as long as the condition is true. It uses loop, which is itself a macro used to write complex iterations (notice that it uses a while symbol as well). progn is a special form that takes multiple expressions, runs them in order, and returns the result of the last one. Instead of writing (loop while ... do (progn ...)), we can use the new while operator, shortening the code and making it similar to C. We just extended the language available to us. (let ((counter 3)) (while (> counter 0) (print counter) (decf counter)));; 3;; 2;; 1 The beginner might not grasp what is really magical here, since, other than the unique syntax, defmacro looks pretty similar to defun, which was used in the first code example to define new functions. To better understand the difference between the two, let’s define a new while as a function, then call it.

(defun fake-while (condition body) (loop while condition do (funcall body)))(let ((counter 3)) (fake-while (> counter 0) (progn (print counter) (decf counter)))) loop is the same macro we used before; funcall calls the function received as the first parameter, in this case body. Try to execute the code above in a Common Lisp REPL. You will receive a condition of type error, similar to this: The value 2 is not of type FUNCTION. This happens because the arguments to fake-while are evaluated immediately. (< x 3) evaluates to t — true. Because progn is a function, its arguments are evaluated. (print counter) prints 3, and (decf counter) returns 2. Since progn returns the value of its last expression, body becomes 2. So fake-while receives t as its first argument and 2 as its second, not a block of code. It will enter the while since the condition is true, and then funcall expects body to be the function to be called, but body is the value 2. This raises the condition above. The code executed by the compiler is the one below. (funcall 2) raises the condition. (loop while t do (funcall 2)) The macro while defined earlier works because it keeps the arguments — condition and body — intact, without evaluating them until needed. We can inspect the transformation that a macro operates in the REPL. This transformation is called expansion. macroexpand is a special Lisp command we can use to expand code. CL-USER> (macroexpand-1 '(while (> counter 0) (print counter) (decf counter)))(LOOP WHILE (> COUNTER 0) DO (PROGN (PRINT COUNTER) (DECF COUNTER))) The code returned from the macro expansion is the one received and executed by the compiler. We can see that this time the while macro preserved the arguments so they will be executed together with the rest of the code. A macro, unlike a function, does not evaluate the arguments in advance. Instead, it treats them as pure data. This is possible because (almost) everything you see in Lisp is made of lists. They are the main data structure, and they are used to write code.

It’s lists all the way down Programs in Lisp are composed of a series of symbolic expressions (abbreviated s-expr). Expression is a mathematical term meaning anything that evaluates to a value; symbolic means that expressions are created using values and symbols. An s-expression is one of two things:

an atom, which is a unit of data (a number, a string, a symbol, etc) a list, which is a collection of elements that can be atoms or other lists

1 ;; An atom number"hello" ;; An atom string'(1 "y" :c) ;; A list of atoms (a number, a string, a keyword)(+ 1 2) ;; A list composed of a symbol (plus) and two numbers Lisp means LISt Processing, so the purpose of the language is to process the series of instructions expressed as lists that compose the program. Since lists are used both as the main data structure and to write the code, we can extract an interesting property of the language. In Lisp, code-as-data means that both code and data are expressed using lists. This property is called homoiconicity. CL-USER> (+ 1 2)3CL-USER> '(+ 1 2)(+ 1 2) I’ve used two lists with the same content; the only difference is that I put ' in front of the second one, but the output is different. The first list is treated as code. It gets evaluated, and specifically the symbol + maps to a function that performs the sum of the arguments. Lisp uses what’s called Polish notation, where the function name is the first element of the list and what follows are the arguments. So instead of writing print("hello"), we write (print "hello"), moving the function name inside the parentheses. The output is 3, which is the result of the code execution. The second list is treated as data. ' is an operator that tells the interpreter not to evaluate the following list and keep it as manipulable data. The output is the same list, not evaluated. The subtle distinction between code and data is what makes macros possible. We can manipulate code as we manipulate data. By transforming the source code, we are able to write programs that write programs, a capability other languages have spent decades trying to imitate.

Extending the language means that by creating new constructs that are more expressive for your program’s specific scope, you can make your code more concise. Repetitive boilerplate code can be replaced with custom macros to shorten the code and save time. New control structures can be created to control resource access and the evaluation of forms. Macros can generate code, improve performance, and, most importantly, create new syntactic abstractions that are easier to use to hide complex and error-prone code. A live system Lisp is not only a programming language, it’s a live system. To clarify this point, let’s start with how different the workflow is between Lisp and other languages. With other languages, you open the project in the editor and start writing code. The first thing a lisper does is start a Lisp process, attach it to the REPL, and then load the project into that process. The Read-Eval-Print Loop is an interactive environment used to evaluate code and immediately see the result. It’s a window into the Lisp living process that is currently running your program. In other languages, once you change the code, you have to stop and compile it, then run the project to observe, test, and debug the changes. If something’s wrong, you go back to the write-compile-run-debug cycle. Instead, a lisper continuously evaluates code in the running process and observes the output directly in the REPL, since the Lisp process is the running program. Every single function can be immediately tested in the REPL as a single unit of code, as well as any other action useful for development, such as querying a database, inspecting variables, or debugging. An example of REPL use with Calva and Clojure. Evaluation results are printed near each lines, as well as in the output window. A Lisp process is usually kept alive for weeks on end since there is no need to stop it. It can stay attached to the REPL in the editor for as long as needed. New functions, macros, and variables are continuously defined and redefined, a process called binding within an internal memory called the environment. This particular workflow even has a name: REPL-driven development. This is a real killer feature. It has been available in Lisp since its invention in the 60s, and it has been imitated by other languages (where possible), like many other Lisp original features.