A curmudgeon tries a language server

I write code roughly the same way I did ten years ago11 Well, this was true when I started drafting this article. I have since adopted robots to do much more of the actual typing of code, unless it’s a project where (a) I care about code quality, or (b) I try to learn something.: I make edits in my text editor, I switch over to a terminal window and I run a command to compile and execute the code. I look at the result, then switch back to my editor. If I am unsure of what happens, I either add tracer prints to the code and restart the program, or I restart the program in a debugger. If I hit an exception that crashes my program, I fix the problem and then restart it. I’m envious of what Lisp programmers do. Lisp development typically happens by typing code directly into the repl to add, remove, and replace parts of the live, running system.22 In practice a Lisp programmer would probably type the code into a scratch pad in their editor and then send it to the repl, but it is as though they typed it directly into the repl. A Lisp programmer does not need to “switch over” to something else because they are already inside their program process. They never “compile and execute” the code because the code is already running and they edit it by hot-swapping code. They never restart in a debugger because they are already inside their program process and can inspect anything they want. They don’t have to restart the program on exceptions, because the condition system allows resuming the crashing code from anywhere in the stack after the system has been patched with a fix. A consequence of this way of working is that early in a Lisp project, there may not even be any source code to speak of. Instead, the evolving definition of the system exists only in the memory image of the running process and nowhere else. This is so unlike how we normally do development that people have expressed trouble understanding what that even means. A comparison that might help is to think of how people treat relational databases in the beginning of a project. Often, people don’t start out with a version-controlled schema definition; rather, the evolving schema exists only in the running database. Sure, at some point when the project matures, the database schema is dumped out to a file, put under version control, and from that point forward, schema changes are made through explicit migrations. Similarly, a Common Lisp project that matures will be dumped from memory image to source code eventually, and then that source code will go under version control, with patches applied more carefully to the live system. But in the early stages? Oh, it’s all evolving live. I’m not a Lisp developer, so I won’t ever get that experience. I’ll stay envious. But I also realised I haven’t really tried to get closer to it. Maybe there are some improvements I can get even in my working language of Haskell. What can we do in Haskell? There are some things we can rule out immediately, and others that seem like partial wins with not too much effort. There’s no way to write Haskell that puts us inside the program process, so we will still need to restart in a debugger. Although I’m sure it’s possible to implement a Lisp-like condition system in Haskell, the types of exceptions (EitherT and async exceptions) used in common libraries don’t have one. On the flip side, fewer things in Haskell lead to exceptions thanks to the powerful type system. That’s a partial win without lifting a finger. For a couple of years, there has been a Haskell language server, an lsp implementation for Haskell. These days, Emacs has built-in support for speaking lsp through the built-in Eglot client. This seems like another cheap partial win to get higher-quality code introspection than I’m used to. Then there is a project called ghcid that watches for changes to Haskell source files and immediately recompiles things that have changed. That alone doesn’t give us much beyond what hls already does, but ghcid can also run a function once the code has compiled successfully. That is powerful, because it can be combined with the foreign-store library (or the higher level Rapid library) to maintain process state even when all the code is replaced! Thus, where a Lisp programmer would incrementally send definitions over to the repl, we use the combination of ghcid and foreign-store to automatically recompile and restart our entire program without losing important state. For many classes of software, these two approaches should lead to similar workflows. The one thing we still cannot do is evaluate code in the repl of the running program – because ghcid doesn’t support it. However, I mostly do this to test that functions work as intended, so what we can do instead is write that code as an actual unit test case, and have ghcid also run tests when it reloads the code. We end up with a process where we still write code in our editor, with the lsp server giving us information about our code. Instead of playing around in the repl to try things out, we write our experiments as unit tests. When we save our changes (tests or application logic), ghcid reloads the code, runs the tests, and restarts the application in a way that doesn’t lose important state. We never leave our editor. It’s neat on paper, but I wasn’t sure how it’d work in practice, so I tried it with a toy project. The toy project At the time of writing this, I was trying to patch up my lack of skill with differential equations and non-linear dynamics. The book I was reading33 Differential Equations; Blanchard, Devaney, Hall; Cengage Learning; 2011 has a pleasant focus on computational methods44 Traditional first classes in differential equations – including the one I took many years ago – had a heavy analytic focus out of necessity. In practice in the industry, differential equations are solved computationally because in many cases calculus is not sufficient to solve them any other way., and the book comes with software for performing the exercises, but I want to write the code on my own. At least in the early chapters, the software is not exactly rocket surgery. Here’s an example of my program producing a slope field in light grey to visualise the general solution to a differential equation, and then in black the specific solution that passes through the coordinate under the cursor. When exploring ideas from the book, I want to write some code, write some tests, and then when I save, the window containing the visualisation should automatically update to reflect the latest changes. It’s not exactly a Lisp development flow, but contains some of its good parts; specifically those that are possible with the technologies mentioned above. It sounded easy on the surface to get there, but it wasn’t. How to get Emacs to talk to hls I’ll first focus on getting Eglot with hls up and running, because that’s the less interesting part. The list below contains a lot of steps, because getting it installed from scratch with a new project takes a lot of steps. After this initial installation, only steps 2 and 4 are required to enable hls in any project. At first I ran cabal init to create a standard empty Haskell project, split into executable, library, and test code. I discovered later that ghci (and therefore also ghcid) don’t behave well when it comes to reloading multi-part projects like that. Thus, I ended up co-locating all code in one executable project. I wouldn’t want to do that for production code because it makes other types of development more difficult, but it works okay for this kind of toy project. I created a flake.nix to wrap the cabal project and provide dependency management, including libraries and build tools. This way, I wouldn’t have to globally install ghcid and hls, but could sandbox them to this project. Nix is great for fearlessly trying new things. I installed direnv with my system package manager, and hooked it into my shell’s user profile.55 Actually, I had already done this earlier for a different project. I should use direnv more. It seems like a good idea. For some reason, I still had to manually call up the Nix development shell in my terminal before I ran the project. I’m not sure why, but it might mean hooking direnv into the shell is an optional step. No idea. I created an .envrc file containing just the instruction use flake in the project directory. This tells direnv that when clients request the environment for any path under this project directory, it should automatically put the Nix flake development shell dependencies in the path. I cloned nix-direnv and imported it in my direnv configuration. This is an optimisation that makes direnv load Nix-based environments much faster. For terminal-only use, it’s optional, but it’s useful once we teach Emacs to direnv, because Emacs loads that environment often. I installed the envrc plug-in for Emacs and set up its hook. This means (some) Emacs commands will pick up the direnv. This is necessary in order for Eglot to be able to connect to hls, since we didn’t install it globally. Manually connecting to hls should be a matter of running M-x eglot, and then functionality can be tested by e.g. hovering over a function call and running xref-find-definition. Eglot also creates event buffers in Emacs containing all traffic between server and client, which can be read for additional troubleshooting information. But at this point, I struggled for a while to get Eglot to work. It failed to find the hls server. I manually enabled envrc-global-mode and then I could run M-x eglot successfully. That was probably a one-off problem, because the envrc hook hadn’t fired for the code buffer I already had open. I still find I have to manually run M-x eglot-reconnect to restart the lsp server sometimes when something gets stuck. The drawback of this is that the server takes a little while to restart, even for a tiny project. Restarting the server has been needed at least once after a general build