programming musings

In the comments section of The Joy of REPL, a reader is posing an interesting question: how do i make my joy persistent? Or, in her words,

Dumb question – you are happily programming in the environment, and the lights go out. Have you lost your state?
How do you save “source” code? I’m interested from the angle of irb, as I like ruby. I still think in the mode of writing the source in an editor, checking it in, etc.
I can’t seem to imagine this environment in terms of day to day work, esp with a development group.

Managing persistence depends largely on your development environment. But of course, the primary method is the traditional one: you write files. You don’t need to literally type your code at the interpreter’s prompt. Any decent editor will let you send to the interpreter expressions written (and, eventually, saved) in any editing buffer. Emacs excels in this aspect, specially if you’re on Lisp and use Slime (or its cousin slime48, which works on scheme48). You can see it in action in Marco Baringer’s excellent tutorial video (bittorrent available here). The important thing to keep in mind is that you need the ability to evaluate individual expressions (as opposed to loading files as a whole), and this is made possible by the joint work of your language’s runtime support and your editor. I’m not a Ruby user, but i bet Emacs or vim, among others, give you similar facilities. That said, i would be surprised if they are as impressive as Slime’s. Because Slime is cheating: it interacts with a programming system (namely, Common Lisps’) that does its very best to allow an incremental, organic development style. How so?

Well, as soon as you go beyond writing little toy snippets and into serious (as in bigger) programs, you’ll need some kind of module system, in the sense of a way of partitioning your namespace to avoid name collisions. Every language out there provides such a mechanism in one way or the other (and Scheme famously provides as many ways as there are implementations; more on this below). Therefore, to keep enjoying our interactive way of life, we need that the interpreter and the editor cooperate to evaluate our code in the correct namespace. Common Lisp’s module system is based on packages. Each symbol known to the system belongs to one of them, and it is customary to begin your files with a form that informs whoever is interested to what package the following code belongs into… and the editor/interpreter team are definitely interested: expressions sent from a buffer to the REPL are evaluated in the correct context. Again, i don’t know whether Ruby or Python offer this synergistic collaboration, but i know that you definitely need it to attain the Joy of REPL.

Common Lisp is not unique in this regard. In the Scheme world, scheme48′s module system was also designed with interactive, incremental development in mind, and taking advantage of it in Emacs required an, in a sense, almost straightforward (but, by all means, worthy) effort (thanks Taylor and Jorgen). As an aside, this is what makes s48 my preferred scheme and keeps me away from otherwise remarkable systems like PLT. (And this is why the current R6RS standard module system proposal is a show-stopper: if you happen to have a friend in the committee, please write him and include a link to Taylor Campbell’s alternative proposal and its accompanying rationale.)

Thus, when lights come back, you recover your previous environment by reloading your files. Good module systems provide means to streamline this operation, typically (but not always) by storing the package definitions in separate files. But this is still a nuisance, isn’t it? I must wait to all my files being reloaded and maybe byte-compiled… Don’t despair, there are better ways. Most CL implementations and several Schemes (MIT/GNU Scheme and, again, scheme48 come to mind) allow you to save your complete state, at any time, in what is called and image file. This image contains a binary snapshot of the interpreter’s state, and you can reload it at any later time. Being a binary representation, it will come to life blazingly fast. Besides Lisp, Smalltalk is the paradigmatic (and possibly pioneer, but i’m not 100% sure on this) image-based language: for instance, in Squeak, the only way to launch the environment is loading a previously saved image, which contains detailed information of your previous state (including the graphical environment). In this sense (and many others), Smalltalk is a dynamic programmer’s dream come true.

Image files make things even better, but not perfect: you still need to save your state every now and then. In an ideal world, persistence should be automatic, managed behind the scenes by the system, even by the operating system. Just like the garbage collector we have come to know and love in our dynamic environments manages memory for us. This nirvana is called Orthogonal Persistence, but unfortunately, we’re not there yet. I first heard of OP from the guys of the Tunes project, where François-René Bân Rideau and friends have envisioned what i view as the ideal computing environment. Unfortunately, up to this day it remains in the Platonic realm of the ideals (but this doesn’t prevent their having one of the best online knowledge bases on computer science). Another interesting project in this regard, with actually running code that may interest the pythonistas among you, is Unununium, an operating system built around the idea of orthogonal persistence. Finally, in this context it is also worth mentioning again Alan Kay’s brainchild Squeak, which provides an environment that, without being an entire OS, in many ways isolates you into a wonderland of its own.

Tags: common lisp, computer science, emacs, lisp, programming, scheme, smalltalk