A view from the hill

There seems to be a roaring discussion about a talk by Robert "Uncle Bob" Martin at this years RailsConf, which mainly seems to be concerned with the question (raised by a quote appointed to Ward Cunningham) whether languages such as Smalltalk make it too easy to create a mess. Languages such as C++ would penalize the mess at least by longer compile times.

As Giles points out that's an old argument, which I too have heard too often in the past about dynamic languages, regardless whether we're talking Smalltalk, Ruby, Lisp, Python or Perl. And the anti-dote was always the current idiot language of the date, i.e. Visual Basic at its high time, Java or C# (for those who considered C or C++ to be too dangerous). Please note that I'm not at all saying that users of those languages are idiots, but those languages are particularly claimed to be usable by idiots, too. This approach to languages or to their classification does a great job of misrepresenting programmers regardless of the language of their choice. It makes the bold claim that there are programmers that are idiots and that it's possible to design a language that even those idiots can do their jobs. And the exaggeration is that its only good business sense to enforce the use of those languages for all development projects, because, as we all know, there are more idiots than intelligent programmers out there, right?

The only problem with this claim is that human stupidity is probably the only thing larger than the universe, if you make something idiot proof, nature will quickly make a better idiot. To put it a little less cynical, to err is human. And there is value in people making mistakes. As for instance, Tom DeMarco claims in his book "Slack" it's a particular good way to learn. Bob Martin tries to sell excessive testing (which isn't particular surprising, giving his background) as one safety guard to which James Robertson responds that this is not always necessary:

Yes, tests are useful. But, the debugger is not something to be feared. Rather, it's a great tool to be used in order to have the computer do all the memory work for you. I can get a lot more done by working with decent tools like the Smalltalk debugger than I can by assuming the doc is good and writing tests that just help me a whole lot less than you might think.

I can relate to this quote, being a fan of interactive development that languages like Lisp or Smalltalk made popular. But I think James is missing the point: tests are not only helpful during development. Sure, using an interactive approach helps getting the code run but does it help to come up with a clean design? Looking back at some code, I can assure you it doesn't, quite to the contrary it's a way to get a working solution quickly, not cleanly. This doesn't make the debugger or the REPL less valuable: using it you can quickly establish your tests, debug your tests, whatever. This is exactly where the value of dynamic languages shines and those "silver-bullet" idiot languages are still behind, regardless of their shiny IDEs.

But the real point summing up the discussion so far is the old Frederick Brooks quote: There is no silver bullet. It's good to be careful and to use the tools at your hands to help you avoid the big mistakes. As Paul Graham puts it for OOP: "For programs that would have ended up as spaghetti anyway, the object-oriented model is good: they will at least be structured spaghetti." But in the end you have to learn how to avoid the mess in the first place and no language and no tool can do that for you.

ObTitle: Dexter Gordon, "Ballads"

Drew Campsie announces ucw-core, the next generation of Uncommon Web, a continuation-based framework for web applications written in Common Lisp. It's the first major announcement since Drew took over development lead of ucw and it seems to focus on establishing a core library of basic technology on top of which other functionality can be implemented (e.g. functionality like those offered by the ucw-ajax). To me as a UCW user this sounds like great news, since I had the impression that ucw seemed to linger in a state of void over the last year or so.

But even more interesting is that ucw just seems to follow a certain trend. For instance, according to the announcement that Ruby on Rails and Merb will merge, one goal is that Rails will be retrofitted to make it easy to start with a “core” version of Rails (like Merb’s current core generator), that starts with all modules out, and makes it easy to select just the parts that are important for your app. The main aspect of such development (at least I hope so) seems to be best described by one of Django design goals): A fundamental goal of Django’s stack is loose coupling and tight cohesion. The various layers of the framework shouldn’t “know” about each other unless absolutely necessary. The python folks have even gone so far as to write down a web server gateway interface specification (WSGI, a quite minimalistic specification).

It will be interesting to see whether there's going to be even more unification, maybe even across language diffferences. I'm not too certain that such an unification would necessarily be a good aim; as always there are probably as many benefits as downsides to it. For instance, a common specification certainly lowers the learning curve for people switching frameworks while at the same time may stiffle further development. Then there's the fact that the different frameworks have different approaches which are certainly reflected in their APIs: for instance, Django explicitly tries to avoid magic at all cost while there's traditionally a lot of magic going on behind the scenes of a Rails application. I'm also not sure where exactly one should draw the line between common "core" functionality and additional "batteries". As an application designer it's probably most important that there is such a line at all so that you can easily pick your poison without having to buy everything.

I've become a python programmer, too, lately, due to a job change. Python is a fine language so far, although to me it's mostly just like Ruby, though with even less functional flavour. However, just as with Ruby, I'm really missing slime, the superior lisp interaction mode for Emacs, when hacking python code. I could now start to write down a list of things I'm missing (which I've intended to do), however, Andy Wingo spares me the hassle, as he has just written an excellent article on slime from a python programmers view.

However, I would like to elaborate a little on the main difference for me: the client/server socket approach of slime. Let me briefly recapulate what this implies: slime consists of two parts, a client written in Emacs lisp and a server written in Common Lisp (AFAIK there is at least also an implementation for clojure, maybe also one for some scheme implementation). In order to use slime in it's full glory, it's hence required that you have a common lisp process running which in turn runs the slime server part. If you now fire up slime, you'll get an interaction buffer over which you can access the REPL of the lisp process, which in python would be the interpreter prompt. You can then interact with the lisp process, evaluating pieces of code from your lisp source code buffer directly in the connected lisp process. What is incredibly useful for me is that you can not only start a new lisp process but also connect to an already running lisp process, given that it has the slime server started (this is obviously mainly useful if the lisp implementation you use has multi-threading capabilities). I use it to connect to a running web server application, which I can then inspect, debug and modify. Modification includes redefinition of functions, macros and classes, which of course is also a particular highlight of Common Lisp. I would like to cite a comment of the reddit user "fionbio" he made wrt. to the linked article: In fact, Python language wasn't designed with lisp-style interactive development in mind. In CL, you can redefine (nearly) anything you want in a running program, and it just does the right thing. In Python, there are some problems, e.g. instances aren't updated if you modify their class. Lisp programmers often, though not always, refer to various things (functions, classes, macros, etc.) using symbols, while Python programs usually operate with direct references, so when you update parts of your program you have much higher chances that there will be a lot of references to obsolete stuff around.

To complement Bill clementsons excellent article series on slime a little, I'm going to describe how I'm using/configuring python-mode to make it match my expectations a little closer. Essentially I would like to access my python process just as I would with slime/Common Lisp, but that's not possible. The reason, btw., is nearly unchanged: I need to code on a web server app (written in Zope) which may not even run on the same machine I'm developing on. Let's first cover the simple stuff: To enable a reasonable command interface to the python interpreter, I require the ipython emacs library. If the python interpreter runs locally, I also use py-complete, so that I can complete my code at least a little. Unfortunately, this breaks when the python interpreter doesn't run locally, because the py-complete needs to setup some things in the running python process, which it does by writing to a local temp file and feeding it to the python process. Unfortunately, the code in py-complete lacks customizability, i.e., you can't specify where that temp file should be located -- I should be able to come up with a small patch in the near future, which I will add below. Finally, I also require doctest-mode as a support for writing doctests, but that's not really relevant.

Now, on to the more involved stuff: I introduce some new variables and a new function py-set-remote-python-environment, which uses the those variables to do a remote call (via ssh) to python. This at least allows me to do things like setting py-remote-python-command to "/home/schauer/zope/foo-project/bin/instance" and py-remote-python-command-args to "debug", so that I can access a remote debug shell of my current zope product. That alone will only allow me to fire up and access the remote python, so I could now develop the code locally, having it executed remote. More typical though is that you would also want to keep the code on the remote machine, too: for this I use tramp, a package for remotely accessing files/directories from within emacs. In combination, this allows me to edit and execute the code on the remote machine. It is still nowhere near what is possible with slime, but at least it allows me to persue my habit of incremental and interactive development from within my usual emacs installation (i.e., it doesn't require me to deal with any Emacs related hassle on the remote machine).

;;; python-stuff.el --- python specific configuration

(when (locate-library "ipython")
  (require 'ipython))

(when (locate-library "doctest-mode")
  (require 'doctest-mode))

(defvar py-remote-connect-command "ssh"
  "*Command for connecting to a remote python, typically \"ssh\"")
(defvar py-remote-connection-args '("user@remotemachine")
  "*List of strings of connection options.")
(defvar py-remote-python-command "python"
  "*Command to execute for python")
(defvar py-remote-python-command-args '("-i")
  "*List of strings arguments to be passed to `py-remote-python-command`.")
(defvar py-remote-python-used nil
  "Remember if remote python is used.")

(defun py-set-remote-python-environment ()
  (interactive)
  (let ((command-args (append py-remote-connection-args 
			      (list py-remote-python-command)
			      py-remote-python-command-args)))
    (setq py-python-command py-remote-connect-command)
    (setq py-python-command-args command-args))
  (setq py-remote-python-used t))

(when (locate-library "py-complete")
  (autoload 'py-complete-init "py-complete")
  (defun my-py-complete-init ()
    "Init py-complete only if we're not using remote python"
    (if (not py-remote-python-used)
        (py-complete-init)))
  (add-hook 'python-mode-hook 'my-py-complete-init))

One of the bigger practical problems with unit testing is isolating the test coverage. Say, you want to test a piece of code from the middle (business) layer. Let's assume further the piece of code under consideration makes some calls to lower level code to retrieve some data. The problem of test coverage isolation is now that if you "simply" call your function, you are implicitly also testing the lower level code, which you shouldn't: if that lower level code gets modified in an incorrect way, you would suddenly see your middle level code fail although there was no change made to it. Let's explore ways to avoid the problems in Common Lisp.

There is a very good reason why you would also want to have such test dependencies to ensure your middle level code still works if the lower level code is extended or modified. But that is no longer unit testing: you are then doing so-called integration tests which are related, but still different beasts.

Now, I was facing exactly the typical dreaded situation: I extended an application right above the database access layer which had not seen much tests yet. And of course, I didn't want to go the long way (which I will eventually have to go anyway) and set up a test database with test data, write setup and tear-down code for the db etc. The typical suggestion (for the xUnit crowd) is to use mock objects which brings us finally on topic.
I was wondering if there are any frameworks for testing with mock objects in Lisp, but a quick search didn't turn up any results (please correct me if I've missed something). After giving the issue a little thought, it seemed quite clear why there aren't any: probably because it's easy enough to use home-grown solutions such as mine.

I'll use xlunit as the test framework, but that's not relevant. Let's look at some sample code we'll want to test:

(defun compare-data (data &connection)
  (let ((dbdata (retrieve-data-by-id (id data))))
    (when (equal (some-value data)
                (some-db-specific-value dbdata))
         t)))

The issue is with retrieve-data-by-id which is our interface to the lower level database access.
And note that we'll use some special functions on the results, too, even if they may just be accessors.
Let's assume the following test code:

(use-package :xlunit)

(defclass comp-data-tc (test-case)
  ((testdata :accessor testdata :initform (make-test-data))))

(def-test-method comp-data-test ((tc comp-data-tc))
  (let ((result (compare-data (testdata tc))))
    (assert-equal result t)))

Now the trouble is: given the code as it is now, the only way to succeed the test is to make sure that make-test-data returns an object whose values match values in the database you're going to use when compare-data get's called. You're ultimately tying your test code (especially the result of make-test-data) to a particular state of a particular database, which is clearly unfortunate. To overcome that problem, we'll use mock objects and mock functions. Let's define a mock-object mock-data and a mock-retrieve-data function, which will simply return a single default mock object.

(defclass mock-data ()
  ((id :accessor id :initarg :id :initform 0)
   (val :accessor some-db-specific-value :initarg :val :initform "foo-0"))))

(defun mock-retrieve-data (testcase)
  (format t "Establish mock for retrieve-data")
  (lambda (id)
      (format t "mock retrieve-data id:~A~%")
      (find-if #'(lambda (elem
			(when (equal (id elem) id)
			  elem))
                    (testdbdata testcase))))

Why that mock-retrieve-data returns a closure will become clear in a second, after we've answered the question how these entirely different named object and function can be of any help. The answer lies in CLs facility to assign different values (or better said) definitions to variables (or better said to function slots of symbols). What we'll do is to simply assign the function definition we've just created as the function to use when retrieve-data is going to be called. This happens in the setup code of the test case:

(defclass comp-data-tc (test-case)
  ((testdata :accessor testdata :initform (make-test-data))
   (testdbdata :accessor testdbdata)
   (func  :accessor old-retrieve-func)))

(defmethod set-up ((tc comp-data-tc))
  ; set up some test data
  (dotimes (number 9)
    (setf (testdbdata tc)
             (append 
                  (list 
                      (make-instance 'mock-data 
                            :id number 
                            :value (concatenate 'string "value-" number)))
                 (testdbdata tc))))
  ; establish our mock function
  (when (fboundp 'retrieve-data)
    (setf (old-retrieve-func tc)
             (fdefinition 'retrieve-data))))
  (setf (fdefinition 'retrieve-data)
           (mock-retrieve-data tc)))

(defmethod tear-down ((tc comp-data-tc))
   ; After the test has run, re-establish the old definition
   (when (old-retrieve-func tc)
      (setf (fdefinition 'retrieve-data)
           (old-retrieve-func tc))))

You can now see why mock-retrieve-data returns a closure: by this way, we can hand the data we establish for the test case down to the mock function without resorting to global variables.

Now, the accessor fdefinition comes in extremely handy here: we use it to assign a different function definition to the symbol retrieve-data which will then be called during the unit-test of compare-data.

..Establish mock for retrieve-data
mock retrieve-data id: 0
F
Time: 0.013


There was 1 failure: ...

There is also symbol-function which could be applied similarly and which might be used to tackle macros and special operators. However, the nice picture isn't as complete as one would like it: methods aren't covered, for instance. And it probably also won't work if the function to mock is used inside a macro. There are probably many more edge cases not covered by the simple approach outlined above. Perhaps lispers smarter than me have found easy solutions for these, too, in which case I would like to learn more about them.

Steve Yegge blogs about XEmacs needs to die and I would like to add my own little comment on the issue. First, perhaps some background: I'm a long-term XEmacs user, since about 1995, I think. I started hacking Elisp around 1996, implementing a major mode for the Otter theorem prover and various other stuff. Since 1997 or so, I had been somewhat active on the xemacs-beta mailing list, reporting build successes/failures, participating in discussions etc. and even writing an article on the then-new XEmacs port to Windows in the german iX magazine. So, perhaps, I'm a bit biased towards XEmacs. That I've been using XEmacs instead of Emacs had a lot to do with two factors: at my university, the local Emacs guru used XEmacs and provided a very complete configuration to start with. To start hacking Prolog source didn't require any configuration on my side at all. When I tried to re-establish that configuration on my own linux box, using Gnu Emacs was totally out of the question, which is basically the second reason: XEmacs came with a lot of batteries included, whereas Gnu Emacs only came with a directly dying low battery at best. And third, the XEmacs interface experience was a lot friendlier than with the naked Gnu Emacs.

Now, at the time I was actively following XEmacs development, XEmacs had a whole lot of man power behind it, with lively discussions and an exciting movement of adding new features. At the same time, RMS discussed switching Gnu Emacs to Guile (a scheme dialect), which me, as a Common Lisp user, scared me off even more. Around 2002 or so, my spare time for XEmacs dropped to zero, so I unsubscribed the development mailing list and was just a happy user. Until two seemingly unrelated things happened: first of all, Unicode was no longer to be ignored and it was obvious that the Mule (multiple languages for emacs or some such) for XEmacs wasn't up to the task, especially not on Windows. That was quite a problem for me back then with the then current stable XEmacs 21.4. Now, six years later, XEmacs 21.5 is still not there (aka released as the new stable version) and it's unicode support still sucks (not so much as it did, say, two years ago, but it still fails a lot of tests from Markus Kuhn unicode test suite). The second development was that the Emacs team, which I had only barely been aware of previously, somehow gained a enourmous momentum and catched up a lot of ground, where XEmacs had been the leader by far. In particular, as of Emacs 22, nobody in its sane mind could argue against the fact that the current Emacs handles Unicode etc. far better than XEmacs (this is especially true for XEmacs 21.4 on Windows, but even the current beta XEmacs 21.5-b28 on Unix isn't where it should be). This seems to be directly related to the fact that a lot of the key developers of the end of the last century no longer actively participate in XEmacs development. I think only few people from the XEmacs crowd would argue against the impression that XEmacs development has more or less cringed to a halt and the distribution of development power between Emacs and XEmacs development is nowadays reversed to the situation from the 90s.

That being said, I still stick with XEmacs. Last time I tried using Gnu Emacs (that's the Emacs 22 from Ubuntu Hardy), I still found that I'm far more accustomed with the XEmacs intrinsics on how to perform specific actions than with how it works in Emacs. And I never got around to clean up my mess of configuration files to properly support the various flavours plus adding the extension libraries that still don't ship with Emacs (Slime, for instance). Finally, with more and more users switching to Emacs, there has to be someone reporting problems,

Now, back to Steves blog post: I think he has some valid points in it, but for the most part I disagree. First of all, while I agree that Eclipse and similar IDEs are still not at the greatest enemy to Emacs (regardless of flavour). I'm currently again using Eclipse at work and it's a pain finding out how to properly associate an unknown file extension with some specific editor (mode). And it's a memory hog that results in unbelievable handling (on my dual core 2GB equipped desktop) that is even worse than my first XEmacs experience on my 486/33 with 8MB ram in 1995. Eight megs and constantly swapping? Hah! I can't believe that even today I have to hear that Emacsen would be too complicated, given that for any task/configurations I spend endless time searching/clicking through the gazillions of settings in Eclipse.

However, I also don't believe that a marriage between Emacs and some web-browser is the way to go. There are much more pressing issues to move Emacs into the next century than switching from Elisp to <whatever>. If you're interested in extending your editor, you have to learn about the way how to do that. But that is totally independent from the question which editor you're going to use in the first place.
What is a much more pressing need in my opinion is finally implementing multi-tasking for the (X)Emacs core. Still being locked in a single-threaded application model is sooo 1990s. It's not becoming more and more ridicuosly, it's utterly unbelievable.

In one other point, though, Steve is right on track: the divergence between APIs is becoming a bigger problem with every passing day. I think it's very unfortunate that a) Emacs developers don't let them inspire by the existing APIs in XEmacs and vice versa and b) that there are too few XEmacs developers that merge current Emacs enhancement into the XEmacs codebase (the other way 'round is always problematic due to license issues).

So, before I would consider pursuing a major architectural change like XUL or some such for Emacs, I would first work on the much lower-hanging fruit of narrowing the gap between the two flavours. It's a pity I myself only have the spare time to write such blog posts instead of actively working on code towards that goal.

In case you're interested in continuation-based web development with Common Lisp, it might be of interest to learn that Drew Campsie has voluntered to take over UCW development. Marco Baringer, initiator and maintainer of UncommonWeb (aka UCW) seemed somwhat relieved that finally somebody with more time on his hand steps up to take over.

That Marco hadn't had the time to keep pushing UCW is quite obvious from the mailing list archive. Lately Attila Lendvai seemed to me (an innocent external observer) to had pushed UCW foreward, however, as he Attila announced, he wants to go into a different direction than Drew.

I've been using ucw-dev during the last two years and the current situation really called for a change, in my opinion. UCWs main problem, as I observe it, is lacking documentation and clearly stated goals which way UCW will go. The automatically extracted documentation doesn't contain information how the bits fit together and all tutorials are a) incomplete, b) outdated as far as they are targetting ucw-dev, whereas development has mainly happened in ucw-ajax. However, it was not all clear that ucw-ajax really would become the main line of UCW.

I like UCWs component architecture a lot and will eagerly follow what is going to happen. With two prominent developers such as Marco and Attila more or less leaving the project, it will be interesting to see with how much activity development will happen from now on.

This post is mainly a reference post about a particular topic whose solution wasn't immediately obvious to me from the docs to CL-SQL. Using CL-SQL with (enable-sql-reader-syntax), I had written a routine that looks basically likes this:

(defun data-by-some-criteria (criteria &key (dbspec +db-spec+)  (dbtype +db-type+))
  (with-database (db dbspec :database-type dbtype :if-exists :old)
    (let (dbresult)
      (if criteria
	  (setq dbresult
		(select 'some-model 'other-model
			:where [and [= [some.criteria] criteria]
			            [= [some.foreign_id] [other.id]]]
			:order-by '([other.name] [some.foreign_id] 
				          [year] [some.name])
			:database db))
          (setq dbresult
		(select 'some-model 'other-model
			:where [and [null [some.criteria]]
			            [= [some.foreign_id] [other.id]]]
			:order-by '([other.name] [some.foreign_id] 
				          [year] [some.name])
			:database db))
      (when dbresult
	(loop for (some other) in dbresult
	      collect some)))))

This is ugly because the only difference between those two select statements is the check for the criteria, but I had no idea how to combine the two select statements into one, because it's not possible to embed lisp code (apart from symbols) into an sql-expression (i.e. the type of arguments for :where or :order etc.).

With the next requirement things would become far worse: The order-by statement needs to get more flexible so that it is possible to sort results by year first. Given the approach shown above this would result in at least four select statements, which is horrible. So, naturally I wanted a single select statement with programmatically obtained :where and :order-by sql expressions.

Step 1: It occured to me that it should be possible to have the arguments in a variable and simply refer to the variable. E.g., using a more simple example:

(let (where-arg)
  (if (exact-comp-needed)
      (setq where-arg '[= [column] someval])
      (setq where-arg '[like [column] someval]))
  (select 'model :where where-arg))

So I could now have my two different where-args and two different order-args and use a single select statement. Main problem solved.

Step 2: But for the :where arg in my original problem, only a small fraction of the sql-expression differs. So how do I avoid hard coding the entire value of where-arg? How can I combine some variable part of an sql-expression with some fixed parts? I.e, ultimately I want something like:

(let (comp-op where-arg)
  (if (exact-comp-needed)
      (setq comp-op '=)
      (setq comp-op 'like))
  (setq where-arg '[ <put comp-op here> [column1] someval])
  (select 'model :where where-arg))

But with CL-SQL modifying the reader, there seems to be no way to make <put comp-op here> work. I didn't knew how to get the usual variable evaluation into the sql-expression, or how to escape from CL-SQL's sql-reader-syntax to normal lisp evaluation.

Somewhere in the back of my head where was that itch that CL-SQL might offer some low-level access to sql expressions. And indeed it does. There are two useful functions, sql-expression and sql-operation. sql-operation "returns an SQL expression constructed from the supplied SQL operator or function operator and its arguments args" (from the cl-sql docs), and we can supply the operator and its arguments from lisp -- which is exactly what I want.

Now, the nice thing is that it's easy to mix partly handcrafted sql expressions with CL-SQL special sql syntax constructs that will be automatically handled by the reader (if you enable it only via enable-sql-reader-syntax, of course). I.e., for <put comp-op here> we can use sql-operation, but the rest stays essentially the same:

(let (where-arg)
  (if (exact-comp-needed)
      (setq where-arg (sql-operation '= [column1] someval))
      (setq where-arg (sql-operation 'like [column1] someval)))
  (select 'model 'other-model  :where where-arg))

Now, coming back to my original problem, based on this approach I can split out the common part of the :where and :order arguments and combine those with the varying parts as needed and hand them down to a single select statement. Problem solved.

It's interesting to see that more and more environments for interactive development are popping up. If you're wondering what I'm refering to, it's what has traditionally been labelled as an interpreter: you get some kind of prompt like in a command shell and can directly enter and run program constructs. In Ruby, irb is the tool to use, whereas most, if not all, Common Lisp systems provide you a REPL whenever you start-up the system. REPL is an abbreviation for read-eval-print-loop and this pretty much sums up how interactive development works: you enter some code into the system, and as soon as you hit enter (i.e. finish up the particular line of code), the system will read and evaluate it ("interpret" it, although CL systems may also compile it automatically), presenting you with the results.

To me, this seems like more people come to understand that the classic edit-compile-debug cycle of traditional compiled languages like C isn't particular well-suited to a bottum-up programming style, as advocated by the agile programming hype. Where is the beef? I've been told that the reason why the asian folk don't use forks and knives when dining stems from the opinion that when dining one merely wants to eat, while all handcrafting (like slicing the meat) belongs to the a-priori cooking. The edit-compile-debug cycle tends to produce code more akin to the western steak style, while interactive development tends to produce smaller code parts, all well prepared (read: tested) from the beginning, so you don't end up with bloody interiors as easily. That's because of two reasons, I think: In an interactive environment testing a piece of code is cheap and easy. Hack up a small routine and enter it. Next step: call the routine with some sample data. No need to compile. In case there's an error, you'll be thrown into the debugger right away. With the traditional edit-compile-debug cycle, you're gonna edit the code, save it, call the compiler, edit another piece of code testing it, compile that, too, run that and only then you'll see if it succeeds. This might scare developers from running such tests to often and in result to produce larger routines/code blocks. Second reason: in the interactive environment you'll typically have somewhat less comfort then in a full-blown editor. So I'm arguing that the slight discomfort will help you in keeping things small because then you'll not gonna need far reaching edit support.

There is, however, a price to pay: For starters, because your interactive system provides you with a single environment during development, it's easy to mess it up. That's different with the traditional edit-compile-debug cycle where every time you compile you run the code in a clean environment. This means, old code refering to refactored functions will result in (usually: compilation) errors whereas in interactive development you may miss some spot in need for modification because the old refactored code is still around.
Second, when fiddling with some functions, it may happen that one hacks up several variants, maybe with only slightly differing names or argument lists, so it may happen you forget some crucial code when finally saving your carefully crafted solution (where saving here typically means copy it over to some editor, not saving an image as you may do in Smalltalk or Lisp). Another thing is that one has to be careful not to confuse bottum-up interactive development with hacking without thinking or design-less programming. This threat is all to common because with an interactive environment it's just so easy to get started. Unfortunately, without thinking you'll soon end up with problems one and two and even worse, it's likely that after a little while nobody (including you, the author) will be able to maintain the resulting code. But if you're aware of the potential pitfalls, interactive development is a nice way to do explorative programming which avoids a lot of time consuming labour you have to do with the traditional edit-compile-debug cycle.

As a side-note, environments offering incremental compilation like Eclipse aim at a similar goal, but only go half the way as they only take care of the edit-compile part. Only when one combines that with a test-driven development style you arrive at a similar point, because now the test code, when automatically run, will take care of immediately running the code in question (the eval/print part as far as testing the code is concerned).

To finally come to the reason for this blog post, I was pleased to learn about the Perl Console, which is not the same as the Perl debugger. I'm eager to try it out, although I'm sure that interactive development with a repl and the executable brain dump that Perl code usually is, is a highly dangerous mix.

Update: There is also a series of articles about implementing a REPL in Perl which may be of interest.

This is just a minor rambling inspired by a recent thread on cll about the ups and downs of using Rails. What I find really overwhelming is the lack of proofs that other (typically Lisp-based) frameworks really have so much benefit over the established more traditional frameworks (e.g. Rails, Zope, plain PHP, ...). All I can see is starter documents, tutorials etc. that show how to program yet another blog or reddit clone in some particular framework. I miss fair comparisons and detailed discussions why and where exactly that one particular way of doing things has benefits. And especially with regard to those reddit clone prototypes: AFAICT, reddit was far beyond implementing some half-baked prototype (in Lisp) when they decided to switch (to Python). Yes, for demo purposes reinventing the wheel with some unround edges might be acceptable, but in reality only rolling wheels will be sold and this is where the challenge is.

I believe that while it's nice that frameworks help with implementing your 500 line application, they really need to show their value with large applications. Where is a discussion of a large on-line shopping system in, say, UCW and CL-SQL, with secure shopping over SSL, login handling, input validation, distribution over multiple servers etc.? Now, I'm not suggesting that UCW, Weblocks etc. can't be used to build such websites, but it would be nice to see a much more in-depth explanation/discussion/comparison of how to do it. That would also help increase the credibility of Lisp or at least, of lispers discussing (other) web frameworks.

In case you're interested in object-oriented programming in Lisp and happen to speak German, you'll be happy to learn that Pascal Costanza of ContextL and AspectL fame (apart of his postings to cll), has given a talk about CLOS, the Common Lisp Object System at the Hasso-Plattner Institut, Potsdam. And while we're at it, Pascal also has a very interesting blog post about the origins of the advice facility.

German:
Falls Dich, lieber Leser, interessiert, wie man objekt-orientiert in Lisp programmiert, solltest Du Dich darüber freuen, dass Pascal Costanza, bekannt durch ContextL und AspectL (abgesehen von seinen Postings in cll), einen Vortrag in Deutsch über CLOS, dem Common Lisp Object System, am Hasso Plattner Institut in Potsdam gehalten hat. Und weil wir gerade dabei sind, Pascal hat außerdem einen sehr interessanten Blogeintrag über die Geschichte von 'advice'.

Every now and then, I'm still blown away by the elegant power that the repl (read-eval-print-loop) of Common Lisp provides for everyday tasks. A recent example: I needed to fix a broken column definition in one of my Postgres databases. I'm not a DB pro, so I will just drop the column. But of course, we want to retain the old data, so here we go (using CL-SQL):

SOMEDB> (connect +db-spec+)
#
SOMEDB> (defvar *olddata* (select [id] [data] :from [sometable]))
*OLDDATA*
SOMEDB> (execute-command "ALTER TABLE sometable DROP COLUMN data;")
NIL
SOMEDB> (execute-command "ALTER TABLE sometable ADD COLUMN data varchar;")
NIL
SOMEDB> (loop for (id value) in *olddata*
                  do (update-records [sometable] 
                                 :attributes '(data) 
                                 :values (list value) 
                                 :where [= [id] id]))
NIL
SOMEDB> (disconnect)

That's it. First we open a connection, store away the olddata (which will be returned as a list of tuples) in a global variable, modify the table and finally restore the data. Now, what I find really nice about this is that I can operate on the data as if I had an intersection of psql, shell and a real programming language, which is pretty much the point of this blog post. I think that Ruby probably provides a similar environment with irb. And, hey, today I learned that XEmacs comes with an interface to postgres, so I might have been able to do it from within my favourite editor, too ...

Some time ago, I was looking at splitting some text with Elisp, Perl, Ruby and Common Lisp. Yesterday, when I again had to do quite the same thing, it occurred to me that the Common Lisp solution was unnecessary complex/long. I'm not a Perl guru, but I believe the following is probably hard to beat even with Perl:

CL-USER> (format t "~{<li>~A</li>~%~}" (cl-ppcre:split "\\|" "Kim Wilde|Transvision Vamp|Ideal|Siouxsie and the Banshees|Nena|Iggy Pop"))

<li>Kim Wilde</li>
<li>Transvision Vamp</li>
<li>Ideal</li>
<li>Siouxsie and the Banshees</li>
<li>Nena</li>
<li>Iggy Pop</li>
NIL

For the uninitiated, it's not the cl-ppcre library which is interesting here but the built-in iteration facilities of format. See the Hyperspec on the control-flow features of format for details. Now, I usually tend to avoid the mini languages that come with Common Lisp like the one of format or loop when writing real programs, but when using Lisp as a glorified shell they come in very handy.

Dear Lazyweb, I'm looking for a way to automate unit testing with the help of ASDF. I'm using XLUNIT at the moment, but this isn't really relevant. What I want to achieve is that on every compilation of some source file, it's corresponding test file will be loaded (and hence the tests it contains run). However, what I seek to avoid is simply adding the test files to the component definition; the test files should be kept separately. From what I gather from the ASDF documentation, this should be possible using :perform forms, but the very same docs leave me wondering how. What I found (looking at Jörg Höhles asdf file for iterate) is how to load and run a complete test-system, but this is not what I want to do. Ideally, I would like to have a single perform instruction looking something like this:

(defmethod asdf:perform ((op test-op) c)
  (when (asdf:glorified-find-component "my-component" c)
        (let ((component-test (find-component-test c)))
            (when component-test
                   (asdf:operate 'asdf:load-op 'component-test)))))

Here #'glorified-find-component needs to recursively follow the components parent and #'find-component-test should return a component c-test (with c expanded, obviously). Now, I guess I'm just clumsily reinventing the wheel and hence I'm wondering if somebody has already solved "the problem".

While I'm at it (it referring to testing), Stefil looks like a very interesting test environment in case you're developing your programs with Emacs and Slime. From there, I found a link to Phil Gregory's test framework comparison which I found much more enlightening than the ALUs list of test frameworks.

Joey seems to have inspired a "list of programming languages you know"-contest over on planet.debian.org, so this is mine:

1. Basic (forgotten)
   I've started using Basic on the Commodore C64 and have used it considerably, apart from the intrinsics of peek and poke, which some would claim was what Basic on the C64 was all about. I also used Basic on the Apple II at school, but was spared any further need to use Basic.
   
2. Cobol (forgotten)
   I remember my old computer science teacher bringing white and green prints on endless paper and teaching us the joys of Cobol. Having used Basic before, it looked somewhat suspicious but not too alien. Fortunately, we only had to solve some simple problems on paper, as the Apple II only had .... DBase II, I think.
   
3. Pascal (forgotten)
   (Turbo) Pascal was the language of choice in the first semester at the University of Koblenz at the start of the nineties. I've did what I had to do in it, but never really liked it. From the occasional look I throw in Wirth's algorithms book I know I can still read it, but the language looks just as baroque as the copy of the book I have.
   
4. Prolog (forgetting)
   I've really done a lot in Prolog, being first exposed in the second semester to it. I remember to marvel at some guys who grokked the language at first usage in the initial course -- to me, having used only imperative languages up to then, it was a mistery. I still think Richard A.O'Keefe's book "The Craft of Prolog" is among the best CS books.
   
5. C/C++ (forgetting)
   About 1992, I somehow got interested in object-oriented programming. Free software was still unknown to me, so I bought a student edition of Turbo C++ and some Addison-Wesley book and of I went inventing another wheel. At the time, I also had to do some C hacking for some course at the university, involving MS Windows programming. I've used C over time in several smaller projects, but would never call myself a C hacker.
   
6. Assembler (forgotten)
   My first exposure to Assembler was with some book for the C64. I was very interested but understood next to nothing. I should program some I/O chip directly instead of just using "load $,8,1"? WTF? Next look was in University, of course. I still know enough bits to be able to grok more or less the output of (disassemble 'foo) of some (tiny) foo-function in CL on an i386-style machine, but that's about it.
   
7. Mercury (had interest in)
   Mercury seemed like a nice successor to Prolog, adding several features of functional languages. I read several manuals, had several installations and never used it for anything.
   
8. Life (had interest in)
   Another possible successor to Prolog, which I've learned about at a summer school for language, logics and information. That's about all I remember.
   
9. XLisp (forgotten)
   My first exposure to Lisp in the context of a course for computational linguistics. The only thing worse than the editor used in the system was the tutor giving the course. A horror of parenthesis is about the only thing I remember.
   
10. Shell scripting (daily usage)
    I think since 1992 or 93 I've been using Unix systems and shell scripts. I no longer can remember whether our Suns dumped you into csh or sh or whatever, but since I've been using Linux systems (since 94), I've been using bash and written bourne shell scripts. Of course, I've also written the occasional csh script abdomination, but "Csh programming considered harmful" was an early lecture.
    
11. Emacs Lisp (forgetting)
    I remember seeing one of my instructors doing nearly miracles with XEmacs and wondering how his fingers could be that flexible -- I know that now. I've written several small utilities in Elisp and keep using it occasionally.
    
12. Common Lisp (improving)
    When I started a student job in 1995, I also started using Common Lisp, using Steele's CLtL2. Having used Elisp before, I no longer was worried about parenthesis. I still keep finding areas I didn't know or used before. And it's a language large enough that keeps you re-learning tricks you thought you knew. Which didn't stop me from giving a course on CL, which in hindsight didn't have the right focus: instead of showing that CL is just as capable of doing whatever you expect, it's also capable of doing you won't expect.
    
13. Smalltalk (interested)
    At the computational linguistics lab in Freiburg, the parser system was implemented in Smalltalk, so when I started there I took a small course in Smalltalk. As I worked more on the knowledge representation side of the system, I only rarely used it, though, so rare indeed, that I never got to do anything serious with the language. I still can read Smalltalk despite it's somewhat unusal syntax and have to do that even sometime.
    
14. Tcl (forgotten)
    We wanted to extend a linguistic annotation written in Tcl, so I bought a book and started digging through the code. Soon enough a student took over the job (and did it rather well), so I started the tiny bits I've learned quite soon. I remember finding the concept that every thing is a string to be rather messy and still see it as a lame excuse for dynamic/weak typing.
    
15. Perl (static)
    For nearly ten years, I had been using shell and/or Emacs to solve any data mangling tasks when I finally gave in and had a closer look at Perl. I still find it amazing how quick and dirty it's possible to solve problems in Perl, but on the other hand I've given up Perl for more complex tasks that require some thought and design, since Perl's syntax and language idiosyncrasies keep leading me to programs I don't even like myself.
    
16. XSLT (improving)
    At about the same time I started using Perl, I also needed to transform a set of XML documents and XSLT was all over the media, so I had a closer look. I dislike the noisy syntax of XSLT and particular the intermix with output elements and it's fever for being a "clean" functional language. But the concept of XPath and the idea of traversing a document via axis I find very compelling.
    
17. make (static)
    For a long time the power of make didn't occur to me. A colleague of mine, however, used it to implement a nifty workflow for document processing, of which I learned a lot of tricks when I needed to mimic it for a different task.
    
18. Ruby (improving)
    When I finally was fed up with Perl, I looked around to find Ruby as a nice halfway compromise between Common Lisp and Perl. It's still fancy and geeky and with Ruby on Rails has a lot of buisness ballyhoo around it, so I hope it will see more use in my current surrounding.
    
19. Java (not interested)
    Like everybody else, I couldn't avoid looking at Java. It seems to be the Cobol of today and as such is the job guarantee today. But Java looks so boring, I can hardly stand it. And this with regard to both the language and the results produced with it. Sigh But still, it's likely that someday I'll going to buy another book (that would be my third Java book, the first being O'Reilly's Java in a nutshell for Java 1.1 in maybe 1997 or so?), since just like Cobol at it's time, avoiding Java is hard ...
    
20. Python (not interested)
    When Python 1.4 hit the scene, there was quite some bubble around it. It burst quickly and as my latest review of the language reveiled last year, it still doesn't seem interesting.
    

Missing in this list is stuff like Scheme, Dylan, Oz, Oberon, Eiffel, Haskell and probably some other "tiny" languages which I looked at at some point in time and really didn't ever took a second look. Of those, I may take another look at Haskell in the future, but currently I have no concrete plans in doing so.

For a review, I needed to get the track list of a given CD. As the track list wasn't available via CDDB, I went to some large online store and found the tracklist. I need to convert it to XML, though. The original data I fetched looks like so:

1. Fox In A Box
2. Loaded Heart
3. All Grown Up
4. Pleasure Unit
...

whereas I need:

<li id="1">Fox In A Box</li>
<li id="2">Loaded Heart</li>
<li id="3">All Grown Up</li>
...

After cutting the original data to my Emacs, writing out a simple file and using Perl for that simple transformation seemed just gross. In the past, I've been an Emacs hacker. But no more, or so it seems, since it took me nearly half an hour just to come up with this simple function:

(defun tracklist-to-li (point mark)
  "Generate a string with <li>-elements containing tracks.
Assumes that one every line of region, a track position 
and the track name is given."
  (interactive "r")
  (save-excursion 
    (goto-char point)
    (let ((current-pos (point))
	  (result ""))
      (while (re-search-forward "^\\([0123456789]+?\\)\.[ \t]+\\(.*\\)$" 
				mark t)
	(setq result
	      (concat result
		      "<li id=\""
		      (match-string 1)
		      "\">"
		      (match-string 2)
		      "</li>\n"))
	(setq current-pos (point)))
      (message result))))

What took the most time was that I've had forgotten to escape the grouping parenthesis in the regular expression and that it took me a little while to accept that there is really no \d or equivalent character class in Emacs regexps. Which probably means that I've been doing too much in Perl, sed and the like. OTOH, it just may hint at the horror of regular expressions handling in Emacs. What I also dislike is that whenever you want some result in Emacs and see it, too, you have to invoke an interactive operation like message. Of course, there is IELM, but this doesn't really help you for interactive functions operating on regions.

And five minutes later, I realize I need to convert some string like "The (International) Noise Conspiracy|The
Hi-Fives|Elastica" into a similar list structure. With a simple cut & paste and roughly 30 seconds later, I have

[bauhaus->~]perl -e '$a="The (International) Noise Conspiracy|The Hi-Fives|Elastica"; @a=split("\|",$a); 
foreach $b  (sort @a) { print "<li>$b</li>\n"; }'

Hmm. Perhaps I've come quite a long way on the dark side already ...

On the other hand, in Ruby, this is just as simple (I'm using irb, the interactive ruby shell here):

irb(main):008:0> a="The (International) Noise Conspiracy|The Hi-Fives|Elastica"
=> "The (International) Noise Conspiracy|The Hi-Fives|Elastica"
irb(main):009:0> a.split("|").each {|string|
irb(main):010:1* print "<li>"
irb(main):011:1> print string
irb(main):012:1> print "</li>\n"
irb(main):013:1> }
<li>The (International) Noise Conspiracy</li>
<li>The Hi-Fives</li>
<li>Elastica</li>
=> ["The (International) Noise Conspiracy", "The Hi-Fives", "Elastica"]

The difference here is the implicit array Ruby generates, which of course in Perl you could hide in the array position of the foreach loop. Note the annyoing misfeature of irb to always show the prompt even when your still continuing your current input line.

In Common Lisp we can do it just as short:

CL-USER> (let* ((a "The (International) Noise Conspiracy|The Hi-Fives|Elastica")
		          (splits (ppcre:split "\\|" a)))
	           (loop
      	              for string in splits
	              do 
	                  (format t "<li>~S</li>~%" string)))
<li>"The (International) Noise Conspiracy"</li>
<li>"The Hi-Fives"</li>
<li>"Elastica"</li>
NIL

The same thing here: The result of the split could have been easily embedded in the loop.

The lesson, of course, is that in the end this example only serves to show that things that are easy to achieve in a high-level are indeed easy to achieve. Or to put it otherwise that the use of regular expressions is no more a discriminating feature between programming languages.