A view from the hill - Programming

There's nothing left to see

nospam@example.com (Holger Schauer) — Mon, 09 Nov 2009 09:28:12 +0100

Via Lambda the Ultimate I came across an interesting article On data abstraction, revisited by William Cook, written for OOPSLA'09. It carefully dissects abstract data types from objects. All theoretical considerations aside that distinguish ADTs and objects, there is one common characteristics given by Cook: you can't inspect the concrete representation of the data you're abstracting. This is in itself interesting and reminded me of two rather practical things.

First of all, I was reminded of a section in Bob Martins Clean code development which discussed the idea that you should on the one hand follow the rule "Tell, don't ask" and on the other hand have data access objects that don't have much, if any behaviour besides providing data. This is obviously directly related to Cooks article: if you want data abstraction, you shouldn't really provide any way to allow other objects/methods to access the internal representation. This somewhat also forbids getters as this is likely to lead to leaky abstraction, since more often than not programmers simply return the value of some data field, directly exposing the representation chosen. Now, please note that this does not necessarily follow from Cooks article, as it is possible to design getters in such a way that you can return whatever you want for a getter method, i.e., you can return a desired return type or an object satisfying a particular interface. For me, the relevant point here is the way of thinking about the kind of object at hand: do I want some behaviour (aka Cooks objects) or do I want a data sink. In the former case, and in line with what is suggested in the clean code book, it is arguably the best way to tell the object to do what is necessary rather than to inspect (get) the data it holds and do it externally in some other object/method. But even in the latter case, I think it is important to give great attention to hiding the internal representation from external access and to only allow very focussed access to the data itself. It could and has been argued that restricting the access to the stored data via getter methods is tedious (see e.g. the discussion in getters/setters/fuxors) and that allowing public access to members is allright, but looking at the issue from a data abstraction point of view it simply boils down to the question whether you want or need data abstraction or not.

Second, I've recently seen these two postings on the merits of the Zope Component Architecture: The emperors new clothes and the reply The success of the ZCA. Malthe asks why one should use the ZCA to override the use of a particular implementation with another instead of using some kind of reloading (or rather says that the latter is the preferrable approach). Relating this to Cooks article, Malthe could be paraphrased roughly as: we have ADTs all over the place and we only should allow only one implementation per ADT (this is what the type system would guarantee in other systems). If you want another implementation (of some interface, as Cook shows for his objects), you should reload the object defintion with the one you want. The use of the ZCA, however, is directly related to the very idea of object oriented programming in the way Cook defines it: you only have interfaces that are the relevant defining characteristics of objects (values) and hence, the use of the ZCA is the way to deal with multiple implentations in Zope (or Python). For me, all I can say is that I'm happy that the ZCA and hence the ability to easily intermingle multiple implementations is there (then again, with me reading computer science theoretic articles I'm arguably not of the angry web designer type whose benefit Malthe is arguing for).

There is another, more puzzling aspect of the article to me. After some considerations, I have to conclude that of all OO languages I happen to know, it's really only Java that seems to be object oriented in Cooks view of the world. This is because in Java, you can define a method to return objects satisfying an interface. In addition, in dynamically typed languages like Python, Ruby, or CLOS, you could try to come away with duck typing, but it's arguably only Python which tries to take it to the heart (for instance in CLOS, most values you're gonna deal with are non-CLOS values and you even have an ETYPECASE statement, which is a switch-statement on type distinction). Funny enough, Cook finishes his Smalltalk analysis with the statement that "one conclusion you could draw from this analysis is that the untyped lambda calculus was the first object-oriented language". But besides the point how some language is "more OO" than another, there is also to the point that in order to program truly object-oriented, you shouldn't (and in Cooks world really can't) rely on type checks, because the whole point of using objects as data abstraction is to rely on behaviour.

ObTitle: Dial M for Murder, "Fiction of her dreams"

Magic theatre not for everybody

nospam@example.com (Holger Schauer) — Thu, 22 Oct 2009 18:38:00 +0200

If you just stumbled over this blog entry searching for what the heck all that buzzing about agile methods amounts to, I may have unfortunate news for you: An agile approach to development might for a lot of people all over the world, but may be not for you. For starters, take a look at the agile manifesto and its principles. These have quite a lot of implications, directly translating into presuppositions about you, if you wanted to participate in an agile development project.

Now, perhaps you're the well communicating, team oriented developer type who happens to work in a surrounding which you respect and general feel well motivated. But real life experience with your typical geek or ex-geek-turned-professional-developer suggests that having a hard time with direct communication, face-to-face, happening very often with those pesky guys and girls you don't really like is a more likely scenario ("Eeek, those business and marketing people" anybody?). Now, you might say, baah, I know some communication is a necessary evil of professional software development, but maybe I can get away with my old habits of trading documents (specs, bug reports, etc.) against direct communication, since that is the trusted old way and what do you mean by face-to-face communication being more effective? Everybody hates those horrible meetings eating up all your time, right?

The point is that if you're wanting to go agile you should better soon adapt your view or else you won't see any of the promised benefits of all of those agile processes. Agile development is not only about using test driven design, timeboxing, iterations and releasing often. More than anything else, agility is about how people effectively interact with each other as quickly and direct as possible in order to come to solutions. To me, this basically boils down mainly to three issues: adaptability to (changing) situations, an open mind toward communication and taking responsibility for the project at hand.

Being lazy, sticking bone-headed to "trusted paths", not seeking confrontation when it's needed, avoiding communication because of being too busy "doing" is the anti-pattern to agility. You could keep a product and sprint backlog, present new features every other week to your customer and even use continuous integration and still would be muddling in pseudo agile, doomed to fail water. What I would suggest is that you should revisit the agile manifesto and think about what those principles might imply for you and your current behaviour. Let's try an example: If you find that you might end up pointing fingers at your customer (or product owner to use the terminology from Scrum) because he won't behave as demanded in the agile manifesto, you should reconsider if that fits in with what the agile manifesto demands of you: you wouldn't do the project any good. You would build up frontiers where there should be a single team including the target of your pointed finger. This is not to say that you shouldn't point out the defect, no communication about problems is another all too familiar reason for frustration and lack of commitment. What is necessary is direct communication and finding a way that works for all people involved.

Let me sum up this posting: if you're interested in agile approaches to software development, that's fine. If you don't feel comfortable about what this might imply for you, that's fine too. But it might also tell you that agile development approaches might not work for you.

Update: Irionically, I stumbled via Infoq about the one essential agile ingrediment which says all about it much nicer than I ever could have done, only that Mark Schumann shows the confidence that agile works for the most part whereas I hold the more pessimistic view that it's not going to work for some, but for exactly the same reasons.

ObTitle: Hermann Hesse, "Steppenwolf"

Version control and testing

nospam@example.com (Holger Schauer) — Fri, 24 Jul 2009 10:30:05 +0200

What have version control and testing to do with each other? Well, first of all, both are common virtues in the clean code community. What you'll find is that both virtues are important on their own ground: version control provides a safety guard in that you can roll back to prior versions if you accidently introduce problems in your code. Testing (automated unit tests) provides a safety guard, too, because you can do regression testing when you work with your code. These are both fine goals but seemingly have little to do with each other.

But in reality they do. For sake of argument, let's take a step back and assume that you have to work in an environment of several developers where neither of these things exists. What will you likely see? What we all have seen several years ago. Commented out code blocks, redundant and often misleading or outdated comments, timestamps with comments cluttered all over the code. And frightened developers that feared each minor change because of the miriad of subtle side effects it might have, let alone major changes to core components. It's an environment in which refactorings as welll as extensions are very hard and expensive, which results in frightened overworked developers and frustrated managers.

So, what happens when you introduce only one of those virtues? Say, we introduce version control. Now, every change gets documented, except that documenting every change requires, from the developers point of view, documentation at the wrong point. They can't see the documented changes and the reason for these changes in the source, they see it only in the version control system -- iff they add a change message with every change at all. Much more likely is that you will see commit messages such as "." or "bug fix", and the same old mess of timestamps, outdated comments and commented out code as before. Why is that? Because your developers are now not as frightened as they used to be (because they can now rely on the version control system to fall back to older versions), but they still have the same need to understand and document the code. And the commit log is both "too far away" from the code and "out of it's purpose" for this task: the commit log shouldn't document what the code is supposed to do, only when something was implemented to behave in a particular way.

This is where a development (unit) test suite comes into the picture: you document every required behaviour in tests. With every change to the code, you also update the test. As a developer, you can now look into your test suite to see what the code is supposed to do. Now developers will likely become much more confident with their changes, because they can run the tests and see what happens (hopefully next to immediately) without requiring time- and resource-consuming manual tests.

But what about documenting the changes to the code? Well, you should simply document any changes in the commit message of your version control system, because it's now no longer necessary to keep the entire version history in mind to understand what the current code state is supposed to do. You have the tests that tell you what the code should do. The commit log now only serves the purpose of documenting what has changed over time and is no longer required to understand what the code should do. So you don't have to keep the clutter in your code, resulting in much cleaner source code files.

Summary: Taken together, the whole of version control and testing adds up to more than a simple addition of their own values.

I guess I'll hang out my tears to dry

nospam@example.com (Holger Schauer) — Mon, 11 May 2009 08:55:33 +0200

There seems to be a roaring discussion about a talk by Robert "Uncle Bob" Martin at this years RailsConf, which mainly seems to be concerned with the question (raised by a quote appointed to Ward Cunningham) whether languages such as Smalltalk make it too easy to create a mess. Languages such as C++ would penalize the mess at least by longer compile times.

As Giles points out that's an old argument, which I too have heard too often in the past about dynamic languages, regardless whether we're talking Smalltalk, Ruby, Lisp, Python or Perl. And the anti-dote was always the current idiot language of the date, i.e. Visual Basic at its high time, Java or C# (for those who considered C or C++ to be too dangerous). Please note that I'm not at all saying that users of those languages are idiots, but those languages are particularly claimed to be usable by idiots, too. This approach to languages or to their classification does a great job of misrepresenting programmers regardless of the language of their choice. It makes the bold claim that there are programmers that are idiots and that it's possible to design a language that even those idiots can do their jobs. And the exaggeration is that its only good business sense to enforce the use of those languages for all development projects, because, as we all know, there are more idiots than intelligent programmers out there, right?

The only problem with this claim is that human stupidity is probably the only thing larger than the universe, if you make something idiot proof, nature will quickly make a better idiot. To put it a little less cynical, to err is human. And there is value in people making mistakes. As for instance, Tom DeMarco claims in his book "Slack" it's a particular good way to learn. Bob Martin tries to sell excessive testing (which isn't particular surprising, giving his background) as one safety guard to which James Robertson responds that this is not always necessary:

Yes, tests are useful. But, the debugger is not something to be feared. Rather, it's a great tool to be used in order to have the computer do all the memory work for you. I can get a lot more done by working with decent tools like the Smalltalk debugger than I can by assuming the doc is good and writing tests that just help me a whole lot less than you might think.

I can relate to this quote, being a fan of interactive development that languages like Lisp or Smalltalk made popular. But I think James is missing the point: tests are not only helpful during development. Sure, using an interactive approach helps getting the code run but does it help to come up with a clean design? Looking back at some code, I can assure you it doesn't, quite to the contrary it's a way to get a working solution quickly, not cleanly. This doesn't make the debugger or the REPL less valuable: using it you can quickly establish your tests, debug your tests, whatever. This is exactly where the value of dynamic languages shines and those "silver-bullet" idiot languages are still behind, regardless of their shiny IDEs.

But the real point summing up the discussion so far is the old Frederick Brooks quote: There is no silver bullet. It's good to be careful and to use the tools at your hands to help you avoid the big mistakes. As Paul Graham puts it for OOP: "For programs that would have ended up as spaghetti anyway, the object-oriented model is good: they will at least be structured spaghetti." But in the end you have to learn how to avoid the mess in the first place and no language and no tool can do that for you.

ObTitle: Dexter Gordon, "Ballads"

Testing and terminology confusion

nospam@example.com (Holger Schauer) — Fri, 06 Feb 2009 11:11:40 +0100

I may be rather late to the game, but over the last one and a half years, I've become quite addicted to writing tests during my development tasks. I've had wanted to dig into test-driven development for quite some time, but it was the seamless integration of Test::Unit, Ruby's unit testing module, in Eclipse that got me going initially. I then did some unit testing with Common Lisp packages and am currently heavily using pyunit and python doctests (mostly in the context of zope testing). Writing tests has become my second development nature: It gives you that warm fuzzy feeling that you have that little safety net while modifying code.

However, there are times when terminology comes along and gives you a headache. A terminology I've learned about during the last year is the difference between unit testing, integration tests and functional tests (for an overview see wikipedia on software testing). But as you can see for instance in this article on integration tests in Rails, it's not always easy to agree on what means what -- Jamis and/or the Rails community seem to have the integration/functional distinction entirely backwards from what, for instance, the Zope community (on testing) thinks.

Now, one might argue that terminology doesn't matter much given that you do write tests at all, but it's not so easy. For instance, if your "unit test" of a given class requires another class, is that still unit testing or is it integration testing? Does it even make sense to talk about unit-testing a class? A class on its own isn't that interesting after all, it's its integration and interoperation with collaborateurs were the semantics of a class and its methods become interesting. Hence, shouldn't you rather test a specific behaviour, which probably involves those other classes? And what now, if your code only makes sense when run on top of a specific framework (Zope, Rails, you name it)? Michael Feathers argues convincingly in his set of unit testing rules that any such tests are probably something else.

Ultimately these questions directly pertain to two aspects: code granularity and code dependencies -- and remember, test code is code after all. These are directly related, of course: if your code is very fine-grained, it's much more likely that it will also be much more entangled (although the dependency might be abstracted with the help of interfaces or some such, you still have the dependency as such). And as a consequence, your test code will have to mimick these dependencies. On the contrary, if your code blocks are more coarse-grained (i.e. cover a greater aspect of funcionality), you might have less (inter-)dependencies, but you won't be able to test functionality on a more fine-grained level. As Martin Fowlers excellent article Mocks aren't stubs discusses in detail, one way to loosen these connections between code and tests is to use mock objects or stubs. Fowlers article also made clear to me that I've used the term "mock object" wrongly in my post on mock objects in Common Lisp: dynamically injecting an object/function/method (as a replacement for a collaborator required for the "code under test") that returns an expected value means using a stub, not a mock -- another sign of not clearly enough defined terminology (btw, the terminology Fowler is using is that of G. Mercezaos xunit patterns book).

It's worth keeping these things apart because of their different impact on test behaviour: mocks will force you to think about behaviour whereas stubs focus on 'results' of code calls (or object state if you think in terms of objects being substituted). As a result, when you change the behaviour of the code under test (say you're changing code paths in order to optimize code blocks) this might (mocks) or might not (stubs) result in changes to the test code.

It's also worth thinking about mocks and stubs because they also shed a new light on the question of test granularity: when you're substituting real objects in either way, you're on your way to much more fine-grained tests, which implies that you loosen the dependency of your tests: You can now modify the code of your collaborateur class without the test for your code under test breaking. Which brings us back full circle to the distinction between unit tests and integration tests: you now might have perfect unit tests, but now you're forced to additionally tests the integration of all the bits and pieces. Otherwise you might have all unit tests succeed but your integrated code still fails. Given this relationship, it seems immediately clear that 100% test coverage might not be the most important issue with unit tests: you might have 100% unit test success, but 100% integration failure at the same time -- if you don't do continuous integration and integration tests, of course. Now what's interesting is that it might be possible to check test coverage on code paths, but it might not be easy to check integration coverage. I would be interested to learn about tools detailing such information.

Recently I had another aha moment with regard to testing terminology: Kevlin Henney's presentation at this years german conference on object oriented programming, the OOP 2009, on know your units: TDD, DDT, POUTing and GUTs: tdd is test driven development, of course. The other ones might be not so obvious: "guts" are just good unit tests and "pout" is "plain old unit testing". I saw myself doing tdd, but come to think of it, I'm mostly applying a combination of tdd, pout (after the fact testing) and ddt: defect driven testing. I find the introduction of a term for testing after the code has been written interesting because it provides a way to talk about how to introduce testing in the first place. Especially defect driven testing, the idea to write a test to pinpoint and overcome an erroneous code path, might be a very powerful way to introduce the habit of regularly writing (some) tests for an existing large code base. So you avoid the pitfall of never being able to test "all this lots of code because there is never the time for it" and you might also motivate people to try writing test before code. And on this level, it might at first not be that relevant to make the distinction between integration and unit tests to clear: start out with whatever is useful.

Core wars

nospam@example.com (Holger Schauer) — Thu, 08 Jan 2009 21:48:00 +0100

Drew Campsie announces ucw-core, the next generation of Uncommon Web, a continuation-based framework for web applications written in Common Lisp. It's the first major announcement since Drew took over development lead of ucw and it seems to focus on establishing a core library of basic technology on top of which other functionality can be implemented (e.g. functionality like those offered by the ucw-ajax). To me as a UCW user this sounds like great news, since I had the impression that ucw seemed to linger in a state of void over the last year or so.

But even more interesting is that ucw just seems to follow a certain trend. For instance, according to the announcement that Ruby on Rails and Merb will merge, one goal is that Rails will be retrofitted to make it easy to start with a “core” version of Rails (like Merb’s current core generator), that starts with all modules out, and makes it easy to select just the parts that are important for your app. The main aspect of such development (at least I hope so) seems to be best described by one of Django design goals): A fundamental goal of Django’s stack is loose coupling and tight cohesion. The various layers of the framework shouldn’t “know” about each other unless absolutely necessary. The python folks have even gone so far as to write down a web server gateway interface specification (WSGI, a quite minimalistic specification).

It will be interesting to see whether there's going to be even more unification, maybe even across language diffferences. I'm not too certain that such an unification would necessarily be a good aim; as always there are probably as many benefits as downsides to it. For instance, a common specification certainly lowers the learning curve for people switching frameworks while at the same time may stiffle further development. Then there's the fact that the different frameworks have different approaches which are certainly reflected in their APIs: for instance, Django explicitly tries to avoid magic at all cost while there's traditionally a lot of magic going on behind the scenes of a Rails application. I'm also not sure where exactly one should draw the line between common "core" functionality and additional "batteries". As an application designer it's probably most important that there is such a line at all so that you can easily pick your poison without having to buy everything.

You're wondering now

nospam@example.com (Holger Schauer) — Fri, 02 Jan 2009 09:18:33 +0100

Herb Sutter announces that Dr.Dobbs Journal is permanently suspending print publication and going web-only as of January 2009. I've used to be a long-time subscriber of DDJ -- several not-so-excited friends that recently helped moving into our new flat are my witnesses. Actually, I've stopped my subscription (and we're talking about a way to expensive subscription via a German importer, here) roughly one or two years ago, since I couldn't keep up with my reading but occasionally I've regret that I no longer receive it. Now, it seems to me like the last printed programming related magazine is going down the drain -- sorry, you Java and PHP magazines floating around, but your content isn't nearly as interesting as what DDJ had as it was ultimately the mix of different topic and opionated editorials that made DDJ so unique. Perhaps now I should hurry and finally buy the DDJ developer library dvd covering 21 years of DDJ articles. I'll keep an eye on what will happen to DDJ as a web publication.

python-mode vs. slime

nospam@example.com (Holger Schauer) — Wed, 10 Dec 2008 10:52:44 +0100

I've become a python programmer, too, lately, due to a job change. Python is a fine language so far, although to me it's mostly just like Ruby, though with even less functional flavour. However, just as with Ruby, I'm really missing slime, the superior lisp interaction mode for Emacs, when hacking python code. I could now start to write down a list of things I'm missing (which I've intended to do), however, Andy Wingo spares me the hassle, as he has just written an excellent article on slime from a python programmers view.

However, I would like to elaborate a little on the main difference for me: the client/server socket approach of slime. Let me briefly recapulate what this implies: slime consists of two parts, a client written in Emacs lisp and a server written in Common Lisp (AFAIK there is at least also an implementation for clojure, maybe also one for some scheme implementation). In order to use slime in it's full glory, it's hence required that you have a common lisp process running which in turn runs the slime server part. If you now fire up slime, you'll get an interaction buffer over which you can access the REPL of the lisp process, which in python would be the interpreter prompt. You can then interact with the lisp process, evaluating pieces of code from your lisp source code buffer directly in the connected lisp process. What is incredibly useful for me is that you can not only start a new lisp process but also connect to an already running lisp process, given that it has the slime server started (this is obviously mainly useful if the lisp implementation you use has multi-threading capabilities). I use it to connect to a running web server application, which I can then inspect, debug and modify. Modification includes redefinition of functions, macros and classes, which of course is also a particular highlight of Common Lisp. I would like to cite a comment of the reddit user "fionbio" he made wrt. to the linked article: In fact, Python language wasn't designed with lisp-style interactive development in mind. In CL, you can redefine (nearly) anything you want in a running program, and it just does the right thing. In Python, there are some problems, e.g. instances aren't updated if you modify their class. Lisp programmers often, though not always, refer to various things (functions, classes, macros, etc.) using symbols, while Python programs usually operate with direct references, so when you update parts of your program you have much higher chances that there will be a lot of references to obsolete stuff around.

To complement Bill clementsons excellent article series on slime a little, I'm going to describe how I'm using/configuring python-mode to make it match my expectations a little closer. Essentially I would like to access my python process just as I would with slime/Common Lisp, but that's not possible. The reason, btw., is nearly unchanged: I need to code on a web server app (written in Zope) which may not even run on the same machine I'm developing on. Let's first cover the simple stuff: To enable a reasonable command interface to the python interpreter, I require the ipython emacs library. If the python interpreter runs locally, I also use py-complete, so that I can complete my code at least a little. Unfortunately, this breaks when the python interpreter doesn't run locally, because the py-complete needs to setup some things in the running python process, which it does by writing to a local temp file and feeding it to the python process. Unfortunately, the code in py-complete lacks customizability, i.e., you can't specify where that temp file should be located -- I should be able to come up with a small patch in the near future, which I will add below. Finally, I also require doctest-mode as a support for writing doctests, but that's not really relevant.

Now, on to the more involved stuff: I introduce some new variables and a new function py-set-remote-python-environment, which uses the those variables to do a remote call (via ssh) to python. This at least allows me to do things like setting py-remote-python-command to "/home/schauer/zope/foo-project/bin/instance" and py-remote-python-command-args to "debug", so that I can access a remote debug shell of my current zope product. That alone will only allow me to fire up and access the remote python, so I could now develop the code locally, having it executed remote. More typical though is that you would also want to keep the code on the remote machine, too: for this I use tramp, a package for remotely accessing files/directories from within emacs. In combination, this allows me to edit and execute the code on the remote machine. It is still nowhere near what is possible with slime, but at least it allows me to persue my habit of incremental and interactive development from within my usual emacs installation (i.e., it doesn't require me to deal with any Emacs related hassle on the remote machine).


;;; python-stuff.el --- python specific configuration

(when (locate-library "ipython")
  (require 'ipython))

(when (locate-library "doctest-mode")
  (require 'doctest-mode))

(defvar py-remote-connect-command "ssh"
  "*Command for connecting to a remote python, typically \"ssh\"")
(defvar py-remote-connection-args '("user@remotemachine")
  "*List of strings of connection options.")
(defvar py-remote-python-command "python"
  "*Command to execute for python")
(defvar py-remote-python-command-args '("-i")
  "*List of strings arguments to be passed to `py-remote-python-command`.")
(defvar py-remote-python-used nil
  "Remember if remote python is used.")

(defun py-set-remote-python-environment ()
  (interactive)
  (let ((command-args (append py-remote-connection-args 
			      (list py-remote-python-command)
			      py-remote-python-command-args)))
    (setq py-python-command py-remote-connect-command)
    (setq py-python-command-args command-args))
  (setq py-remote-python-used t))

(when (locate-library "py-complete")
  (autoload 'py-complete-init "py-complete")
  (defun my-py-complete-init ()
    "Init py-complete only if we're not using remote python"
    (if (not py-remote-python-used)
        (py-complete-init)))
  (add-hook 'python-mode-hook 'my-py-complete-init))

Unit tests with mockups in Lisp

nospam@example.com (Holger Schauer) — Tue, 11 Nov 2008 19:41:55 +0100

One of the bigger practical problems with unit testing is isolating the test coverage. Say, you want to test a piece of code from the middle (business) layer. Let's assume further the piece of code under consideration makes some calls to lower level code to retrieve some data. The problem of test coverage isolation is now that if you "simply" call your function, you are implicitly also testing the lower level code, which you shouldn't: if that lower level code gets modified in an incorrect way, you would suddenly see your middle level code fail although there was no change made to it. Let's explore ways to avoid the problems in Common Lisp.

There is a very good reason why you would also want to have such test dependencies to ensure your middle level code still works if the lower level code is extended or modified. But that is no longer unit testing: you are then doing so-called integration tests which are related, but still different beasts.

Now, I was facing exactly the typical dreaded situation: I extended an application right above the database access layer which had not seen much tests yet. And of course, I didn't want to go the long way (which I will eventually have to go anyway) and set up a test database with test data, write setup and tear-down code for the db etc. The typical suggestion (for the xUnit crowd) is to use mock objects which brings us finally on topic.
I was wondering if there are any frameworks for testing with mock objects in Lisp, but a quick search didn't turn up any results (please correct me if I've missed something). After giving the issue a little thought, it seemed quite clear why there aren't any: probably because it's easy enough to use home-grown solutions such as mine.

I'll use xlunit as the test framework, but that's not relevant. Let's look at some sample code we'll want to test:


(defun compare-data (data &connection)
  (let ((dbdata (retrieve-data-by-id (id data))))
    (when (equal (some-value data)
                (some-db-specific-value dbdata))
         t)))

The issue is with retrieve-data-by-id which is our interface to the lower level database access.
And note that we'll use some special functions on the results, too, even if they may just be accessors.
Let's assume the following test code:


(use-package :xlunit)

(defclass comp-data-tc (test-case)
  ((testdata :accessor testdata :initform (make-test-data))))

(def-test-method comp-data-test ((tc comp-data-tc))
  (let ((result (compare-data (testdata tc))))
    (assert-equal result t)))

Now the trouble is: given the code as it is now, the only way to succeed the test is to make sure that make-test-data returns an object whose values match values in the database you're going to use when compare-data get's called. You're ultimately tying your test code (especially the result of make-test-data) to a particular state of a particular database, which is clearly unfortunate. To overcome that problem, we'll use mock objects and mock functions. Let's define a mock-object mock-data and a mock-retrieve-data function, which will simply return a single default mock object.


(defclass mock-data ()
  ((id :accessor id :initarg :id :initform 0)
   (val :accessor some-db-specific-value :initarg :val :initform "foo-0"))))

(defun mock-retrieve-data (testcase)
  (format t "Establish mock for retrieve-data")
  (lambda (id)
      (format t "mock retrieve-data id:~A~%")
      (find-if #'(lambda (elem
			(when (equal (id elem) id)
			  elem))
                    (testdbdata testcase))))

Why that mock-retrieve-data returns a closure will become clear in a second, after we've answered the question how these entirely different named object and function can be of any help. The answer lies in CLs facility to assign different values (or better said) definitions to variables (or better said to function slots of symbols). What we'll do is to simply assign the function definition we've just created as the function to use when retrieve-data is going to be called. This happens in the setup code of the test case:


(defclass comp-data-tc (test-case)
  ((testdata :accessor testdata :initform (make-test-data))
   (testdbdata :accessor testdbdata)
   (func  :accessor old-retrieve-func)))

(defmethod set-up ((tc comp-data-tc))
  ; set up some test data
  (dotimes (number 9)
    (setf (testdbdata tc)
             (append 
                  (list 
                      (make-instance 'mock-data 
                            :id number 
                            :value (concatenate 'string "value-" number)))
                 (testdbdata tc))))
  ; establish our mock function
  (when (fboundp 'retrieve-data)
    (setf (old-retrieve-func tc)
             (fdefinition 'retrieve-data))))
  (setf (fdefinition 'retrieve-data)
           (mock-retrieve-data tc)))

(defmethod tear-down ((tc comp-data-tc))
   ; After the test has run, re-establish the old definition
   (when (old-retrieve-func tc)
      (setf (fdefinition 'retrieve-data)
           (old-retrieve-func tc))))

You can now see why mock-retrieve-data returns a closure: by this way, we can hand the data we establish for the test case down to the mock function without resorting to global variables.

Now, the accessor fdefinition comes in extremely handy here: we use it to assign a different function definition to the symbol retrieve-data which will then be called during the unit-test of compare-data.


..Establish mock for retrieve-data
mock retrieve-data id: 0
F
Time: 0.013


There was 1 failure: ...

There is also symbol-function which could be applied similarly and which might be used to tackle macros and special operators. However, the nice picture isn't as complete as one would like it: methods aren't covered, for instance. And it probably also won't work if the function to mock is used inside a macro. There are probably many more edge cases not covered by the simple approach outlined above. Perhaps lispers smarter than me have found easy solutions for these, too, in which case I would like to learn more about them.

The release warp

nospam@example.com (Holger Schauer) — Fri, 17 Oct 2008 08:24:47 +0200

The Release Warp Lyrics

It's astounding, time is fleeting
Madness takes its toll
But listen closely, not for very much longer
I've got to keep control

I remember doing the Release Warp
Drinking those moments when
The blackness would hit me and the void would be calling
Let's do the release warp again...
Let's do the release warp again!

It's just a bug to the left
And then an error to the right
With your hands on your keys
You bring your fixes in tight
But it's the pelvic thrust that really drives you insane,
Let's do the Release Warp again!

It's so dreamy, oh fantasy free me
So you can't find me, no not at all
In another dimension, with capitalistic intention
Well-secluded, I see all
With a bit of a mind flip
You're there in the bug slip
And nothing can ever be the same
You're spaced out on sensation, like you're under sedation
Let's do the Release Warp again!

Well I was typing down the plan just a-having a think
When a snake of a marketeer gave me an evil wink
He shook me up, he took me by surprise
He had a business plan and the devil's eyes.
He stared at me and I felt a change
Time meant nothing, never would again
Let's do the Release Warp again!

(... with sincere apologies to The Rocky Horror Picture Show by Richard O'Brien)

UCW leadership change

nospam@example.com (Holger Schauer) — Wed, 30 Apr 2008 22:06:42 +0200

In case you're interested in continuation-based web development with Common Lisp, it might be of interest to learn that Drew Campsie has voluntered to take over UCW development. Marco Baringer, initiator and maintainer of UncommonWeb (aka UCW) seemed somwhat relieved that finally somebody with more time on his hand steps up to take over.

That Marco hadn't had the time to keep pushing UCW is quite obvious from the mailing list archive. Lately Attila Lendvai seemed to me (an innocent external observer) to had pushed UCW foreward, however, as he Attila announced, he wants to go into a different direction than Drew.

I've been using ucw-dev during the last two years and the current situation really called for a change, in my opinion. UCWs main problem, as I observe it, is lacking documentation and clearly stated goals which way UCW will go. The automatically extracted documentation doesn't contain information how the bits fit together and all tutorials are a) incomplete, b) outdated as far as they are targetting ucw-dev, whereas development has mainly happened in ucw-ajax. However, it was not all clear that ucw-ajax really would become the main line of UCW.

I like UCWs component architecture a lot and will eagerly follow what is going to happen. With two prominent developers such as Marco and Attila more or less leaving the project, it will be interesting to see with how much activity development will happen from now on.

Escaping from sql-reader-syntax in CL-SQL

nospam@example.com (Holger Schauer) — Fri, 22 Feb 2008 12:04:50 +0100

This post is mainly a reference post about a particular topic whose solution wasn't immediately obvious to me from the docs to CL-SQL. Using CL-SQL with (enable-sql-reader-syntax), I had written a routine that looks basically likes this:


(defun data-by-some-criteria (criteria &key (dbspec +db-spec+)  (dbtype +db-type+))
  (with-database (db dbspec :database-type dbtype :if-exists :old)
    (let (dbresult)
      (if criteria
	  (setq dbresult
		(select 'some-model 'other-model
			:where [and [= [some.criteria] criteria]
			            [= [some.foreign_id] [other.id]]]
			:order-by '([other.name] [some.foreign_id] 
				          [year] [some.name])
			:database db))
          (setq dbresult
		(select 'some-model 'other-model
			:where [and [null [some.criteria]]
			            [= [some.foreign_id] [other.id]]]
			:order-by '([other.name] [some.foreign_id] 
				          [year] [some.name])
			:database db))
      (when dbresult
	(loop for (some other) in dbresult
	      collect some)))))

This is ugly because the only difference between those two select statements is the check for the criteria, but I had no idea how to combine the two select statements into one, because it's not possible to embed lisp code (apart from symbols) into an sql-expression (i.e. the type of arguments for :where or :order etc.).

With the next requirement things would become far worse: The order-by statement needs to get more flexible so that it is possible to sort results by year first. Given the approach shown above this would result in at least four select statements, which is horrible. So, naturally I wanted a single select statement with programmatically obtained :where and :order-by sql expressions.

Step 1: It occured to me that it should be possible to have the arguments in a variable and simply refer to the variable. E.g., using a more simple example:


(let (where-arg)
  (if (exact-comp-needed)
      (setq where-arg '[= [column] someval])
      (setq where-arg '[like [column] someval]))
  (select 'model :where where-arg))

So I could now have my two different where-args and two different order-args and use a single select statement. Main problem solved.

Step 2: But for the :where arg in my original problem, only a small fraction of the sql-expression differs. So how do I avoid hard coding the entire value of where-arg? How can I combine some variable part of an sql-expression with some fixed parts? I.e, ultimately I want something like:


(let (comp-op where-arg)
  (if (exact-comp-needed)
      (setq comp-op '=)
      (setq comp-op 'like))
  (setq where-arg '[ <put comp-op here> [column1] someval])
  (select 'model :where where-arg))

But with CL-SQL modifying the reader, there seems to be no way to make <put comp-op here> work. I didn't knew how to get the usual variable evaluation into the sql-expression, or how to escape from CL-SQL's sql-reader-syntax to normal lisp evaluation.

Somewhere in the back of my head where was that itch that CL-SQL might offer some low-level access to sql expressions. And indeed it does. There are two useful functions, sql-expression and sql-operation. sql-operation "returns an SQL expression constructed from the supplied SQL operator or function operator and its arguments args" (from the cl-sql docs), and we can supply the operator and its arguments from lisp -- which is exactly what I want.

Now, the nice thing is that it's easy to mix partly handcrafted sql expressions with CL-SQL special sql syntax constructs that will be automatically handled by the reader (if you enable it only via enable-sql-reader-syntax, of course). I.e., for <put comp-op here> we can use sql-operation, but the rest stays essentially the same:


(let (where-arg)
  (if (exact-comp-needed)
      (setq where-arg (sql-operation '= [column1] someval))
      (setq where-arg (sql-operation 'like [column1] someval)))
  (select 'model 'other-model  :where where-arg))

Now, coming back to my original problem, based on this approach I can split out the common part of the :where and :order arguments and combine those with the varying parts as needed and hand them down to a single select statement. Problem solved.

Interactive development gets more popular

nospam@example.com (Holger Schauer) — Fri, 25 Jan 2008 10:14:07 +0100

It's interesting to see that more and more environments for interactive development are popping up. If you're wondering what I'm refering to, it's what has traditionally been labelled as an interpreter: you get some kind of prompt like in a command shell and can directly enter and run program constructs. In Ruby, irb is the tool to use, whereas most, if not all, Common Lisp systems provide you a REPL whenever you start-up the system. REPL is an abbreviation for read-eval-print-loop and this pretty much sums up how interactive development works: you enter some code into the system, and as soon as you hit enter (i.e. finish up the particular line of code), the system will read and evaluate it ("interpret" it, although CL systems may also compile it automatically), presenting you with the results.

To me, this seems like more people come to understand that the classic edit-compile-debug cycle of traditional compiled languages like C isn't particular well-suited to a bottum-up programming style, as advocated by the agile programming hype. Where is the beef? I've been told that the reason why the asian folk don't use forks and knives when dining stems from the opinion that when dining one merely wants to eat, while all handcrafting (like slicing the meat) belongs to the a-priori cooking. The edit-compile-debug cycle tends to produce code more akin to the western steak style, while interactive development tends to produce smaller code parts, all well prepared (read: tested) from the beginning, so you don't end up with bloody interiors as easily. That's because of two reasons, I think: In an interactive environment testing a piece of code is cheap and easy. Hack up a small routine and enter it. Next step: call the routine with some sample data. No need to compile. In case there's an error, you'll be thrown into the debugger right away. With the traditional edit-compile-debug cycle, you're gonna edit the code, save it, call the compiler, edit another piece of code testing it, compile that, too, run that and only then you'll see if it succeeds. This might scare developers from running such tests to often and in result to produce larger routines/code blocks. Second reason: in the interactive environment you'll typically have somewhat less comfort then in a full-blown editor. So I'm arguing that the slight discomfort will help you in keeping things small because then you'll not gonna need far reaching edit support.

There is, however, a price to pay: For starters, because your interactive system provides you with a single environment during development, it's easy to mess it up. That's different with the traditional edit-compile-debug cycle where every time you compile you run the code in a clean environment. This means, old code refering to refactored functions will result in (usually: compilation) errors whereas in interactive development you may miss some spot in need for modification because the old refactored code is still around.
Second, when fiddling with some functions, it may happen that one hacks up several variants, maybe with only slightly differing names or argument lists, so it may happen you forget some crucial code when finally saving your carefully crafted solution (where saving here typically means copy it over to some editor, not saving an image as you may do in Smalltalk or Lisp). Another thing is that one has to be careful not to confuse bottum-up interactive development with hacking without thinking or design-less programming. This threat is all to common because with an interactive environment it's just so easy to get started. Unfortunately, without thinking you'll soon end up with problems one and two and even worse, it's likely that after a little while nobody (including you, the author) will be able to maintain the resulting code. But if you're aware of the potential pitfalls, interactive development is a nice way to do explorative programming which avoids a lot of time consuming labour you have to do with the traditional edit-compile-debug cycle.

As a side-note, environments offering incremental compilation like Eclipse aim at a similar goal, but only go half the way as they only take care of the edit-compile part. Only when one combines that with a test-driven development style you arrive at a similar point, because now the test code, when automatically run, will take care of immediately running the code in question (the eval/print part as far as testing the code is concerned).

To finally come to the reason for this blog post, I was pleased to learn about the Perl Console, which is not the same as the Perl debugger. I'm eager to try it out, although I'm sure that interactive development with a repl and the executable brain dump that Perl code usually is, is a highly dangerous mix.

Update: There is also a series of articles about implementing a REPL in Perl which may be of interest.

Twentieth century boys

nospam@example.com (Holger Schauer) — Tue, 22 Jan 2008 19:04:00 +0100

This is just a minor rambling inspired by a recent thread on cll about the ups and downs of using Rails. What I find really overwhelming is the lack of proofs that other (typically Lisp-based) frameworks really have so much benefit over the established more traditional frameworks (e.g. Rails, Zope, plain PHP, ...). All I can see is starter documents, tutorials etc. that show how to program yet another blog or reddit clone in some particular framework. I miss fair comparisons and detailed discussions why and where exactly that one particular way of doing things has benefits. And especially with regard to those reddit clone prototypes: AFAICT, reddit was far beyond implementing some half-baked prototype (in Lisp) when they decided to switch (to Python). Yes, for demo purposes reinventing the wheel with some unround edges might be acceptable, but in reality only rolling wheels will be sold and this is where the challenge is.

I believe that while it's nice that frameworks help with implementing your 500 line application, they really need to show their value with large applications. Where is a discussion of a large on-line shopping system in, say, UCW and CL-SQL, with secure shopping over SSL, login handling, input validation, distribution over multiple servers etc.? Now, I'm not suggesting that UCW, Weblocks etc. can't be used to build such websites, but it would be nice to see a much more in-depth explanation/discussion/comparison of how to do it. That would also help increase the credibility of Lisp or at least, of lispers discussing (other) web frameworks.

CLOS Video in German/auf Deutsch

nospam@example.com (Holger Schauer) — Thu, 20 Dec 2007 00:40:02 +0100

In case you're interested in object-oriented programming in Lisp and happen to speak German, you'll be happy to learn that Pascal Costanza of ContextL and AspectL fame (apart of his postings to cll), has given a talk about CLOS, the Common Lisp Object System at the Hasso-Plattner Institut, Potsdam. And while we're at it, Pascal also has a very interesting blog post about the origins of the advice facility.

German:
Falls Dich, lieber Leser, interessiert, wie man objekt-orientiert in Lisp programmiert, solltest Du Dich darüber freuen, dass Pascal Costanza, bekannt durch ContextL und AspectL (abgesehen von seinen Postings in cll), einen Vortrag in Deutsch über CLOS, dem Common Lisp Object System, am Hasso Plattner Institut in Potsdam gehalten hat. Und weil wir gerade dabei sind, Pascal hat außerdem einen sehr interessanten Blogeintrag über die Geschichte von 'advice'.