A view from the hill

There seems to be a roaring discussion about a talk by Robert "Uncle Bob" Martin at this years RailsConf, which mainly seems to be concerned with the question (raised by a quote appointed to Ward Cunningham) whether languages such as Smalltalk make it too easy to create a mess. Languages such as C++ would penalize the mess at least by longer compile times.

As Giles points out that's an old argument, which I too have heard too often in the past about dynamic languages, regardless whether we're talking Smalltalk, Ruby, Lisp, Python or Perl. And the anti-dote was always the current idiot language of the date, i.e. Visual Basic at its high time, Java or C# (for those who considered C or C++ to be too dangerous). Please note that I'm not at all saying that users of those languages are idiots, but those languages are particularly claimed to be usable by idiots, too. This approach to languages or to their classification does a great job of misrepresenting programmers regardless of the language of their choice. It makes the bold claim that there are programmers that are idiots and that it's possible to design a language that even those idiots can do their jobs. And the exaggeration is that its only good business sense to enforce the use of those languages for all development projects, because, as we all know, there are more idiots than intelligent programmers out there, right?

The only problem with this claim is that human stupidity is probably the only thing larger than the universe, if you make something idiot proof, nature will quickly make a better idiot. To put it a little less cynical, to err is human. And there is value in people making mistakes. As for instance, Tom DeMarco claims in his book "Slack" it's a particular good way to learn. Bob Martin tries to sell excessive testing (which isn't particular surprising, giving his background) as one safety guard to which James Robertson responds that this is not always necessary:

Yes, tests are useful. But, the debugger is not something to be feared. Rather, it's a great tool to be used in order to have the computer do all the memory work for you. I can get a lot more done by working with decent tools like the Smalltalk debugger than I can by assuming the doc is good and writing tests that just help me a whole lot less than you might think.

I can relate to this quote, being a fan of interactive development that languages like Lisp or Smalltalk made popular. But I think James is missing the point: tests are not only helpful during development. Sure, using an interactive approach helps getting the code run but does it help to come up with a clean design? Looking back at some code, I can assure you it doesn't, quite to the contrary it's a way to get a working solution quickly, not cleanly. This doesn't make the debugger or the REPL less valuable: using it you can quickly establish your tests, debug your tests, whatever. This is exactly where the value of dynamic languages shines and those "silver-bullet" idiot languages are still behind, regardless of their shiny IDEs.

But the real point summing up the discussion so far is the old Frederick Brooks quote: There is no silver bullet. It's good to be careful and to use the tools at your hands to help you avoid the big mistakes. As Paul Graham puts it for OOP: "For programs that would have ended up as spaghetti anyway, the object-oriented model is good: they will at least be structured spaghetti." But in the end you have to learn how to avoid the mess in the first place and no language and no tool can do that for you.

ObTitle: Dexter Gordon, "Ballads"

It's interesting to see that more and more environments for interactive development are popping up. If you're wondering what I'm refering to, it's what has traditionally been labelled as an interpreter: you get some kind of prompt like in a command shell and can directly enter and run program constructs. In Ruby, irb is the tool to use, whereas most, if not all, Common Lisp systems provide you a REPL whenever you start-up the system. REPL is an abbreviation for read-eval-print-loop and this pretty much sums up how interactive development works: you enter some code into the system, and as soon as you hit enter (i.e. finish up the particular line of code), the system will read and evaluate it ("interpret" it, although CL systems may also compile it automatically), presenting you with the results.

To me, this seems like more people come to understand that the classic edit-compile-debug cycle of traditional compiled languages like C isn't particular well-suited to a bottum-up programming style, as advocated by the agile programming hype. Where is the beef? I've been told that the reason why the asian folk don't use forks and knives when dining stems from the opinion that when dining one merely wants to eat, while all handcrafting (like slicing the meat) belongs to the a-priori cooking. The edit-compile-debug cycle tends to produce code more akin to the western steak style, while interactive development tends to produce smaller code parts, all well prepared (read: tested) from the beginning, so you don't end up with bloody interiors as easily. That's because of two reasons, I think: In an interactive environment testing a piece of code is cheap and easy. Hack up a small routine and enter it. Next step: call the routine with some sample data. No need to compile. In case there's an error, you'll be thrown into the debugger right away. With the traditional edit-compile-debug cycle, you're gonna edit the code, save it, call the compiler, edit another piece of code testing it, compile that, too, run that and only then you'll see if it succeeds. This might scare developers from running such tests to often and in result to produce larger routines/code blocks. Second reason: in the interactive environment you'll typically have somewhat less comfort then in a full-blown editor. So I'm arguing that the slight discomfort will help you in keeping things small because then you'll not gonna need far reaching edit support.

There is, however, a price to pay: For starters, because your interactive system provides you with a single environment during development, it's easy to mess it up. That's different with the traditional edit-compile-debug cycle where every time you compile you run the code in a clean environment. This means, old code refering to refactored functions will result in (usually: compilation) errors whereas in interactive development you may miss some spot in need for modification because the old refactored code is still around.
Second, when fiddling with some functions, it may happen that one hacks up several variants, maybe with only slightly differing names or argument lists, so it may happen you forget some crucial code when finally saving your carefully crafted solution (where saving here typically means copy it over to some editor, not saving an image as you may do in Smalltalk or Lisp). Another thing is that one has to be careful not to confuse bottum-up interactive development with hacking without thinking or design-less programming. This threat is all to common because with an interactive environment it's just so easy to get started. Unfortunately, without thinking you'll soon end up with problems one and two and even worse, it's likely that after a little while nobody (including you, the author) will be able to maintain the resulting code. But if you're aware of the potential pitfalls, interactive development is a nice way to do explorative programming which avoids a lot of time consuming labour you have to do with the traditional edit-compile-debug cycle.

As a side-note, environments offering incremental compilation like Eclipse aim at a similar goal, but only go half the way as they only take care of the edit-compile part. Only when one combines that with a test-driven development style you arrive at a similar point, because now the test code, when automatically run, will take care of immediately running the code in question (the eval/print part as far as testing the code is concerned).

To finally come to the reason for this blog post, I was pleased to learn about the Perl Console, which is not the same as the Perl debugger. I'm eager to try it out, although I'm sure that interactive development with a repl and the executable brain dump that Perl code usually is, is a highly dangerous mix.

Update: There is also a series of articles about implementing a REPL in Perl which may be of interest.

Joey seems to have inspired a "list of programming languages you know"-contest over on planet.debian.org, so this is mine:

1. Basic (forgotten)
   I've started using Basic on the Commodore C64 and have used it considerably, apart from the intrinsics of peek and poke, which some would claim was what Basic on the C64 was all about. I also used Basic on the Apple II at school, but was spared any further need to use Basic.
   
2. Cobol (forgotten)
   I remember my old computer science teacher bringing white and green prints on endless paper and teaching us the joys of Cobol. Having used Basic before, it looked somewhat suspicious but not too alien. Fortunately, we only had to solve some simple problems on paper, as the Apple II only had .... DBase II, I think.
   
3. Pascal (forgotten)
   (Turbo) Pascal was the language of choice in the first semester at the University of Koblenz at the start of the nineties. I've did what I had to do in it, but never really liked it. From the occasional look I throw in Wirth's algorithms book I know I can still read it, but the language looks just as baroque as the copy of the book I have.
   
4. Prolog (forgetting)
   I've really done a lot in Prolog, being first exposed in the second semester to it. I remember to marvel at some guys who grokked the language at first usage in the initial course -- to me, having used only imperative languages up to then, it was a mistery. I still think Richard A.O'Keefe's book "The Craft of Prolog" is among the best CS books.
   
5. C/C++ (forgetting)
   About 1992, I somehow got interested in object-oriented programming. Free software was still unknown to me, so I bought a student edition of Turbo C++ and some Addison-Wesley book and of I went inventing another wheel. At the time, I also had to do some C hacking for some course at the university, involving MS Windows programming. I've used C over time in several smaller projects, but would never call myself a C hacker.
   
6. Assembler (forgotten)
   My first exposure to Assembler was with some book for the C64. I was very interested but understood next to nothing. I should program some I/O chip directly instead of just using "load $,8,1"? WTF? Next look was in University, of course. I still know enough bits to be able to grok more or less the output of (disassemble 'foo) of some (tiny) foo-function in CL on an i386-style machine, but that's about it.
   
7. Mercury (had interest in)
   Mercury seemed like a nice successor to Prolog, adding several features of functional languages. I read several manuals, had several installations and never used it for anything.
   
8. Life (had interest in)
   Another possible successor to Prolog, which I've learned about at a summer school for language, logics and information. That's about all I remember.
   
9. XLisp (forgotten)
   My first exposure to Lisp in the context of a course for computational linguistics. The only thing worse than the editor used in the system was the tutor giving the course. A horror of parenthesis is about the only thing I remember.
   
10. Shell scripting (daily usage)
    I think since 1992 or 93 I've been using Unix systems and shell scripts. I no longer can remember whether our Suns dumped you into csh or sh or whatever, but since I've been using Linux systems (since 94), I've been using bash and written bourne shell scripts. Of course, I've also written the occasional csh script abdomination, but "Csh programming considered harmful" was an early lecture.
    
11. Emacs Lisp (forgetting)
    I remember seeing one of my instructors doing nearly miracles with XEmacs and wondering how his fingers could be that flexible -- I know that now. I've written several small utilities in Elisp and keep using it occasionally.
    
12. Common Lisp (improving)
    When I started a student job in 1995, I also started using Common Lisp, using Steele's CLtL2. Having used Elisp before, I no longer was worried about parenthesis. I still keep finding areas I didn't know or used before. And it's a language large enough that keeps you re-learning tricks you thought you knew. Which didn't stop me from giving a course on CL, which in hindsight didn't have the right focus: instead of showing that CL is just as capable of doing whatever you expect, it's also capable of doing you won't expect.
    
13. Smalltalk (interested)
    At the computational linguistics lab in Freiburg, the parser system was implemented in Smalltalk, so when I started there I took a small course in Smalltalk. As I worked more on the knowledge representation side of the system, I only rarely used it, though, so rare indeed, that I never got to do anything serious with the language. I still can read Smalltalk despite it's somewhat unusal syntax and have to do that even sometime.
    
14. Tcl (forgotten)
    We wanted to extend a linguistic annotation written in Tcl, so I bought a book and started digging through the code. Soon enough a student took over the job (and did it rather well), so I started the tiny bits I've learned quite soon. I remember finding the concept that every thing is a string to be rather messy and still see it as a lame excuse for dynamic/weak typing.
    
15. Perl (static)
    For nearly ten years, I had been using shell and/or Emacs to solve any data mangling tasks when I finally gave in and had a closer look at Perl. I still find it amazing how quick and dirty it's possible to solve problems in Perl, but on the other hand I've given up Perl for more complex tasks that require some thought and design, since Perl's syntax and language idiosyncrasies keep leading me to programs I don't even like myself.
    
16. XSLT (improving)
    At about the same time I started using Perl, I also needed to transform a set of XML documents and XSLT was all over the media, so I had a closer look. I dislike the noisy syntax of XSLT and particular the intermix with output elements and it's fever for being a "clean" functional language. But the concept of XPath and the idea of traversing a document via axis I find very compelling.
    
17. make (static)
    For a long time the power of make didn't occur to me. A colleague of mine, however, used it to implement a nifty workflow for document processing, of which I learned a lot of tricks when I needed to mimic it for a different task.
    
18. Ruby (improving)
    When I finally was fed up with Perl, I looked around to find Ruby as a nice halfway compromise between Common Lisp and Perl. It's still fancy and geeky and with Ruby on Rails has a lot of buisness ballyhoo around it, so I hope it will see more use in my current surrounding.
    
19. Java (not interested)
    Like everybody else, I couldn't avoid looking at Java. It seems to be the Cobol of today and as such is the job guarantee today. But Java looks so boring, I can hardly stand it. And this with regard to both the language and the results produced with it. Sigh But still, it's likely that someday I'll going to buy another book (that would be my third Java book, the first being O'Reilly's Java in a nutshell for Java 1.1 in maybe 1997 or so?), since just like Cobol at it's time, avoiding Java is hard ...
    
20. Python (not interested)
    When Python 1.4 hit the scene, there was quite some bubble around it. It burst quickly and as my latest review of the language reveiled last year, it still doesn't seem interesting.
    

Missing in this list is stuff like Scheme, Dylan, Oz, Oberon, Eiffel, Haskell and probably some other "tiny" languages which I looked at at some point in time and really didn't ever took a second look. Of those, I may take another look at Haskell in the future, but currently I have no concrete plans in doing so.

I love the Unix toolbox:

[elendil->Get_the_gore]mp3info -p "%S\t%m:%02s\t%t\n" * | perl -ne '{ $time=$time+$1 if (/^(\d+)[ \t]+(.*)$/); print $2."\n"; } END { print "Total seconds: $time\n"; $min=$time/60; $secs=$time%60; print(sprintf("Total time: %d:%d\n",$min,$secs)); }'
2:10 Fox in a box
3:16 Loaded heart
2:53 All grown up
3:07 Pleasure unit
2:51 Where evil grows
3:16 Casino
2:38 Don't cry
2:24 Mary Ann
3:48 You lied to me before
1:50 So sophisticated
2:50 Little baby
2:44 Sweet potato
2:56 Voodoo doll
1:51 Hammer stomp
Total seconds: 2314
Total time: 38:34


Printing out the track number and the track title works only if the file contains an ID3 tag, of course.

For a review, I needed to get the track list of a given CD. As the track list wasn't available via CDDB, I went to some large online store and found the tracklist. I need to convert it to XML, though. The original data I fetched looks like so:

1. Fox In A Box
2. Loaded Heart
3. All Grown Up
4. Pleasure Unit
...

whereas I need:

<li id="1">Fox In A Box</li>
<li id="2">Loaded Heart</li>
<li id="3">All Grown Up</li>
...

After cutting the original data to my Emacs, writing out a simple file and using Perl for that simple transformation seemed just gross. In the past, I've been an Emacs hacker. But no more, or so it seems, since it took me nearly half an hour just to come up with this simple function:

(defun tracklist-to-li (point mark)
  "Generate a string with <li>-elements containing tracks.
Assumes that one every line of region, a track position 
and the track name is given."
  (interactive "r")
  (save-excursion 
    (goto-char point)
    (let ((current-pos (point))
	  (result ""))
      (while (re-search-forward "^\\([0123456789]+?\\)\.[ \t]+\\(.*\\)$" 
				mark t)
	(setq result
	      (concat result
		      "<li id=\""
		      (match-string 1)
		      "\">"
		      (match-string 2)
		      "</li>\n"))
	(setq current-pos (point)))
      (message result))))

What took the most time was that I've had forgotten to escape the grouping parenthesis in the regular expression and that it took me a little while to accept that there is really no \d or equivalent character class in Emacs regexps. Which probably means that I've been doing too much in Perl, sed and the like. OTOH, it just may hint at the horror of regular expressions handling in Emacs. What I also dislike is that whenever you want some result in Emacs and see it, too, you have to invoke an interactive operation like message. Of course, there is IELM, but this doesn't really help you for interactive functions operating on regions.

And five minutes later, I realize I need to convert some string like "The (International) Noise Conspiracy|The
Hi-Fives|Elastica" into a similar list structure. With a simple cut & paste and roughly 30 seconds later, I have

[bauhaus->~]perl -e '$a="The (International) Noise Conspiracy|The Hi-Fives|Elastica"; @a=split("\|",$a); 
foreach $b  (sort @a) { print "<li>$b</li>\n"; }'

Hmm. Perhaps I've come quite a long way on the dark side already ...

On the other hand, in Ruby, this is just as simple (I'm using irb, the interactive ruby shell here):

irb(main):008:0> a="The (International) Noise Conspiracy|The Hi-Fives|Elastica"
=> "The (International) Noise Conspiracy|The Hi-Fives|Elastica"
irb(main):009:0> a.split("|").each {|string|
irb(main):010:1* print "<li>"
irb(main):011:1> print string
irb(main):012:1> print "</li>\n"
irb(main):013:1> }
<li>The (International) Noise Conspiracy</li>
<li>The Hi-Fives</li>
<li>Elastica</li>
=> ["The (International) Noise Conspiracy", "The Hi-Fives", "Elastica"]

The difference here is the implicit array Ruby generates, which of course in Perl you could hide in the array position of the foreach loop. Note the annyoing misfeature of irb to always show the prompt even when your still continuing your current input line.

In Common Lisp we can do it just as short:

CL-USER> (let* ((a "The (International) Noise Conspiracy|The Hi-Fives|Elastica")
		          (splits (ppcre:split "\\|" a)))
	           (loop
      	              for string in splits
	              do 
	                  (format t "<li>~S</li>~%" string)))
<li>"The (International) Noise Conspiracy"</li>
<li>"The Hi-Fives"</li>
<li>"Elastica"</li>
NIL

The same thing here: The result of the split could have been easily embedded in the loop.

The lesson, of course, is that in the end this example only serves to show that things that are easy to achieve in a high-level are indeed easy to achieve. Or to put it otherwise that the use of regular expressions is no more a discriminating feature between programming languages.

What I like about Ruby is that it's close to Perl in that it's nearly as concise as Perl and that many idioms from Perl are nearly the same in Ruby.

What I dislike about Ruby is that it's close to Perl in that it's nearly as concise as Perl and that many idioms from Perl are nearly the same in Ruby.

I.e., while that eases getting started a lot, it also implies that you will stumble over a lot of nitty gritty details, e.g. forgetting that you'll need to announce a parameter to a block AKA an anonymous subroutine. It also lures one into keeping old habits, e.g. just using a hash to store data instead of creating a new object/class that uses hashes internally or using Ruby as another imperative programming language.

I'm finally fed up with Perl. And I'm more the occasional user than a professional. Probably I should attribute it to my cons-cell based stomach that always nearly made me throw up when I commited an attrocity such like accessing an array as a hash value. As there's more to consider than personal taste when choosing programming languages, I'm afraid, CL won't be the next tool. And no, I won't create a DSL to generate Perl code from Lisp, although the idea has some appeal to it. Instead I will take a closer look at one of the more lisp-like script languages. After some googling around, Ruby seems more attractive to me than Python, in particular as functional programming seems to be closer tied to the community.

But starting with it on my current project already showed some of it problems. To name the most prominent one: lack of good documentation. I miss (documentation 'foo :function) and perldoc -f foo alike, and no, ri foo is nowhere near perldoc. Another problem is the Emacs integration: coming from Slime, ruby-mode and especially the interactive submode for irb reminds me of vacuum cleaners. Perhaps I should take this as a hint to take another look at Eclipse?