A view from the hill - Programming

I'll never be anybody's hero now

nospam@example.com (Holger Schauer) — Fri, 20 Apr 2012 09:00:26 +0200

One aspect I think is important for a Scrum Master or Project Manager is to make sure that your team doesn't go on a trip to Vienna (if that term doesn't ring a bell, search for "Tom DeMarco peopleware"). Quite contrary to popular management belief, I think in general it's not okay if "occasionally" somebody on the team "puts in some extra work". There is a reason why many agile methodologies insist on keeping a sustainable pace. Besides all of the very good reasons for making sure your team members stay healthy (see this Burnout story as a negative example), there is also a management point to it: your understanding of what the team is capable of (in terms of results/effort, aka velocity) decreases substantially if you have to take "heroic behaviour" into account. It's particular bad when you don't see the connection between reached goals and involved effort, i.e. when team members just move their card from "working" to "done" late in the evening without making clear that it involved five hours more than initially estimated.

Heroic behaviour just can't be counted on, because nobody will be able to keep it up over a substantial amount of time (that's the very definition of not being substainable). It's highly understandable that project members after having committed to some goal can be tempted to go out of their way to reach it. What team members might miss is that "heroic behaviour" can only have an influence on the "time" aspect of the magic triangle of "time, budget and quality". Extra effort is just that: effort. Hence, it comes with a cost, with the cost it just takes to reach the goal. I've also seen that there is a misunderstanding of the term "commitment". It's not an unconditional promise of "I can do that task with the effort I think it takes", there is also the implicit condition of "I understand correctly what the task involves and there is no other external negative influence" (e.g. the urgent bug that needs to be looked at, or the lack of sleep during three days of the week due to the kids being sick at home).

Commitment to a particular goal might at times conflict with taking responsibility for the project as a whole. As a general rule of thumb it's nearly always much more important to think about the entire project / the big picture than about a small aspect of it. There is the exceptional situation that needs exceptional reaction and maybe exceptional effort. But it's important to treat it like an exceptional situation. And for these exceptional situations it's vital that they get treated like a mini project: they should have a clear purpose and have fixed start and end dates. Plus, they should come with a compensation. Scrum Masters and project managers alike should communicate clearly that exceptions are exceptions, not the rule. And team members should clearly communicate that it takes what it takes. When it comes to professional work, follow the 501 manifesto (in case you don't directly understand the "501" part like I did: it's not about jeans, but about leaving at 5:01pm).

ObTitle: Morrissey, from "Ringleader of the tormentors"

New year meme

nospam@example.com (Holger Schauer) — Wed, 28 Dec 2011 22:38:00 +0100

Tarek Ziades repeated his Python new year meme in which he summarizes his accomplishments in the Python world this year and his considerations for the new year. As I stopped working as a Python developer some time ago, I will hence drop the Python restriction although I will keep the restriction on only looking at developer questions.

1. What’s the coolest developer application, framework or library you have discovered in 2011?

Mostly stuff I already new about but hadn't specifically the chance to use previously, so actually there was probably nothing which I would call 'cool'. For one, I started using git this year. For other projects, I took the next step and started hosting my Mercurial repositories at bitbucket. For another, I used nose for unit testing my Python stuff.

2. What new programming technique did you learn in 2011?

Probably the most interesting stuff I learned this year was from me finally reading AMOP (the Art of the Meta-Object Protocol), thereby learning a lot about the meta object protocol in Common Lisp and how it's related to CLOS. However, I didn't really put it to use much, although in hindsight I recognized that I've used stuff from it previously without understanding the complete picture. There was also a lot of 'enterprise technology' I looked into this year but nothing could be directly be labeled a 'programming technique'. Otherwise I think I mainly stuck to the techniques I'm used to.

3. What’s the name of the open source project you contributed the most in 2011? What did you do?

My contributions to the free software world continue to be at a very low level (typically, I don't contribute anything of value besides small hints on the net, complaints and the occasional bug report), which I keep seeing as a personal weak spot. This year at least I managed to implement something for CL-SQL. I also wrote two small utilities, mainly to scratch my own itch, which I still need to clean up a more before I could publicly present them.

4. What was the development blog or website you read the most in 2011?

That's certainly proggit, followed by InfoQ. I also follow Planet Lisp and Planet Python quite closely. I started again to listen to the podcasts by Heise developer and the software engineering radio, but I have to admit I'm too easily distracted to really keep going.

5. What are the three top things you want to learn in 2012?

I would like to continue to learn Clojure, finally writing some useful code in it. In particular, I would like to mess around with parallel computations, although there is a huge chance I'll apply it for small silly tools or web programming first. I would also like to work on some semantic web stuff, probably with RDFa. And maybe I'll take a look at programming with some of the more or less free tablet stuff, either WebOS or Maemo/Qt. I'll probably also continue to look into Java programming and common architectures on that platform, which is an enormous universe.

6. What are the top software, app or lib you wish someone would write in 2012?

I would like to see the semantic web getting a little bit more traction, mainly by easy integration of stuff like RDFa into content management systems. And, like probably every year since 1999, I wish somebody would finally write an (open source) application/set of clients to finally solve the synchronisation problems between different computers/phones/tablets. Note: I've given up on the wish that somebody writes an enterprise acceptable open source replacement for Lotus Notes or Outlook (and I'm talking only about calender and mail, sigh).

Agile development -- what is the benefit for the business?

nospam@example.com (Holger Schauer) — Mon, 24 Oct 2011 14:38:00 +0200

In a recent meeting, a colleague of mine mentioned that we wanted to use agile development this time with that new project and that I would provide some insight as an 'agile development expert' (not that this would be a term I would use). This in turn brought me some curious looks and a pretty general question during the discussion: what is the benefit of an agile development model from a business perspective anyway?

If I remember correctly, my answer went in the direction that with an agile approach you can change your mind about what you really need/want to implement throughout the project, if you learn something 'new' about what is really required. The second important aspect is that you get quicker feedback which allows for more influence in case things don't turn out right. After the meeting I started pondering the question a little, as I had the impression that my answer wasn't that convincing, so what you find below is a more detailed written elaboration of it.

Basically, what "you can change your mind throughout the project" boils down to is that agile development gives you more options to make use of opportunities coming up along the way/throughout the time you need to implement the project. The most important idea here is that you don't make decisions too early, because that limits your options. For an example, let's assume that one feature we specify is that we want to have a 'Facebook like'-button on the order page. Now, if we're following the waterfall style 'implementation follows after specification has been completed' development model, you would specify that and design the page with the button and at some time in the future, you hopefully get your Facebook button. But maybe during the hypothetical nine months from now in which development happens, what if Germany's privacy laws enforce new severe restrictions on the use of such buttons and, btw., Google+ is now the new hip site you have to support? You have to go through a (with larger features typically quite costly) change process, redo the specification, etc. This essentially makes the cost you had in writing the specification as well as for the design etc. a double problem: you didn't get out any ROI, obviously, but additionally time has passed while writing specs, doing design etc. in which you didn't really work on any results for the customer (lost opportunity: gather new clients with at least some new features).

So, you would have been better off with having the idea that at some point throughout the project you want 'social media integration for marketing reasons' and delaying what to do exactly until you're really there (from a priority point of view) where this is a feature that you want implemented -- i.e. you do the specification when your planning that feature to get implemented in the next development phase (which is typically two to four weeks with an agile development team). The idea is to take small, well defined steps that can be taken fast and starting out with the most important/valuable ones first, rather than to spend a lot of time on some 'complete specification' which withers fast throughout the projects implementation time.

The problematic point here is obviously, that it's pretty difficult to say when exactly is the right point in time to make a decision. In general, there is no 'right point' -- also often called the last responsible moment after which some opportunity is lost (I think the phrase was coined by Mary and Tom Poppendieck, see the excerpt from 'Lean software development') -- but multiple points in time, depending on many aspects. Influencing aspects are things like business opportunities (e.g. a new feature nobody else has at that time) or reducing risks (projects risks) (cf. Alistair Coburn reconsidering the least responsible moment). The best answer I currently have seen when to make a decision is that it should be taken when you have 'enough knowledge', i.e. when you have carefully considered your options and evaluated them (cf. real options). Even thinking about options alone is typically a highly valuable undertaking in itself, because all to often people just assume that the 'obvious way' is the best one.

So, the general answer is probably along the way of: for the business side, agile development has the benefit that it provides more flexibility and more influence throughout the project phase while at the same time (possibly) generating real value/revenue more early. There is, of course, more to it that also has appeal from a business perspective, but that's probably the main point. Feel free to add different opionions.

Report about the European Lisp Symposium 2011

nospam@example.com (Holger Schauer) — Mon, 04 Apr 2011 21:25:00 +0200

So, ELS 2011 is over which was the first conference I attended that was solely aimed at Lisp programmers. Overall I am quite happy with it although not all talks have been of the same quality. In particular I wasn't too excited about all three key notes, although all had interesting topics. The first one by Craig Zilles about best effort code optimization was about things intelligent compilers could do. Very interesting stuff for sure and I learned a lot about low-level soft- and hardware architectures but there was no apparent direct relation to Lisp. A similar problem troubled the talk about Scala: perhaps it was due to my late arrival (I got on the right subway but in the wrong direction, not for the first time) but the part I attended left me wondering why Scala is relevant on a Lisp conference. Marc Battayani's invited talk about his use of Common Lisp for programming FPGAs was nice, but first of all it was difficult to follow (not due to the content) and not many details were given about how the specialized Lisp embedded DSLs get converted to the FPGA specific code and what problems he had to overcome.

Now for some of the interesting regular talks: on Thursday, the report about porting SBCL to the supercomputing Blue Gene/P was nice and raised an interesting question: what can/needs to be re-discovered from old Lisp dialects for parallel programming for Lisps when more and more parallel cpus are becoming available to programmers. An issue that came up in both the talk about the futures implementation for ACL2 and in Nicolas Neuss' initial digression about his experiences in parallelizing Femlisp was that garbage collection can get in the way of effective parallelization, up to the point where much of the expected speedup is lost. The motivation of the talk about actors framework named Jobim for Clojure nicely fitted in with an immediate question that came to my mind when I saw how Clojure connects to Java: What do you do so that Java's semantics don't leak into your application code? They seem to have found a nice way to abstract away the underlying Java libraries in their framework. In the last session of the first day, the lightning talk session, two things were interesting: Ralf Möller talked about using user-defined method combinations as a more powerful approach than design patterns, showing how one might implement html specific print-object methods, and Didier Verna talked about user-extensible format directives which he wants to turn in as a CLRFI. Having done a lot of work in computational linguistics, the talk about S-NLTK, the Scheme toolkit for natural language processing, was nice to see. Damir Cavar did a good job promoting the toolkit which has a similar aim like the Python NLTK, although I would have liked to learn more about the API and implementation issues. Finally, Alec Berryman by ITA gave a last minute presentation about things ITA learned about optimizing stuff for SBCL and about issues arising when adopting the old code to multi-threaded programming. Interestingly they didn't report about gc issues but that may be related to their extensive use of object caches of pre-allocated objects.

The final panel discussion with James Knight, Christophe Rodes, James Anderson and Martin Simmons went back and forth about concurrency, distribution and efficiency vs. performance. The discussion took up several points from the talks, including gc and hardware issues. I took away from the discussion that unsurprisingly a lot of open questions need to be solved of which people are aware while at the same time there doesn't seem to be much momentum, which, given that the community isn't that big, isn't surprising either after all.

Summing up I liked it a lot. For one, it was very nice to see people you only know via the net (while I haven't been active in the community in the last years, I was surprised to see people recognize my name). For another, I also think that the organizers made a good decision to select a main theme for the conference and an important one, too. It really set the main theme for the conference and the discussions, and hence nicely reached its goal. Generally speaking, the conference was nicely organized so thanks for a pleasant time in Hamburg.

Lisp, sql, git and mercurial

nospam@example.com (Holger Schauer) — Tue, 29 Mar 2011 21:25:00 +0200

This has been a rather unpleasant month (don't ask, I won't tell) but right now I'll look forward toward its end because of two reasons: for one, I'll be in Hamburg for the European Lisp Symposium for the next two days; the program for the ELS has also been published in between. I'm really looking forward to an interesting set of talks. For another, some patches to CL-SQL which add support for autoincrement behaviour for Postgresql, are probably going to be released soon. To clarify, "autoincrement" is a column constraint in MySQL (among others) that automatically increments the value of the column when a new row is inserted when no value for the autoincrement column is given (cf. MySQL docs on AUTOINCREMENT), a behaviour that Postgresql supports with the serial constraint (cf. this wikibook on converting between MySQL and Postgres). Actually, that has been my first substantial amount of Common Lisp programming in the last two years, which has been triggered by an upgrade of my Debian system. This upgrade implied that an old application of mine would now use CL-SQL version 5.0 which in turn broke the app: I had simply specified a db-type of "serial" previously, but the new CL-SQL code wouldn't recognize that it had to fetch the automatically generated value from the DB when inserting a new record. More details on the patches can be found on the CL-SQL mailing list.

The developement of this addition was also the first time I had a real-world setup developing with git. In my own projects I use mercurial, so I was eager to learn a little bit more about the differences. It's funny that a recent opinonated article "Why I like mercurial better than git" more or less talks only about the one point that I found confusing: branch handling. For more background information, I suggest reading this article "A guide to branching in mercurial". Basically, in my current projects where I use mercurial, I'm using the "branching with clones" approach Steve is describing there. When working on the patches for CL-SQL, I was working on the existing autoincrement branch but when I was through I wanted to port my patches to the master branch. When using mercurial with the described approach, selecting (pulling or pushing) my patches and only my patches to the master branch is dead easy: you just issue a pull/push command restricted to the "right" changesets. Doing this is even supported by Subversion these days via svn cherry picking. Looking at the docs for git pull, fetch and merge, I wasn't able to figure out what the corresponding "right" incantation for git might look like, if there is one at all. As I didn't want to hose my "working copy" (sorry for the SVN term again), I resorted to git format-patch, git am resp., which worked fine. Please note that I'm not suggesting that it's not possible with another approach, quite to the contrary I would be happy to learn about it. One thing that I found rather useful is git's stash command which let's you safely abandon your current work to fall back to the last commited version, in order to be able to work on something that popped up in between (typically a minor unrelated problem you encounter while working on a larger piece of changes). I understand that mercurials patch queues enable a similar functionality, but I haven't used them sofar. Another thing that I found very useful is git's very easy way to correct (or in git terminology "amend") a commit by just issuing "git commit -a". I also like the idea of the "index" or more exactly that you have to explictly "add" which changes you want to commit. A similar behaviour is possible with SVN "changelist" command, but the mere existance of a changelist is not automatically honoured by SVN's commit.

European Lisp Symposium 2011 in Hamburg

nospam@example.com (Holger Schauer) — Sun, 27 Feb 2011 17:13:00 +0100

Hamburg, I'll see you soon -- for the 2011 edition of the European Lisp Symposium. I haven't done much Lisp programming in the last two years, but as I can combine visiting a good friend with a nice conference, I just couldn't resist. I'm really looking forward to meet several people I've had contact with over the net in reality. (And don't forget about that unresolved tension.

Laziness for the win

nospam@example.com (Holger Schauer) — Tue, 14 Dec 2010 19:10:00 +0100

Loren Segal writes a blog discussing the question whether users of dynamically typed languages are just too lazy to type? The argument goes roughly like this: The dynamic is not in "passing wildly arbitrary ever-changing types to methods" and we're just (okay, not only just, but mainly) doing dynamic code generation and are too lazy to add type information. Regarding a particular, but not too relevant code example, Loren says:

It’s easy to see that this code is executed immediately at load time. Although it’s possible for someone to generate a method at run-time (by run-time I mean much later after the initial .rb file was require‘d), it’s a fairly rare occurrence. What we’re really doing here is just avoiding to extra LoC involved in typing out those def’s each time. What we’re doing here really is just runtime code generation.

Yes, Loren, it's just runtime code generation, but calling it 'just' is somewhat misssing the point. I mean, if runtime code generation is not dynamic, what is? Loren goes on to say that programs in dynamic languages with these runtime modification behaviours often stop modifying their behaviour after a certain amount of "load time". I agree, this is also what I would expect. But the entire point is that once again, "dynamic" means that it is not necessary that you have to (statically) pre-determine at some fixed time (compile time or initial process launch time) which stable behaviour it should be. Of course, after a certain amount of load time, it's likely that the behaviour of your program (or better said of the running process) becomes stable and all functionality needed is loaded. But a different process of the same program may use a different set of possible behaviour. Plus: you might be able to add new functionality to a running programm. Additionally, you don't necessarily have to provide all possible behaviours at start-up time, runtime code generation (or runtime code loading) allows you to add new behaviour later on. In some cases (Ruby and Lisp come to mind) it is even possible to alter existing behaviour. Now, if "typical" dynamic programs really don't do that routinely (as suggested by the studies Loren is citing) this says more about how often we (really don't) need such dynamic behaviour but it sure is dynamic.

However, I must admit that I'm a little surprised by the cited results of the studies. I've seen systems that have the ability to change their behaviour at runtime one time too often. First, there are programs that embed some extension mechanism (be it embedded scripting languages, plugin mechanisms or other) that are specifically crafted for users to tinker with the running system, which clearly is all runtime code generation. Second, a particular interesting aspect of web programming is that many templating engines have some kind of import/include statement that are typically used to build complex templates out of smaller ones. In combination with other features of template languages, specifically conditions, loops and even macros (cf. stupid template languages, not so stupid template languages) this feature can be and is used to alter behaviour at runtime. So, obviously there is a need for dynamic behaviour, and people find ways to scratch their itch. Note that neither of my two examples require dynamic languages, so it's not a rebuttal of the studies or Lorens claims. However, thinking about it, I would expect that a lot of programs written in your todays typical dynamic language are 'scripts', which might be the reason why most people treat the idea of 'dynamic language' as synonym to 'scripting language' (it isn't -- Common Lisp, for instance, is surely a dynamic language, but hardly a scripting language). That such scripts don't require much dynamic change in runtime behaviour isn't too surprising, after all.

Now ultimately I wouldn't really disagree with the impression that quite often one "value" of dynamic languages lies in reduced typing and that on the contrary some type related problems would be found more quickly in static typed languages. I would even go so far and say that wrt.

I’d really love to see some research on case studies of how often arbitrary structural typing (that cannot be refactored into polymorphic relationships) is really used in dynamically typed languages. If anyone knows of any research/papers in this field, I’d love to hear about it.

it's highly likely that it's extremely rare to find "arbitrary structural typing", if such a thing exists at all (ironically I would guess that using 'void *' as a type specifier in C functions is where one comes closest to "arbitrary"). To me the very idea looks suspiciously like a strawmen for an argument that if there were "no arbitrary structural typing", we could/should go for static typing anyway, as we could "reduce" the solutions we implement with dynamic typed languages to solutions in statically typed languages (an argument coming close to the 'turing complete' hammer). This looks like a distinction between 'we can't use type information at all' and 'we need to use type information at each and every place' and I don't believe that the world is only black or white.

The point I want to make is this: Let's assume it may be very much the case that in all places where dynamic typing is used you can find a static typed solution, too. Then the benefit is of course not that "you're doing something that's not possible in statically typed languages" (which might still be true under the interpretation that the statically typed solution probably looks rather different). But there might be plenty of room for benefit in that coming up with a solution in the dynamic language might reduce cost (e.g. less development time, greater flexibility for extension later on). The arguments by proponents of static languages that the cost reduction lies in other aspects (e.g. earlier detection of errors) are surely worth a comparison. If studies for this question of associated costs beyond anecdotical tales exist, I would be interesting in seeing them.

Update: Uncle Bob raises another important point that with introducing static type information we'll get closer coupling:

But in the end, we like dynamic language not because we are too lazy to "Type". We like dynamic languages because we are tired of untangling couplings.

Coders at work

nospam@example.com (Holger Schauer) — Wed, 22 Sep 2010 20:38:00 +0200

For the holidays I finally bought Peter Seibels Coders at work, which is a very unusable book about programming: it consists solely of interviews with pretty well known programmers or "coders". It's an interesting constellation: On the one hand, Peter Seibel is well known in the Common Lisp community for his book Practical Common Lisp which gives a modern view on Lisp: not only is it an introduction to the language but also to several libraries and the general setting of modern lisp programming. On the other (fifteen) hands, there are people like Jamie Zawinski (XEmacs, Netscape), Don Knuth (TeX, Art of Computer Programming), Guy Steele (Lisp, Scheme, Java), Peter Norvig (PAIP, Google), Brendan Eich (Javascript) and Ken Thompson (Unix) -- just to note the ones that are probably the most well known.

I've had resisted the urge to buy the book because I've always felt that programming is a craft that ultimately forces you to make your own experiences. I mean, you can read all the books you like but ultimately you have to make your own hands dirty to really get knowledge about the issues involved. So, what could I learn from other peoples experiences? On the other hand, as a lightweight (in terms of reading attention) holiday book it seemed about right, so I finally gave in.

Well, the book turned out to be a real page turner for me. It's a fascinating read because of the re-occuring topics Seibel is addressing and the various opinions he got. He addresses topics you would expect like preferred tools (e.g. editor), worst bugs, debugging techniques, asssertions and verification, literate programming (which suprises me a little), design approaches and team work, but of course the main focus is the personal experiences and how they wound up with whatever made these guys known. One thing that I liked is that Seibel has a way to ask good follow-up questions to the responses he gets, without ever letting his own experiences or opinions getting in the way, which I can imagine has probably made for pleasant interview situations (at least I take away that impression). I wouldn't have imagined beforehand that I would find the different stories how the guys (and one woman) got into coding so interesting. There are very few people in this book whose experience doesn't go back to teletype and time sharing systems. Of course, as a result these stories tend to be similar, but the details differ enough that's it doesn't get too boring. Starting with computers in the early 80s, I don't have any experiences with such systems and which I frankly don't miss at all after reading more about it. But just to get to this conclusion is interesting: the constant comparison with your own experiences and opinions you can't help but make while reading this book alone is worth buying and reading it.

Over all, it's hard to say which interviews I found the most interesting one, essentially each has some unique point or other. That being said, the interviews with Joe Armstrong and Guy Steele made a lot of impresssion on me, whereas I'm a little disappointed by the one with Peter Norvig (though he had the funniest quotes), but I can't really nail down why. I didn't particular like the interview with Brad Fitzpatrick, it didn't seem to contain as much information as the others. And Joshua Bloch seemed to hype Java all the time which I found not very convincing -- the idea that todays larger context for programming contains quite a few different languages and approaches seems to elude him.

There are some points I took away from this book: For one, most of the interviewed people seem to be much more concerned with data types than I am, even the ones who have done extensiv work on dynamically or weakly typed languages. I guess I should really take a closer look at that topic and, to make it more concrete, play around with e.g. Haskell. Another point is that concurrency or parallel programming is a topic that (IIRC) all of the interviewees have seen as being responsible for the worst bugs they encountered and as a result are interested in newer approaches like STM. So, it might be worthwhile to look closer into such developments, for example by playing with Clojure, Erlang. or the transaction monad, if I'll ever really play around with Haskell. A third point is that I realized that I'm not keeping up with academic research in CS and, not having TAoCP, might never have been up to date at all. I'm following a few online references like LtU, but not closely and it's pretty rare these days that I look deeply into some research paper. This is something else I should probably change, if time permits.

There's nothing left to see

nospam@example.com (Holger Schauer) — Mon, 09 Nov 2009 09:28:12 +0100

Via Lambda the Ultimate I came across an interesting article On data abstraction, revisited by William Cook, written for OOPSLA'09. It carefully dissects abstract data types from objects. All theoretical considerations aside that distinguish ADTs and objects, there is one common characteristics given by Cook: you can't inspect the concrete representation of the data you're abstracting. This is in itself interesting and reminded me of two rather practical things.

First of all, I was reminded of a section in Bob Martins Clean code development which discussed the idea that you should on the one hand follow the rule "Tell, don't ask" and on the other hand have data access objects that don't have much, if any behaviour besides providing data. This is obviously directly related to Cooks article: if you want data abstraction, you shouldn't really provide any way to allow other objects/methods to access the internal representation. This somewhat also forbids getters as this is likely to lead to leaky abstraction, since more often than not programmers simply return the value of some data field, directly exposing the representation chosen. Now, please note that this does not necessarily follow from Cooks article, as it is possible to design getters in such a way that you can return whatever you want for a getter method, i.e., you can return a desired return type or an object satisfying a particular interface. For me, the relevant point here is the way of thinking about the kind of object at hand: do I want some behaviour (aka Cooks objects) or do I want a data sink. In the former case, and in line with what is suggested in the clean code book, it is arguably the best way to tell the object to do what is necessary rather than to inspect (get) the data it holds and do it externally in some other object/method. But even in the latter case, I think it is important to give great attention to hiding the internal representation from external access and to only allow very focussed access to the data itself. It could and has been argued that restricting the access to the stored data via getter methods is tedious (see e.g. the discussion in getters/setters/fuxors) and that allowing public access to members is allright, but looking at the issue from a data abstraction point of view it simply boils down to the question whether you want or need data abstraction or not.

Second, I've recently seen these two postings on the merits of the Zope Component Architecture: The emperors new clothes and the reply The success of the ZCA. Malthe asks why one should use the ZCA to override the use of a particular implementation with another instead of using some kind of reloading (or rather says that the latter is the preferrable approach). Relating this to Cooks article, Malthe could be paraphrased roughly as: we have ADTs all over the place and we only should allow only one implementation per ADT (this is what the type system would guarantee in other systems). If you want another implementation (of some interface, as Cook shows for his objects), you should reload the object defintion with the one you want. The use of the ZCA, however, is directly related to the very idea of object oriented programming in the way Cook defines it: you only have interfaces that are the relevant defining characteristics of objects (values) and hence, the use of the ZCA is the way to deal with multiple implentations in Zope (or Python). For me, all I can say is that I'm happy that the ZCA and hence the ability to easily intermingle multiple implementations is there (then again, with me reading computer science theoretic articles I'm arguably not of the angry web designer type whose benefit Malthe is arguing for).

There is another, more puzzling aspect of the article to me. After some considerations, I have to conclude that of all OO languages I happen to know, it's really only Java that seems to be object oriented in Cooks view of the world. This is because in Java, you can define a method to return objects satisfying an interface. In addition, in dynamically typed languages like Python, Ruby, or CLOS, you could try to come away with duck typing, but it's arguably only Python which tries to take it to the heart (for instance in CLOS, most values you're gonna deal with are non-CLOS values and you even have an ETYPECASE statement, which is a switch-statement on type distinction). Funny enough, Cook finishes his Smalltalk analysis with the statement that "one conclusion you could draw from this analysis is that the untyped lambda calculus was the first object-oriented language". But besides the point how some language is "more OO" than another, there is also to the point that in order to program truly object-oriented, you shouldn't (and in Cooks world really can't) rely on type checks, because the whole point of using objects as data abstraction is to rely on behaviour.

ObTitle: Dial M for Murder, "Fiction of her dreams"

Magic theatre not for everybody

nospam@example.com (Holger Schauer) — Thu, 22 Oct 2009 18:38:00 +0200

If you just stumbled over this blog entry searching for what the heck all that buzzing about agile methods amounts to, I may have unfortunate news for you: An agile approach to development might for a lot of people all over the world, but may be not for you. For starters, take a look at the agile manifesto and its principles. These have quite a lot of implications, directly translating into presuppositions about you, if you wanted to participate in an agile development project.

Now, perhaps you're the well communicating, team oriented developer type who happens to work in a surrounding which you respect and general feel well motivated. But real life experience with your typical geek or ex-geek-turned-professional-developer suggests that having a hard time with direct communication, face-to-face, happening very often with those pesky guys and girls you don't really like is a more likely scenario ("Eeek, those business and marketing people" anybody?). Now, you might say, baah, I know some communication is a necessary evil of professional software development, but maybe I can get away with my old habits of trading documents (specs, bug reports, etc.) against direct communication, since that is the trusted old way and what do you mean by face-to-face communication being more effective? Everybody hates those horrible meetings eating up all your time, right?

The point is that if you're wanting to go agile you should better soon adapt your view or else you won't see any of the promised benefits of all of those agile processes. Agile development is not only about using test driven design, timeboxing, iterations and releasing often. More than anything else, agility is about how people effectively interact with each other as quickly and direct as possible in order to come to solutions. To me, this basically boils down mainly to three issues: adaptability to (changing) situations, an open mind toward communication and taking responsibility for the project at hand.

Being lazy, sticking bone-headed to "trusted paths", not seeking confrontation when it's needed, avoiding communication because of being too busy "doing" is the anti-pattern to agility. You could keep a product and sprint backlog, present new features every other week to your customer and even use continuous integration and still would be muddling in pseudo agile, doomed to fail water. What I would suggest is that you should revisit the agile manifesto and think about what those principles might imply for you and your current behaviour. Let's try an example: If you find that you might end up pointing fingers at your customer (or product owner to use the terminology from Scrum) because he won't behave as demanded in the agile manifesto, you should reconsider if that fits in with what the agile manifesto demands of you: you wouldn't do the project any good. You would build up frontiers where there should be a single team including the target of your pointed finger. This is not to say that you shouldn't point out the defect, no communication about problems is another all too familiar reason for frustration and lack of commitment. What is necessary is direct communication and finding a way that works for all people involved.

Let me sum up this posting: if you're interested in agile approaches to software development, that's fine. If you don't feel comfortable about what this might imply for you, that's fine too. But it might also tell you that agile development approaches might not work for you.

Update: Irionically, I stumbled via Infoq about the one essential agile ingrediment which says all about it much nicer than I ever could have done, only that Mark Schumann shows the confidence that agile works for the most part whereas I hold the more pessimistic view that it's not going to work for some, but for exactly the same reasons.

ObTitle: Hermann Hesse, "Steppenwolf"

Version control and testing

nospam@example.com (Holger Schauer) — Fri, 24 Jul 2009 10:30:05 +0200

What have version control and testing to do with each other? Well, first of all, both are common virtues in the clean code community. What you'll find is that both virtues are important on their own ground: version control provides a safety guard in that you can roll back to prior versions if you accidently introduce problems in your code. Testing (automated unit tests) provides a safety guard, too, because you can do regression testing when you work with your code. These are both fine goals but seemingly have little to do with each other.

But in reality they do. For sake of argument, let's take a step back and assume that you have to work in an environment of several developers where neither of these things exists. What will you likely see? What we all have seen several years ago. Commented out code blocks, redundant and often misleading or outdated comments, timestamps with comments cluttered all over the code. And frightened developers that feared each minor change because of the miriad of subtle side effects it might have, let alone major changes to core components. It's an environment in which refactorings as welll as extensions are very hard and expensive, which results in frightened overworked developers and frustrated managers.

So, what happens when you introduce only one of those virtues? Say, we introduce version control. Now, every change gets documented, except that documenting every change requires, from the developers point of view, documentation at the wrong point. They can't see the documented changes and the reason for these changes in the source, they see it only in the version control system -- iff they add a change message with every change at all. Much more likely is that you will see commit messages such as "." or "bug fix", and the same old mess of timestamps, outdated comments and commented out code as before. Why is that? Because your developers are now not as frightened as they used to be (because they can now rely on the version control system to fall back to older versions), but they still have the same need to understand and document the code. And the commit log is both "too far away" from the code and "out of it's purpose" for this task: the commit log shouldn't document what the code is supposed to do, only when something was implemented to behave in a particular way.

This is where a development (unit) test suite comes into the picture: you document every required behaviour in tests. With every change to the code, you also update the test. As a developer, you can now look into your test suite to see what the code is supposed to do. Now developers will likely become much more confident with their changes, because they can run the tests and see what happens (hopefully next to immediately) without requiring time- and resource-consuming manual tests.

But what about documenting the changes to the code? Well, you should simply document any changes in the commit message of your version control system, because it's now no longer necessary to keep the entire version history in mind to understand what the current code state is supposed to do. You have the tests that tell you what the code should do. The commit log now only serves the purpose of documenting what has changed over time and is no longer required to understand what the code should do. So you don't have to keep the clutter in your code, resulting in much cleaner source code files.

Summary: Taken together, the whole of version control and testing adds up to more than a simple addition of their own values.

I guess I'll hang out my tears to dry

nospam@example.com (Holger Schauer) — Mon, 11 May 2009 08:55:33 +0200

There seems to be a roaring discussion about a talk by Robert "Uncle Bob" Martin at this years RailsConf, which mainly seems to be concerned with the question (raised by a quote appointed to Ward Cunningham) whether languages such as Smalltalk make it too easy to create a mess. Languages such as C++ would penalize the mess at least by longer compile times.

As Giles points out that's an old argument, which I too have heard too often in the past about dynamic languages, regardless whether we're talking Smalltalk, Ruby, Lisp, Python or Perl. And the anti-dote was always the current idiot language of the date, i.e. Visual Basic at its high time, Java or C# (for those who considered C or C++ to be too dangerous). Please note that I'm not at all saying that users of those languages are idiots, but those languages are particularly claimed to be usable by idiots, too. This approach to languages or to their classification does a great job of misrepresenting programmers regardless of the language of their choice. It makes the bold claim that there are programmers that are idiots and that it's possible to design a language that even those idiots can do their jobs. And the exaggeration is that its only good business sense to enforce the use of those languages for all development projects, because, as we all know, there are more idiots than intelligent programmers out there, right?

The only problem with this claim is that human stupidity is probably the only thing larger than the universe, if you make something idiot proof, nature will quickly make a better idiot. To put it a little less cynical, to err is human. And there is value in people making mistakes. As for instance, Tom DeMarco claims in his book "Slack" it's a particular good way to learn. Bob Martin tries to sell excessive testing (which isn't particular surprising, giving his background) as one safety guard to which James Robertson responds that this is not always necessary:

Yes, tests are useful. But, the debugger is not something to be feared. Rather, it's a great tool to be used in order to have the computer do all the memory work for you. I can get a lot more done by working with decent tools like the Smalltalk debugger than I can by assuming the doc is good and writing tests that just help me a whole lot less than you might think.

I can relate to this quote, being a fan of interactive development that languages like Lisp or Smalltalk made popular. But I think James is missing the point: tests are not only helpful during development. Sure, using an interactive approach helps getting the code run but does it help to come up with a clean design? Looking back at some code, I can assure you it doesn't, quite to the contrary it's a way to get a working solution quickly, not cleanly. This doesn't make the debugger or the REPL less valuable: using it you can quickly establish your tests, debug your tests, whatever. This is exactly where the value of dynamic languages shines and those "silver-bullet" idiot languages are still behind, regardless of their shiny IDEs.

But the real point summing up the discussion so far is the old Frederick Brooks quote: There is no silver bullet. It's good to be careful and to use the tools at your hands to help you avoid the big mistakes. As Paul Graham puts it for OOP: "For programs that would have ended up as spaghetti anyway, the object-oriented model is good: they will at least be structured spaghetti." But in the end you have to learn how to avoid the mess in the first place and no language and no tool can do that for you.

ObTitle: Dexter Gordon, "Ballads"

Testing and terminology confusion

nospam@example.com (Holger Schauer) — Fri, 06 Feb 2009 11:11:40 +0100

I may be rather late to the game, but over the last one and a half years, I've become quite addicted to writing tests during my development tasks. I've had wanted to dig into test-driven development for quite some time, but it was the seamless integration of Test::Unit, Ruby's unit testing module, in Eclipse that got me going initially. I then did some unit testing with Common Lisp packages and am currently heavily using pyunit and python doctests (mostly in the context of zope testing). Writing tests has become my second development nature: It gives you that warm fuzzy feeling that you have that little safety net while modifying code.

However, there are times when terminology comes along and gives you a headache. A terminology I've learned about during the last year is the difference between unit testing, integration tests and functional tests (for an overview see wikipedia on software testing). But as you can see for instance in this article on integration tests in Rails, it's not always easy to agree on what means what -- Jamis and/or the Rails community seem to have the integration/functional distinction entirely backwards from what, for instance, the Zope community (on testing) thinks.

Now, one might argue that terminology doesn't matter much given that you do write tests at all, but it's not so easy. For instance, if your "unit test" of a given class requires another class, is that still unit testing or is it integration testing? Does it even make sense to talk about unit-testing a class? A class on its own isn't that interesting after all, it's its integration and interoperation with collaborateurs were the semantics of a class and its methods become interesting. Hence, shouldn't you rather test a specific behaviour, which probably involves those other classes? And what now, if your code only makes sense when run on top of a specific framework (Zope, Rails, you name it)? Michael Feathers argues convincingly in his set of unit testing rules that any such tests are probably something else.

Ultimately these questions directly pertain to two aspects: code granularity and code dependencies -- and remember, test code is code after all. These are directly related, of course: if your code is very fine-grained, it's much more likely that it will also be much more entangled (although the dependency might be abstracted with the help of interfaces or some such, you still have the dependency as such). And as a consequence, your test code will have to mimick these dependencies. On the contrary, if your code blocks are more coarse-grained (i.e. cover a greater aspect of funcionality), you might have less (inter-)dependencies, but you won't be able to test functionality on a more fine-grained level. As Martin Fowlers excellent article Mocks aren't stubs discusses in detail, one way to loosen these connections between code and tests is to use mock objects or stubs. Fowlers article also made clear to me that I've used the term "mock object" wrongly in my post on mock objects in Common Lisp: dynamically injecting an object/function/method (as a replacement for a collaborator required for the "code under test") that returns an expected value means using a stub, not a mock -- another sign of not clearly enough defined terminology (btw, the terminology Fowler is using is that of G. Mercezaos xunit patterns book).

It's worth keeping these things apart because of their different impact on test behaviour: mocks will force you to think about behaviour whereas stubs focus on 'results' of code calls (or object state if you think in terms of objects being substituted). As a result, when you change the behaviour of the code under test (say you're changing code paths in order to optimize code blocks) this might (mocks) or might not (stubs) result in changes to the test code.

It's also worth thinking about mocks and stubs because they also shed a new light on the question of test granularity: when you're substituting real objects in either way, you're on your way to much more fine-grained tests, which implies that you loosen the dependency of your tests: You can now modify the code of your collaborateur class without the test for your code under test breaking. Which brings us back full circle to the distinction between unit tests and integration tests: you now might have perfect unit tests, but now you're forced to additionally tests the integration of all the bits and pieces. Otherwise you might have all unit tests succeed but your integrated code still fails. Given this relationship, it seems immediately clear that 100% test coverage might not be the most important issue with unit tests: you might have 100% unit test success, but 100% integration failure at the same time -- if you don't do continuous integration and integration tests, of course. Now what's interesting is that it might be possible to check test coverage on code paths, but it might not be easy to check integration coverage. I would be interested to learn about tools detailing such information.

Recently I had another aha moment with regard to testing terminology: Kevlin Henney's presentation at this years german conference on object oriented programming, the OOP 2009, on know your units: TDD, DDT, POUTing and GUTs: tdd is test driven development, of course. The other ones might be not so obvious: "guts" are just good unit tests and "pout" is "plain old unit testing". I saw myself doing tdd, but come to think of it, I'm mostly applying a combination of tdd, pout (after the fact testing) and ddt: defect driven testing. I find the introduction of a term for testing after the code has been written interesting because it provides a way to talk about how to introduce testing in the first place. Especially defect driven testing, the idea to write a test to pinpoint and overcome an erroneous code path, might be a very powerful way to introduce the habit of regularly writing (some) tests for an existing large code base. So you avoid the pitfall of never being able to test "all this lots of code because there is never the time for it" and you might also motivate people to try writing test before code. And on this level, it might at first not be that relevant to make the distinction between integration and unit tests to clear: start out with whatever is useful.

Core wars

nospam@example.com (Holger Schauer) — Thu, 08 Jan 2009 21:48:00 +0100

Drew Campsie announces ucw-core, the next generation of Uncommon Web, a continuation-based framework for web applications written in Common Lisp. It's the first major announcement since Drew took over development lead of ucw and it seems to focus on establishing a core library of basic technology on top of which other functionality can be implemented (e.g. functionality like those offered by the ucw-ajax). To me as a UCW user this sounds like great news, since I had the impression that ucw seemed to linger in a state of void over the last year or so.

But even more interesting is that ucw just seems to follow a certain trend. For instance, according to the announcement that Ruby on Rails and Merb will merge, one goal is that Rails will be retrofitted to make it easy to start with a “core” version of Rails (like Merb’s current core generator), that starts with all modules out, and makes it easy to select just the parts that are important for your app. The main aspect of such development (at least I hope so) seems to be best described by one of Django design goals): A fundamental goal of Django’s stack is loose coupling and tight cohesion. The various layers of the framework shouldn’t “know” about each other unless absolutely necessary. The python folks have even gone so far as to write down a web server gateway interface specification (WSGI, a quite minimalistic specification).

It will be interesting to see whether there's going to be even more unification, maybe even across language diffferences. I'm not too certain that such an unification would necessarily be a good aim; as always there are probably as many benefits as downsides to it. For instance, a common specification certainly lowers the learning curve for people switching frameworks while at the same time may stiffle further development. Then there's the fact that the different frameworks have different approaches which are certainly reflected in their APIs: for instance, Django explicitly tries to avoid magic at all cost while there's traditionally a lot of magic going on behind the scenes of a Rails application. I'm also not sure where exactly one should draw the line between common "core" functionality and additional "batteries". As an application designer it's probably most important that there is such a line at all so that you can easily pick your poison without having to buy everything.

You're wondering now

nospam@example.com (Holger Schauer) — Fri, 02 Jan 2009 09:18:33 +0100

Herb Sutter announces that Dr.Dobbs Journal is permanently suspending print publication and going web-only as of January 2009. I've used to be a long-time subscriber of DDJ -- several not-so-excited friends that recently helped moving into our new flat are my witnesses. Actually, I've stopped my subscription (and we're talking about a way to expensive subscription via a German importer, here) roughly one or two years ago, since I couldn't keep up with my reading but occasionally I've regret that I no longer receive it. Now, it seems to me like the last printed programming related magazine is going down the drain -- sorry, you Java and PHP magazines floating around, but your content isn't nearly as interesting as what DDJ had as it was ultimately the mix of different topic and opionated editorials that made DDJ so unique. Perhaps now I should hurry and finally buy the DDJ developer library dvd covering 21 years of DDJ articles. I'll keep an eye on what will happen to DDJ as a web publication.