Emacs

I'm a long standing user of Gnus, the Emacs mail and news reader (news as in usenet for those geeks young enough to not have used it in the 90s). I've been using Gnus in various settings whereas the latest configuration was to talk to a local leafnode NNTP server on my workstation. This was all fine and dandy and also survived several distribution level upgrades of the linux installation running on the system. However, some time ago, I bought new hardware and opted for a completely new installation. Of course, I backed up my leafnode and gnus configuration and restored it once the system was running.

Which was when the trouble started. I ran into the problem that I couldn't see any articles on my local leafnode, although I could retrieve them with other newsreaders just fine. I then opted to go and do a gnus-cache-generate-nov-databases. This turned out to be a pretty bad idea -- as a result I couldn't retrieve my mail any longer. After digging quite a lot around, it occured to me that the active file might have a problem. Indeed, it was empty. Fortunately, I could restore that quite easily, so I could access my mail again. Otherwise, however, I had gained nothing: still no news wasn't so good news. Next try: see if the server buffer tells me anything new. It did: the connection to my leafnode was open and I could even access the new messages on it this way. But this didn't have any effect on my normal group buffer.

Having learned to look around for the active file the hard way made me look into all kind of directions. Sometime in the past, I've used the agent-mode of gnus which resulted in a larger subdirectory tree in my News directory. And where which information was stored in the first place seems to have changed over time (or modifications I might to my configuration), too, so my News directory had lots of old files. After several tries and errors, I finally moved the News/cache directory out of the way and finally started to see new articles again. So, I'm now back to reading usenet.

ObTitle: Porcupine Tree, "On the sunday of life"

Betrand Mathieus blog article on cleaning up trailing whitespace comes in handy for me as I've frequently run into problems with the Python test coverage tool stumbling over trailing whitespace. It also reminded me of an emacs snippet I recently installed to detect a mix of space and tabs in my Python buffers -- I do set (setq indent-tabs-mode nil) in my .emacs for python-mode, however, I still occasionally somehow manage to insert some tabs in my source buffers. So I came up with the following snippet which validates that a buffer in python-mode doesn't contain any tabs. It's hooked up with the very general write-file-hook, but there is no python-specific hook on saving buffers. In case the buffer does contain any tab, it will leave the point (the place where your cursor will be) at the fount tab.

; code snippet GUID 32F94179-A86B-4780-8645-8A526BD8533A
(defun no-tab-validation (buffer)
  (interactive "bValidate buffer:")
  (save-excursion
    (unless (equal (buffer-name (current-buffer)) buffer)
      (switch-to-buffer buffer))
    (if (re-search-forward "\t" nil t)
	(error "Buffer %s contains tabs" buffer))))

(defun py-tab-validate-on-save-hook ()
  (when (equal mode-name "Python")
    (no-tab-validation (buffer-name (current-buffer)))))

(add-hook 'write-file-hook 'py-tab-validate-on-save-hook)

Enjoy.

Update: I should clarify that my previous reference to (set-indent-tabs-mode nil) is just a local function around setting the variable indent-tabs-mode directly. I use it also for a local toggle function, like so:

;;; inhibit tabs in some modes
(defun set-indent-tabs-mode (value)
  "Set indent-tabs-mode to value."
  (setq indent-tabs-mode value))

(defun toggle-tabs ()
  "Toggle `indent-tabs-mode'."
  (interactive)
  (set-indent-tabs-mode
   (not indent-tabs-mode)))

(defun disable-tabs-mode ()
  (set-indent-tabs-mode nil))

(add-hook 'sgml-mode-hook 'disable-tabs-mode)
(add-hook 'xml-mode-hook 'disable-tabs-mode)
(add-hook 'python-mode 'disable-tabs-mode)

ObTitle: Bloc Party, "Weekend in the city"

One of the nicer extensions of mozilla I use is It's all text. It allows you to call an external editor to edit text fields which is a really nice thing if you're editing longer wiki pages, for instance.
Not surprisingly, my external editor of choice is XEmacs. But my XEmacs is heavily configured and loads a lot of stuff. Even worse, I usually run a beta version with debugging turned on, so that it runs extra slow, so the time gap between clicking on that little "edit" button below the text field and the moment when I could finally start typing started to annoy me quickly.

I remembered that some years ago I used to log in remotely to a running machine, fire up some script and have either my running XEmacs session connected or a new XEmacs process started. Connecting to the running XEmacs happens via gnuclient, so much was clear but gnuclient doesn't start a new XEmacs if there is none running (actually, the server process gnuserv will die when the XEmacs process terminates, so there isn't much gnuclient can do). The emacs wiki page linked to above already contains a number of scripts that eventually do what I need to do, but I've found none of them convincing, so here is my version. It's linux specific in its BSD style call to 'ps' to determine the running processes, but should be portable sh otherwise. I could have sprinkled the fetch_procs function with OS specific variants, but as I'm currently running my XEmacs on linux only, I left that as an exercise to the reader.

#!/bin/sh

emacs=xemacs
gnuclient=gnuclient
gnuserv=gnuserv
user=`id -un`

gnuclientopts=""

fetch_procs () {
 ps xU $user
}

runninggnuserv=`fetch_procs | grep "[^]]$gnuserv"`
runningemacs=`fetch_procs | grep "[^]]$emacs"`
if [ "$?" -eq "0" ]; then
    gnuservpid=`echo $runninggnuserv | cut -f1 -d' '`
    emacspid=`echo $runningemacs | cut -f1 -d' '`
    echo "Using running gnuserv $gnuservpid of emacs $emacspid"
    $gnuclient $gnuclientopts "$@"
else
    echo "Starting new $emacs"
    $emacs "$@"
fi

ObTitle: Panic at the disco, from the album "A fever you can't sweat out"

I've recently blogged about my usage of Emacs for editing python code, in which I said that it sometimes involves remote editing via Tramp. At roughly the time I wrote the entry, I joined a different project in which the central development server only has a not-so-recent version of Emacs installed (besides the fact that it doesn't serve my favourite flavour of Emacs at all). This time, however, my naive usage of tramp failed: when accessing files via C-x C-f /[ssh/schauer@remote]/home/schauer/ all I would receive would be "Waiting 60s for prompt from remote shell" and tramp would be hanging. Searching the web had two interesting tips for debugging (regardless of what problem you have with tramp): go to the *scratch* buffer and evaluate (i.e. hit C-x C-e behind each of the closing parantheses of the following statements):

(setq tramp-debug-buffer t) 
(setq tramp-verbose 10)

The former setting will result in an additional buffer *debug tramp\*, while the latter will enhance the verbosity of information shown in the usual *tramp\* buffer that holds the actual remote shell interaction. Another hint then let me too a potential problem with my shell prompt setup, which is even mentioned in the FAQ. After clearing that issue by setting my fancy prompt only if the "TERM" environment variable isn't set to "dumb" or "emacs", I ran into another bit: I was getting an error about tramp not being able to issue the command "test -e". I'm still at a loss how that can happen, as 'test' is surely built in to the default shell I'm getting on that system and "/usr/bin/test" is also directly available when I log in. However, adding (add-to-list 'tramp-remote-path "/usr/bin") to my .emacs finally solved the problem.

To complete this post, I would also like to point at the manual on default method configuration, another interesting piece for my configuration.

I've become a python programmer, too, lately, due to a job change. Python is a fine language so far, although to me it's mostly just like Ruby, though with even less functional flavour. However, just as with Ruby, I'm really missing slime, the superior lisp interaction mode for Emacs, when hacking python code. I could now start to write down a list of things I'm missing (which I've intended to do), however, Andy Wingo spares me the hassle, as he has just written an excellent article on slime from a python programmers view.

However, I would like to elaborate a little on the main difference for me: the client/server socket approach of slime. Let me briefly recapulate what this implies: slime consists of two parts, a client written in Emacs lisp and a server written in Common Lisp (AFAIK there is at least also an implementation for clojure, maybe also one for some scheme implementation). In order to use slime in it's full glory, it's hence required that you have a common lisp process running which in turn runs the slime server part. If you now fire up slime, you'll get an interaction buffer over which you can access the REPL of the lisp process, which in python would be the interpreter prompt. You can then interact with the lisp process, evaluating pieces of code from your lisp source code buffer directly in the connected lisp process. What is incredibly useful for me is that you can not only start a new lisp process but also connect to an already running lisp process, given that it has the slime server started (this is obviously mainly useful if the lisp implementation you use has multi-threading capabilities). I use it to connect to a running web server application, which I can then inspect, debug and modify. Modification includes redefinition of functions, macros and classes, which of course is also a particular highlight of Common Lisp. I would like to cite a comment of the reddit user "fionbio" he made wrt. to the linked article: In fact, Python language wasn't designed with lisp-style interactive development in mind. In CL, you can redefine (nearly) anything you want in a running program, and it just does the right thing. In Python, there are some problems, e.g. instances aren't updated if you modify their class. Lisp programmers often, though not always, refer to various things (functions, classes, macros, etc.) using symbols, while Python programs usually operate with direct references, so when you update parts of your program you have much higher chances that there will be a lot of references to obsolete stuff around.

To complement Bill clementsons excellent article series on slime a little, I'm going to describe how I'm using/configuring python-mode to make it match my expectations a little closer. Essentially I would like to access my python process just as I would with slime/Common Lisp, but that's not possible. The reason, btw., is nearly unchanged: I need to code on a web server app (written in Zope) which may not even run on the same machine I'm developing on. Let's first cover the simple stuff: To enable a reasonable command interface to the python interpreter, I require the ipython emacs library. If the python interpreter runs locally, I also use py-complete, so that I can complete my code at least a little. Unfortunately, this breaks when the python interpreter doesn't run locally, because the py-complete needs to setup some things in the running python process, which it does by writing to a local temp file and feeding it to the python process. Unfortunately, the code in py-complete lacks customizability, i.e., you can't specify where that temp file should be located -- I should be able to come up with a small patch in the near future, which I will add below. Finally, I also require doctest-mode as a support for writing doctests, but that's not really relevant.

Now, on to the more involved stuff: I introduce some new variables and a new function py-set-remote-python-environment, which uses the those variables to do a remote call (via ssh) to python. This at least allows me to do things like setting py-remote-python-command to "/home/schauer/zope/foo-project/bin/instance" and py-remote-python-command-args to "debug", so that I can access a remote debug shell of my current zope product. That alone will only allow me to fire up and access the remote python, so I could now develop the code locally, having it executed remote. More typical though is that you would also want to keep the code on the remote machine, too: for this I use tramp, a package for remotely accessing files/directories from within emacs. In combination, this allows me to edit and execute the code on the remote machine. It is still nowhere near what is possible with slime, but at least it allows me to persue my habit of incremental and interactive development from within my usual emacs installation (i.e., it doesn't require me to deal with any Emacs related hassle on the remote machine).

;;; python-stuff.el --- python specific configuration

(when (locate-library "ipython")
  (require 'ipython))

(when (locate-library "doctest-mode")
  (require 'doctest-mode))

(defvar py-remote-connect-command "ssh"
  "*Command for connecting to a remote python, typically \"ssh\"")
(defvar py-remote-connection-args '("user@remotemachine")
  "*List of strings of connection options.")
(defvar py-remote-python-command "python"
  "*Command to execute for python")
(defvar py-remote-python-command-args '("-i")
  "*List of strings arguments to be passed to `py-remote-python-command`.")
(defvar py-remote-python-used nil
  "Remember if remote python is used.")

(defun py-set-remote-python-environment ()
  (interactive)
  (let ((command-args (append py-remote-connection-args 
			      (list py-remote-python-command)
			      py-remote-python-command-args)))
    (setq py-python-command py-remote-connect-command)
    (setq py-python-command-args command-args))
  (setq py-remote-python-used t))

(when (locate-library "py-complete")
  (autoload 'py-complete-init "py-complete")
  (defun my-py-complete-init ()
    "Init py-complete only if we're not using remote python"
    (if (not py-remote-python-used)
        (py-complete-init)))
  (add-hook 'python-mode-hook 'my-py-complete-init))

While I'm usually a happy VC user (the SCM mode of Emacs) as the next guy, I recently haven't always been so happy. In particular when I'm working on a larger set of changes in multiple files (using Subversion or Mercurial), the commit-per-file approach of VC just doesn't cut it, because usually you want to commit all changes you have to make for a fix or feature to happen in a single transaction. That's simply not possible with VC, so I usually fall back to the command line. That I can then use Emacs for the commit message is only a slight quantum of solace.
An integration of SCM features into dired (which I happen not to be a big fan of, btw.) looks to me like the obvious way to go. I'm hence very happy to learn that this is more or less happening (or has already happened) with the past-Emacs 22.1 version, according to this blog entry on manipulating changeset-based VCS from within Emacs. It's unlikely that this will be up to match Tortoise or TortoiseHg, but still ...

Steve Yegge blogs about XEmacs needs to die and I would like to add my own little comment on the issue. First, perhaps some background: I'm a long-term XEmacs user, since about 1995, I think. I started hacking Elisp around 1996, implementing a major mode for the Otter theorem prover and various other stuff. Since 1997 or so, I had been somewhat active on the xemacs-beta mailing list, reporting build successes/failures, participating in discussions etc. and even writing an article on the then-new XEmacs port to Windows in the german iX magazine. So, perhaps, I'm a bit biased towards XEmacs. That I've been using XEmacs instead of Emacs had a lot to do with two factors: at my university, the local Emacs guru used XEmacs and provided a very complete configuration to start with. To start hacking Prolog source didn't require any configuration on my side at all. When I tried to re-establish that configuration on my own linux box, using Gnu Emacs was totally out of the question, which is basically the second reason: XEmacs came with a lot of batteries included, whereas Gnu Emacs only came with a directly dying low battery at best. And third, the XEmacs interface experience was a lot friendlier than with the naked Gnu Emacs.

Now, at the time I was actively following XEmacs development, XEmacs had a whole lot of man power behind it, with lively discussions and an exciting movement of adding new features. At the same time, RMS discussed switching Gnu Emacs to Guile (a scheme dialect), which me, as a Common Lisp user, scared me off even more. Around 2002 or so, my spare time for XEmacs dropped to zero, so I unsubscribed the development mailing list and was just a happy user. Until two seemingly unrelated things happened: first of all, Unicode was no longer to be ignored and it was obvious that the Mule (multiple languages for emacs or some such) for XEmacs wasn't up to the task, especially not on Windows. That was quite a problem for me back then with the then current stable XEmacs 21.4. Now, six years later, XEmacs 21.5 is still not there (aka released as the new stable version) and it's unicode support still sucks (not so much as it did, say, two years ago, but it still fails a lot of tests from Markus Kuhn unicode test suite). The second development was that the Emacs team, which I had only barely been aware of previously, somehow gained a enourmous momentum and catched up a lot of ground, where XEmacs had been the leader by far. In particular, as of Emacs 22, nobody in its sane mind could argue against the fact that the current Emacs handles Unicode etc. far better than XEmacs (this is especially true for XEmacs 21.4 on Windows, but even the current beta XEmacs 21.5-b28 on Unix isn't where it should be). This seems to be directly related to the fact that a lot of the key developers of the end of the last century no longer actively participate in XEmacs development. I think only few people from the XEmacs crowd would argue against the impression that XEmacs development has more or less cringed to a halt and the distribution of development power between Emacs and XEmacs development is nowadays reversed to the situation from the 90s.

That being said, I still stick with XEmacs. Last time I tried using Gnu Emacs (that's the Emacs 22 from Ubuntu Hardy), I still found that I'm far more accustomed with the XEmacs intrinsics on how to perform specific actions than with how it works in Emacs. And I never got around to clean up my mess of configuration files to properly support the various flavours plus adding the extension libraries that still don't ship with Emacs (Slime, for instance). Finally, with more and more users switching to Emacs, there has to be someone reporting problems,

Now, back to Steves blog post: I think he has some valid points in it, but for the most part I disagree. First of all, while I agree that Eclipse and similar IDEs are still not at the greatest enemy to Emacs (regardless of flavour). I'm currently again using Eclipse at work and it's a pain finding out how to properly associate an unknown file extension with some specific editor (mode). And it's a memory hog that results in unbelievable handling (on my dual core 2GB equipped desktop) that is even worse than my first XEmacs experience on my 486/33 with 8MB ram in 1995. Eight megs and constantly swapping? Hah! I can't believe that even today I have to hear that Emacsen would be too complicated, given that for any task/configurations I spend endless time searching/clicking through the gazillions of settings in Eclipse.

However, I also don't believe that a marriage between Emacs and some web-browser is the way to go. There are much more pressing issues to move Emacs into the next century than switching from Elisp to <whatever>. If you're interested in extending your editor, you have to learn about the way how to do that. But that is totally independent from the question which editor you're going to use in the first place.
What is a much more pressing need in my opinion is finally implementing multi-tasking for the (X)Emacs core. Still being locked in a single-threaded application model is sooo 1990s. It's not becoming more and more ridicuosly, it's utterly unbelievable.

In one other point, though, Steve is right on track: the divergence between APIs is becoming a bigger problem with every passing day. I think it's very unfortunate that a) Emacs developers don't let them inspire by the existing APIs in XEmacs and vice versa and b) that there are too few XEmacs developers that merge current Emacs enhancement into the XEmacs codebase (the other way 'round is always problematic due to license issues).

So, before I would consider pursuing a major architectural change like XUL or some such for Emacs, I would first work on the much lower-hanging fruit of narrowing the gap between the two flavours. It's a pity I myself only have the spare time to write such blog posts instead of actively working on code towards that goal.

I'm using XEmacs whereever I can, but for preaching to the still unconvinced, the guided tour to Emacs as of the FSF will be just as useful. Of course, XEmacs ships with a much larger portion of modes and utilities, but unfortunately, development has been quite slowly in the last years and Emacs 22 as of today has come a long way to overcome a lot of the obstacles of older versions.

In an older blog post, I noted that there is no editor integration for the newer distributed versioning systems with Emacs. Turns out, I was quite wrong. First of all, there is an entire page about handling version control in Emacs at the Emacs Wiki. I haven't looked at all of them, but stumbling over DVC, I must confine that there is work under way to have a decent interface in Emacs to those new kids on the version control block. Looks as if DVC is even build or shipping with XEmacs. No more excuses around here, move along ...

Moving back in time is easy when the tools you're using are a versioning control system like CVS or RCS [1], (X)Emacs, it's versioning control interface VC, a Common Lisp system such as SBCL and Slime, the superior lisp interaction mode for Emacs. Yesterday, I was debugging a part of my current hobby project which involves a web interface made with Uncommon Web (UCW). One bit of that web interface that I hadn't paid particular attention to in the last round of modifications had stopped working. But the source that dealt with that particular bit seemed quite innocent to me.

I then looked through the log of modifications I did to the file (which is under version control via RCS) by hitting "C-x v l" (equivalent to "M-x vc-print-log"). Turned out that I did some modifications between versions 1.5 and 1.6 to the part under consideration. Hitting "C-x v ~" (vc-version-other-window) and entering numbers 1.5 (and afterwards 1.6) re-generated the old versions, XEmacs directly visiting these corresponding buffers. This feature alone is worth noting: When extracting the older versions, the current version won't be overwrittten, the older versions will be put into the filesystem with the version number added as a suffix, e.g. foo.lisp.~1.5~.

Now, to see what was happening in between those two versions, I first looked at the differences between the two files. "M-x ediff" and selecting the two buffers, it was easy to spot the part of the code I changed. Next, have a look at the effects caused by the changes: For that I did a "C-c C-k" (slime-compile-and-load-file) on the old versions respectively, so I could then see the old behaviour in real life -- i.e., I could directly use the web app in that older version in my Firefox without any further ado. Of course, this also has to do with the fact that there weren't many interactions between the part of the app in question and other parts of the system, in particular I needn't to check out older versions of other files for the compilation to succeed. Actually, loading that old version certainly broke other functionality of the app, but that wasn't the issue: I only wanted to figure out how the old version worked wrt. to that particular part that wasn't working in my current state, hence a fully working old version wasn't what I was looking for.

Now, it did require a little more close attention to the changes I made to discover the mistake. But I think it's amazing in how few steps it's possible to regenerate an old behaviour. All the while, my current state of affairs remained largely untouched, I didn't need to create any copies or move files around or do a larger recompilation. I think that's a highly effective way of discovering the introduction of a bug.

Update: Kent Pitman, who's giving cll another of his (lately very rare) visits, pointed to one of his older papers (Accelerating Hindsight), which I'm surely have seen before but totally forgotten, which talks about how good Common Lisp is for rapid prototyping. What I've shown above is actually just an example of two points in this paper: dynamic redefinition and editor integration. Go figure.

Footnotes:
[1] Yeah, I know, using rcs or cvs and not bazaar, mercurial or git makes me look like an oldtimer. That said, I do use subversion in my day job, too. I stick with rcs and cvs for smaller (read: one person involved) projects mainly due to one reason: There is no support for the newer kids on the block in the VC shipping with XEmacs. Yes, I know that the VC shipping with FSF Emacs has support for subversion (at least), but unfortunately it's not compatible.

Getting started with CL-SQL was actually a pretty easy experience. I stumbled over one problem though, and it has a name, even. The packages shipping with Sarge are outdated. So outdated, that I couldn't get my files compiled that would otherwise work fine from the REPL or even load successfully as source files. And, hey, most of the time they would even compile with ASDF, unless I was using load-system from Slime.

Actually, that wasn't the first time I had trouble with Common Lisp packages on Debian. Getting UCW to work actually required a far newer version of sbcl than that old 0.8.12 shipping with Sarge. I built one myself, however, thanks to the work of Rene van Bevern (why isn't he on Planet Lisp?), a recent version (0.9.11) is available from backports.org, so that shouldn't be a problem anymore. New versions of CL-SQL are not available, though, so I copied the newer version of the files shipping with Breezy, which finally fixed the problem.