Friday, 29 July 2011

Output redirection in bash shell scripts

I've been writing some bash scripts recently and it took a while to get the output redirection working correctly. All I wanted to do was log script output to a file, but I kept getting stderr coming to the shell.  After a bit of research, I think I've now got a decent understanding of how this works and why.
Here's what I was doing.
./ 2>&1 >> results.log      # wrong
So I'm redirecting stderr to stdout, then redirecting stdout to a file. Except that it didn't work. stderr was still coming to the shell, and not going to the log file.
It turns out what is actualy happening is a bit different than I expected. stderr isn't being redirected to stdout, it's being redirected to wherever stdout is going right now, as the shell parses the command line, from left to right. At the point when the shell comes across the  2>&1 directive stdout is going to the shell, so that's where stderr output goes. It's only later that the >> directive sends stdout to the file.
So to get this working properly you need to do this:
./  >> results.log 2>&1     # correct
So stdout is first sent to the file, then stderr is sent to wherever stdout is going, which is the file. Hurrah!
However before decaring victory and moving on, suppose you want to send both stderr and stdout to a file and to the shell. Check this out.
./   2>&1 | tee results.log    # correct
Wait... what? How can that be correct? Well, it turns out that the shell scans the command line in several passes. The pipe command | is parsed in an earlier run-through than > or >> by the shell. So by the time the 2>&1 is parsed, the shell already knows all about the upcoming pipe and is already redirecting stdout to the tee command.
This works out quite nicely, after all you couldn't put the 2>&1 after the pipe because then it would be operating in the tee command's context.

Wednesday, 27 April 2011

Thoughts on the iPad 2

I bought an iPad 2 a few weeks ago and I'm very pleased with it. Here are some thoughts.

My habit was to listen to podcasts on my way between my home and work. Now during the time I spend sitting on the train I read using the iPad. I still keep up with my favourite podcasts such as TWiT,  The History of Rome, Philosophy Bites and In Our Time  via iTunes, but iBooks is a fantastic way to read e-books and PDFs.

Web browsing on the iPad is fun. You can sit comfortably while reading and navigation is very easy and intuitive. However it's at it's best as a passive experience. I participate in several forums and I've found that posting to them is painful. The main issue is quite a technical one. The little text editor windows that web sites provide to edit your post are fixed in size and have no scroll bars. If the text content is longer than the editor window, the iPad provides no way to move up and down inside the text document. On a desktop computer you'd use the arrow keys, but the iPad virtual keyboard doesn't have any.

Last weekend I visited my 94 year old grandmother. Over lunch we browsed through a collection of family photos on the iPad, watched some family videos, then I used it to skype to my wife and children who are in China right now. She couldn't believe she was seeing and talking to her granddaughters right there in front of her. The large screen was important as a phone's screen would have been difficult for her to see clearly.

I had a similar experience using the maps application with my mother. We used it to look at some of the places she had lived and worked during my childhood. I showed her how to drop a pin and 'walk' around the streets using streetview. She spontaneously reached out and started using it herself to look around a school where she had been head teacher. She was delighted to see that the school sign incorporated a drawing she had made of the school years before. Her attempts to navigate streetview was hit and miss at first, but clearly this was a device she instinctively felt should could use herself.

I'm building up a collection of games and apps for my girls to play when they come home with my wife in a week's time. I've also downloaded a dozen or so free books from the iBooks store. There is an extensive collection of project Gutenberg books in there. I have collections by Hans Christian Anderson, the Brothers Grimm, Rudyard Kipling and many other and can't wait to read them with the girls. I'm on a Pendragon kick as well right now so I downloaded copies of Mallory and Historia Brittonum which I'm using as companion pieces as I work my way through reading The Great Pendragon Campaign pdf.

It seems like a lot of money to spend for such simple uses, iPads are not cheap, but the quality and ease of the experience is astonishing.

Converting an application from PyQt to PySide

I'm in the middle of porting an application from PyQt to PySide. I put it off for a while because it's a big app, about a dozen files and maybe 7,000 lines of code. Some of those files are 2,000 lines or more. I blame only myself!

Actually so far it's been fairly painless, if laborious. Nokia provides reasonable documentation on the process and I'd been fairly sensible about building the app in logical functional modules so that I can build up the converted version in stages, adding functionality and testing it as I go. As far as I can tell so far, PySide is as stable and comparable in performance to PyQt.

I found that my oldest code used old style signals and slots, while the newer modules used the new improved syntax so I've been regularising them all to the new syntax as that's the only one PySide supports. Eli Bendersky has a good post on the advantages of the new style, as well as a neat trick using lambda to pass extra arguments to a slot.

Perhaps the biggest problem, in terms of drudgery, is getting rid of every QVariant and QString and replacing them with ordinary Python objects. The app uses the Qt MVC framework where values are passed to and from the model as QVariants, so I had a lot to change there. In particular, when you pass an integer to the model it arrives as a QVariant which you then 'un pack' to a Python integer and a boolean like this:

            elif column == POPULATION :
                new_index, ok = value.toInt()

The boolean just tells you whether the conversion worked or not, and I was actually just throwing it away. This now gets changed to...

            elif column == POPULATION :
                new_index = value

Much nicer!

Monday, 25 April 2011

Using python plugin scripts with py2exe

Py2exe is great. It rolls up you python application's dependencies, including the python interpreter and any stdlib or third party libraries you have included, into an executable and a few support files. Your users don't need to install Python on their Windows box, but can still run your Python application. Even if the user has Python installed, they don't need to worry about installed packages, the python version, etc. It just works!

The only problem is that now your python scripts are swallowed up inside an exe or zip file and can't be modified. Sometimes it's useful to be able to modify the behaviour of the app beyond what is convenient to do through a config file.

YAPSY is a simple python plugin system that can be confiured to scan a directory for plugin scripts. A plugin consists of a python script and an accompanying config file. Fortunately I've found that it's possible to create a py2exe project is such a way that the plugins directory and the scripts inside it are still exposed in the py2exe distribution of your app and don't get bundled away out of sight.

Py2exe is configured using a script. Here's an example, showing how to include the plugin scripts so that they are still accessible.
from distutils.core import setup
import py2exe
      author='Simon D. Hibbs',
      windows=[{'script' : 'Starbase.pyw'}],
      options={'py2exe' : {'includes' : ['sip'], 'bundle_files' : '1'}},
      data_files=[('', ['projects.ini']),
                  ('plugins', ['plugins/',

Additional plugins can be installed by the user later simply by copying the plugin script and associated yapsy config file into the plugins directory. YAPSY will then load the plugins at runtime.

So now your application is easily distributable, and also scriptable through a nicely implemented plugin system.