Friday, 24 December 2010

Choir class audition. Dec'10

When I started my studies in the Conservatoire (about three months ago), I thought that my Choir class would be quite a chore... but it has turned out to be quite enjoyable, actually. On Wednesday 22nd we took part in the yearly Christmas concert, and a video of our performance is at:

The quality of the video is horrible, but not bad given the circumstances: it was shot by a relative of mine with no prior experience with video cameras, from a bad back seat, and while she was taking care of three small children...

Sunday, 19 December 2010

Classical guitar progress logging (Dec'10)

The recording for this month is still quite rough on the edges, but here it goes:

Based on Basie by Andrew Forrest from Angel de Vicente on Vimeo.

The music is a guitar trio by Andrew Forrest: "Based on Basie", available at (where you can also get the PDF of the score:

I used OMR software to get a MIDI file of the music. I play the first guitar, and the computer plays the rest. The web page says that this piece should be for grades 4-5, and I'm just starting grade 3, so there is still plenty of room for improvement...

Technical issue

I found an issue with my Jack+Rosegarden+Qsynth setting: I realized that the computer was out of tune (which could be easily seen by using the "Free Music Instrument Tuner", and playing the "concert A", which should be 440Hz.

With help from some guys at the Ubuntu Studio mailing list, I figured out that the problem was with sampling rates in Jack and Qsynth. When both were at the same sampling rate (I tried with 48000), then everything was OK.

Thursday, 18 November 2010

Recording anything that gets played in your computer

I knew it was possible, but I had forgotten the details, so here they go...

Sometimes you might want to record audio played by an application in your computer (system sounds, music in a webpage, etc... basically anything that the computer can output to speakers/headphones). Recording that is actually very easy... In Ubuntu 10.04 I got the instructions from

I decided to try recording a song from Spotify (which is running inside Wine in my PC) using Audacity. The only extra software that I needed was PulseAudio Volume Control, and that was also the only one that I needed to configure.

When you start playing a song in Spotify and start recording in Audacity, it probably does not record anything, because Audacity will try to record from the "wrong" source. To route the sound properly, you just need to open the PulseAudio Volume Control and (while recording, otherwise this option is not available) change the option "ALSA Capture from" to "Monitor of Internal Audio Analog Stereo", as shown below, and Audacity will record them properly:

Monday, 15 November 2010

New Al Zimmermann's Programming Contest

Another one of the famous Al Zimmermann programming contests has just started. The contest runs until 12 Feb 2011, so there is plenty of time to compete. These contests are very competitive and the level is quite high, so it is difficult to stay in the top. Given that, and that I can only devote a limited amount of time to this, I will just try to compete "a little". For me, the goal will be to end within the first 10% of participants. Right now there are 125 contestants, so I should try to get in rank 12 or better. I quickly wrote a basic program and got in rank 33. Now it is time to improve it and see if I can do better than that over time. I will chart my progress below...

Update (14/2/2011): Oh well, the contest ended two days ago, and in the end I didn't get very far at all. My final position was 110, which with 362 contestants sets me at rank around 30.39%. I lost steam in the final weeks of the contest, when I realized that I had run out of easy ideas, and real work was not letting me try harder ones... Let's see if I can do better in the next contest (which is already set to start on Jul 9th).

Saturday, 13 November 2010

Editing videos from a Pentax Optio M40 camera

In a previous post I have a link to a Vimeo video, which was created by:
  1. recording a video with my Pentax Optio M40 camera
  2. cutting the beginning and the end of the video with the PiTiVi software
PiTiVi is not the video editing software that I would choose to do anything serious, but I had a problem with Kdenlive and OpenShot: the audio and video would be out of sync when importing the clip to any of them; but worse still, after cutting the beginning of the video, the problem just got much worse, and the audio would be out of sync by around a full second. With PiTiVi it was also out of sync, but only by a fraction of a second, so it was not too bad.

Today I decided to find a solution to this, so that I could edit the clip with Kdenlive, which is right now my preferred video editing software. The solution seems to use an intermediate software to create the clips to be imported to Kdenlive: Avidemux.

I just load the AVI file from the camera in Avidemux, select the starting and end points of the clip I want to use, and press the "Save" button. A screenshot can be seen here:

Then this clip can be imported into Kdenlive, without any synchronization problems. With Kdenlive we can now add a title, add effects, etc. and if we want a decent quality we can render it as the template MPEG-4 10000k 2-pass (which translates to:
f=mp4 acodec=libmp3lame ab=128k ar=44100 vcodec=mpeg4 minrate=0 b=10000k aspect=%dar mbd=2 trellis=1 mv4=1 pass=2).

The resulting file has perfect synchronization and can be uploaded to Vimeo without any problems and with much better quality than the one generated in my previous post. Here is the resulting video.

Friday, 12 November 2010

File Version Control

I'm a fan of version control systems, but for some reason I end up not using them as regularly as I should, and then I tend to forget how to do the most common operations, so this time I'm logging it here.

I do all my editing from inside Emacs, so I will be using VC, which allows me to use a version control system from within Emacs. VC works with many different version control systems, but for the moment I will use it with a very simple one: RCS. Later on, perhaps I will need a more sophisticated one, since RCS does not work over a network, and it only works at the level of individual files, but for the moment this is all I need.

The integration Emacs-RCS is very tight, and the basics for checking-in, checking-out are just performed with the same command: C-x v v. The first time we do this command, Emacs will offer to create the RCS directory, if it doesn't exist, and it will register (and unlock) the file as version 1.1. To edit the file again, just type C-x v v again to lock the file. When done and when we register the file again, the *VC-log* buffer will open, so that we can add a log entry. We continue this way, so that different versions of the file will be registered.

So now, some of the most common things that I need to do:

* Be able to see what has been changed in the file:
    - When editing the file: C-x v =
    - When writing the log entry: C-c C-d in the *VC-log* window (after saving the file being edited)

* Compare different versions:
Perhaps the most flexible one is just to type C-x v ~ REVISION, which allows us to view any previouos revision of the file, and then we can compare any two versions with something like ediff-buffers

 * See history of changes:
 C-x v l gives us the revision control change history. In a different buffer, we will have a list of all the revisions made to the file, together with the text of the log entries. This buffer is also very convenient to do some useful operations:
    - With p and n, we can go the the lines corresponding to the previous or the next revision respectively.
    - With e we can modify the log entry text.
    - With f we can visit the revision indicated at the current line (like C-x v ~ above).
    - With d see the diff between the revision at the current line and the next earlier revision.

* Undo changes:
C-x v u revert the file to the revision from which you started editing.

Later on I will add to my daily version control fix how to work with branches, but for the moment this is 90% of what I really need from a CVS.

Tuesday, 9 November 2010

Classical guitar progress logging (Nov'10)

Today I'm starting my guitar progress logging, my plan being to record and upload every month a piece that I'm learning, whatever its status, to keep a record of my progress, my struggles, etc.

We start with the Étude Simple n.VI by Leo Brouwer, which can be seen at

Étude Simple n.VI by Leo Brouwer from Angel de Vicente on Vimeo.

My camera doesn't have an input for an external microphone, so I have to rely in its poor quality built-in microphone. (Later on I hope I will be able to learn how to easily and accurately sinchronize video recorded with the camera and audio recorded directly to the computer via microphone, but until then...)

Technical stuff: after editing it with PiTiVi, I render it with the options: VGA(640x480), Audio Output: CD; Container: FFmpeg AVI format muxer; Audio Codec: TwoLAME mp2 encoder; Video Codec: FFmpeg MPEG-4 part 2 encoder

Other codecs gave me problems and the video could not be converted by the Vimeo Servers.

Monday, 8 November 2010

Free classical guitar lessons on the internet

Free classical guitar lessons on the internet

Lessons given by Jean-François Delcamp

(Professor at the Brest Conservatory of Music, Dance and Drama - France)

The lessons will begin in November 2010 and finish in June 2011.
These lessons correspond to the first three years at the conservatory
The conditions for registering are available on the forum

More details at:

Thursday, 4 November 2010

The best video editor in Linux? OpenShot vs. Kdenlive? The jury is still out.

I'm starting to create videos more or less frequently, and the software that I tried so far, while good, was not working that well. My video editing needs are simple: put together a number of sequences, mix them with audio, apply some effects and transitions (fade in, fade out, etc.), and add some titles, credits, etc., and export it all to different formats: for Internet, DVD, etc.; and obviously, all within Linux.

When I start creating videos more regularly, perhaps I change my mind about the software I've used, but so far the fight is between OpenShot and Kdenlive. The video editing software that I have used:

Pitivi, was behaving properly, it looked stable and easy to use, but transition effects are not yet implemented, only fades.

Open Movie Editor, the interface does not look very polished, although it was quite OK for some of the videos I did, but the main problem for me was that I didn't find a way to zoom in or zoom out in the timeline. This was a problem when I had very short clips (perhaps just a few seconds), because it would be difficult to select them, in order to move them around or cut them, etc.

Kdenlive, this is probably the most complete editor of them all, and it has all those features that I want and many more, but in the past it crashed on me many times. Today I'm trying it again, with a new version (, which apparently squashed many bugs, and it looks very promising. If I manage to run this version without crashes, this is probably the one that will win my heart. A screenshot of the video I'm editing right now:

OpenShot is the new kid in the block, and the interface is quite clean and it is feature rich. So far, my only gripe is that if I apply effects to consecutive video clips, sometimes artifacts are created in the audio, but I think this also happens with Kdenlive, so I need to investigate. The screenshot of the same project with Openshot:

There are a few more programs to try, (see, but I think one of these two is going to be more than enough for my needs.

Friday, 29 October 2010

Youtube videos to MP3 (or mpeg)

Today I wanted to download the music of a YouTube video, in order to show it to some friends off-line. It turns out that this is quite easy (for reference, this is using Ubuntu 10.04 LTS). (Information consulted: and

The first step is to get the aptly named script youtube-dl. The version available in Ubuntu 10.04 is 2010.04.04, but this does not work properly anymore, and it gives the following error when trying to download a video:

angelv~$ youtube-dl
[youtube] Setting language
[youtube] BTwhEPiv0U8: Downloading video info webpage
[youtube] BTwhEPiv0U8: Extracting video information
ERROR: format not available for video

If we download the latest version (2010.10.24) from, and just copy the file to /usr/bin/youtube-dl, then we have no trouble and the video (in .flv format) is saved:

angelv~$ youtube-dl
[youtube] Setting language
[youtube] BTwhEPiv0U8: Downloading video webpage
[youtube] BTwhEPiv0U8: Downloading video info webpage
[youtube] BTwhEPiv0U8: Extracting video information
[download] Destination: BTwhEPiv0U8.flv
[download] 100.0% of 7.73M at   59.74k/s ETA 00:00

angelv~$ ls -lt *flv
-rw-r--r-- 1 angelv dialout 8100631 2010-10-29 10:01 BTwhEPiv0U8.flv

Then, in order to convert it to an MP3 you will need ffmpeg and the libmp3lame0 library (and in Ubuntu 10.04, perhaps also in others, the package libavcodec-unstripped-52, so that ffmpeg cand find libmp3lame). With this in place, a small script will take care of everything:

youtube-dl --output=$x "$1"
ffmpeg -i $x -ar 44100 -ab 160k -ac 2 "$2"
rm $x

As an example:

angelv~/temp$ yt2mp3 chiclana.mp3
[youtube] Setting language
[youtube] BTwhEPiv0U8: Downloading video webpage
[youtube] BTwhEPiv0U8: Downloading video info webpage
[youtube] BTwhEPiv0U8: Extracting video information
[download] Destination: 17660.flv
[download] 100.0% of 7.73M at   59.97k/s ETA 00:00
FFmpeg version SVN-r0.5.1-4:0.5.1-1ubuntu1, Copyright (c) 2000-2009 Fabrice Bellard, et al.
  configuration: --extra-version=4:0.5.1-1ubuntu1 --prefix=/usr --enable-avfilter --enable-avfilter-lavf --enable-vdpau --enable-bzlib --enable-libgsm --enable-libschroedinger --e\
nable-libspeex --enable-libtheora --enable-libvorbis --enable-pthreads --enable-zlib --disable-stripping --disable-vhook --enable-runtime-cpudetect --enable-gpl --enable-postproc \
--enable-swscale --enable-x11grab --enable-libdc1394 --enable-shared --disable-static
  libavutil     49.15. 0 / 49.15. 0
  libavcodec    52.20. 1 / 52.20. 1
  libavformat   52.31. 0 / 52.31. 0
  libavdevice   52. 1. 0 / 52. 1. 0
  libavfilter    0. 4. 0 /  0. 4. 0
  libswscale     0. 7. 1 /  0. 7. 1
  libpostproc   51. 2. 0 / 51. 2. 0
  built on Mar  4 2010 12:41:55, gcc: 4.4.3

Seems stream 0 codec frame rate differs from container frame rate: 49.99 (4999/100) -> 25.00 (25/1)
Input #0, flv, from '17660.flv':
  Duration: 00:03:24.44, start: 0.000000, bitrate: 253 kb/s
    Stream #0.0: Video: h264, yuv420p, 320x240 [PAR 1:1 DAR 4:3], 253 kb/s, 25 tbr, 1k tbn, 49.99 tbc
    Stream #0.1: Audio: aac, 22050 Hz, stereo, s16
Output #0, mp3, to 'chiclana.mp3':
    Stream #0.0: Audio: libmp3lame, 44100 Hz, stereo, s16, 160 kb/s
Stream mapping:
  Stream #0.1 > #0.0
Press [q] to stop encoding
size=    4003kB time=204.93 bitrate= 160.0kbits/s
video:0kB audio:4003kB global headers:0kB muxing overhead 0.000781%

angelv~/temp$ ls -lt *mp3
-rw-r--r-- 1 angelv dialout 4098644 2010-10-29 10:34 chiclana.mp3

Despite its name, we can also use the script as-is to convert Youtube videos for example to mpeg format:

angelv@vaso:~$ yt2mp3 funny.mpg

UPDATE (10-Apr-2013): With Ubuntu 12.10, youtube-dl works perfect right now to download YouTube videos, and it even has an option to extract automatically the audio, so what I was achieving with this post is now a no-brainer:

angelv@pilas:~$ youtube-dl --extract-audio --audio-format=mp3
[youtube] Setting language
[youtube] suTKOmmoBtw: Downloading video webpage
[youtube] suTKOmmoBtw: Downloading video info webpage
[youtube] suTKOmmoBtw: Extracting video information
[download] Destination: suTKOmmoBtw.flv
[download] 100.0% of 13.89M at   80.60k/s ETA 00:00
[avconv] Destination: suTKOmmoBtw.mp3

Monday, 18 October 2010

From music sheet to MIDI (fourth round)

Time for SmartScore:

the Joropó piece, which comes from a PDF file generated with a music notation software (oh, well, if you look at the original score, you will see that this is far from perfect... not so Smart, after all!):

Given the previous performance, I couldn't expect too much for the old scanned document at, but for completeness sake here it goes (although is almost useless):

And what about the scanned copy of one of my music books? Well, it gets confused with some of the bars, and it certainly would require some manual tweaking.

As per PhotoScore, MIDI files cannot be exported in the demo version, so I cannot test how well the music would be encoded.

So far, from the different softwares that I have tried, I think the ones that I would to try a bit better are SharpEye and PhotoScore

Friday, 15 October 2010

Private Repository with Fedora 13 (part 1)

At my work we are considering ways to efficiently allow users to install/remove software in their Fedora workstations, but without requiring the root password, and in a controlled way (which basically means allowing them to install/remove only some packages and not others).

So I set up to try the following:
  • a Fedora13 workstation which downloads software from the standard repositories, to which regular users have no permission
  • to create a private repository in another local server, in which we can have all the software local to our workplace, and from which users can install/remove software without requiring administration privileges

So I start by downloading the 32bits Fedora 13 Desktop Edition, and installing it in a VirtualBox Virtual Machine. The Live System comes up OK inside VirtualBox, but for some reason, the "Install to Hard Drive" option does not seem to work... I'm not the first one to have this problem, and setting the "Base Memory" for the Virtual Machine to 1GB seems to solve the problem, and the installation to hard disk starts OK. Apparently after installation we don't need so much memory, so I change it back to 512MB and Fedora13 boots without any trouble:

So, we do the same for another virtual machine, the one that will act as the private repository. Next, we need to be able to boot both machines at the same time and configure networking so that they will see each other. After reading the VirtualBox instructions I decide configure each machine to have two adapters, one with NAT, and the other one as an internal network (which they will use to see each other):

And inside the guests, I configure eth1 manually with the addresses and

 Then we add two lines to the file /etc/hosts in both guests:

[root@ws ~]# head -n 2 /etc/hosts repo ws

And both machines can see each other without trouble:

Since we haven't changed any configuration regarding the Fedora repositories, right now both machines will get all the software upgrades from the standard repositories, and only root can do them.

So now, we have to be able to create a local repository in the machine repo and use it as an additional repository in machine ws, but one from which a regular user can install software.

In order to create a local repository, we follow more or less these instructions:

yum install httpd
chkconfig --levels 235 httpd on
/etc/init.d/httpd start

But we cannot contact the web server at http://repo, since the firewall is preventing it. We open it by running the command  system-config-firewall and selecting "http" as trusted. Then, from ws there is no problem to load the default Apache webpage.

Then we create the local directories:

mkdir /var/www/html/yum/SIE/fedora-13/i386

In that directory we copy the following RPM xzgv-0.9.1-2.fc13.i686.rpm and then run the createrepo command:

createrepo /var/www/html/yum/SIE/fedora-13/i386

Then we test in ws whether it is working. First we create the file SIE.repo:

(Modify to whatever metadata_expire suits you: see

And we see that a simple install would work:

But when I really try to install something, then I see that we get problems with permissions. By default Apache simply does not list the .rpm files, and we cannot download them directly (if we rename a .rpm file to a file without extension, then there is no problem...). This has something to do with SELinux! Temporarily running "setenforce 0" solves the problem (see

Installing something as root is then no problem. In order to be able to run it as a regular user, I create a script:

and add the following in the /etc/sudoers of ws (see sudoers documentation):

ALL ALL=(ALL) NOPASSWD: /usr/bin/SIE.install

This is not ideal. The user can install but cannot remove the package. We should find a better way to make sure that a regular user can do all the usual tasks with yum, but only if they affect a package coming from our local repository (search packages, install them, remove them, be informed automatically of any updates...), but for the moment we leave it like this. Later on, we have to have a good read of PackageKit to see if it provides better control for this (it uses PolicyKit).

So now we have (an alpha version!) most of the staff in place, and we should just document how to create a RPM package out of our local software, and how to make sure that users get notified of updates to our software. This will have to go in another post...

Ear-training: fourths and fifths

(A continuation of the Ear-training thread).

Wednesday, 13 October 2010

From music sheet to MIDI (third round)

Today is the time to try another piece of software: PhotoScore Ultimate 6

As with the others, we start with an easy one, the Joropó piece, which comes from a PDF file generated with a music notation software. And PhotoScore Ultimate 6 gets this:

A perfect performance. It is a pity that I could not save it to a MIDI file to try to load it with Rosegarden, since this was the Demo version, and that feature is not present.

The old scanned document at does not get recognized very well with this program either:

And for my scanned copy of my book, PhotoScore does a quite decent job, although not better than SharpEye:

 It is a pity that the demo version cannot export to MIDI, because it would be nice to be able to compare PhotoScore's generated MIDI to that generated by SharpEye.

Sunday, 10 October 2010

Ear-training: thirds

(A continuation of the Ear-training thread).

Saturday, 9 October 2010

From music sheet to MIDI (second round)

Yesterday I tried PDFtoMusic Pro and Audiveris for getting MIDI files out of music sheets. Today I'm trying another software, which looks much more promising: SharpEye.

So, let's start with the easy one, the Joropó piece, which comes from a PDF file generated with a music notation software. SharpEye gets this:

Not a single mistake for the first seven bars, for all the guitars. Not bad at all. And the generated MIDI file can be loaded into Rosegarden without any trouble.

How does SharpEye cope with the old scanned document at Well, the side-by-side view of the original document and the music recognized shows that there is still plenty of room for improvement, although it does work much better than Audiveris.

But it is true that the quality of the original is a bit poor, so let's be a bit nicer with the poor silicon brain, and let's compare Audiveris and SharpEye with a nicely scanned piece (I scan a random piece in one of my books at 600 dpi, though put it back to 300 dpi when transforming it to JPG (for Audiveris) and TIFF (for SharpEye)), and the first 9 bars of the scanned document look like:

What Audiveris gets is this:

You can see that many of the notes are correct, but this would definitely require some time to tidy it up. Worse is the fact that the MIDI created out of this seems to be broken, and when trying to load it in Rosegarden it simply refuses to do it, showing the message: "Attempt to read past MIDI file end"

SharpEye does a much better job:

For the first nine bars, it looks like SharpEye has done an excellent job. Actually, for the whole piece (24 bars), I counted only three mistakes, which is very good, but we shouldn't get too carried away... When the created MIDI file is imported into Rosegarden, the first nine bars look like this:

A lot of mistakes now... More or less OK to recognize the piece, but given the quality inside SharpEye, one would like to get a better MIDI file...

The quest for the perfect sheet-to-MIDI tool is still on!

Friday, 8 October 2010

Let's get serious about ear-training (seconds)

I seriously need to train my ear, and my ear-training companion is going to be GNU Solfege. The idea is to try (daily or very nearly) one of the many tests that are available with that software and move to the next one when I consistently (three times in a row) can get a higher than 90% accuracy in each one. So, let's start from the beginning, and plot here the performance data for  seconds:

From music sheet to MIDI (originally from PDFs to MIDI)

There is a LOT of music out there... thousands and thousands of pieces, and not much time to practice all of them, so it would be very nice to be able to get a PDF file, convert it to MIDI and quickly have an idea of how it is supposed to be, and then perhaps use that MIDI file in order to practice. So I have been trying two different software tools to do it. The first one is PDFtoMusic Pro, which is not available for Linux and is not free, and the other one  Audiveris, which is Open Source Software and will happily work in Linux.

In order to test each software, I start with the piece Joropó (for four guitars), which can be downloaded from

PDFtoMusic Pro

I don't have Windows, but I installed this in my Ubuntu 10.04 machine with Wine 1.1.42, and it runs without any (apparent) trouble.

I had no trouble at all to load the file, and to play directly inside it, without exporting it to MIDI, but actually I'm more interested in being able to export it to a MIDI file, so that later I can tweak it from inside Rosegarden or similar.

Exporting the file (or only selected pages) to MIDI is trivial, and when loaded in Rosegarden, there is no problem with playback. If we want to edit the file with the Notation Editor in Rosegarden, the result is pretty good, as can be seen from the first measures:

In order to get the same notation as the original score, we would have to tweak some of the sharps and turn them into flats, connect some notes with ligados, and change the clef of the fourth guitar, but those are quite small things.

If we make changes to the score and then want to print the music, then the result is also quite good from within Rosegarden. In this case, I just changed the clef of the fourth guitar, and printed it from within Rosegarden (which uses Lilypond as the printing engine), and the result was quite acceptable:

So, with PDFs that are generated with a music notation software, PDFtoMusic Pro works very well, but it cannot handle scanned music. For instance, when trying to load the file it will come up with the message:


Audiveris is different from PDFtoMusic Pro (besides being Open Source), in which it is actually an OMR software and it will scan any document for music, even if it hasn't been generated by a music notation software.

I start Audiveris like:

angelv@vaso:~/audiveris$ java -Xmx256M -jar dist/audiveris-3.4.jar

Audiveris can only handle one page at at time, so we decide to go for the first page, which I save as a PDF (with evince), but the result when loaded is just not there:

So, I convert it to png first, but then Audiveris cannot even load it:

angelv@vaso:~/temp$ convert -density 300 first_page.pdf first_page.png


omr.sheet.picture.PictureLoader.loadFile( -- INFO: Loading image from /home/angelv/temp/first_page.png ...
 omr.sheet.picture.PictureLoader$Listener.imageComplete( -- INFO: Image loading complete
 omr.step.SheetSteps$LoadTask.doit( -- WARNING: Exception [java.lang.RuntimeException: Unsupported pixel size:16]
 omr.step.Step.doStepRange( -- WARNING: Processing aborted [omr.step.StepException: java.lang.RuntimeException: Unsupported pixel size:16]

With JPG all looks much better:

convert -density 300 first_page.pdf first_page.jpg

And all is fine now:

So far we have only loaded the image. If we now want to perform the OMR step, we just go to the menu Step-Score, and we get the scanned music. If you look carefully below, there are some mistakes, but most of it is in the right place.

Now, in theory one can export this to MusicXML format and use something like Lilypond to create a printable score, but something is not working. If I export it to MusicXML (Score-Export), and then try to use musicxml2ly I get the following errors:

angelv@vaso:~/temp$ musicxml2ly first_page.xml
Reading MusicXML from first_page.xml ...
Traceback (most recent call last):
  File "/usr/bin/musicxml2ly", line 2969, in
  File "/usr/bin/musicxml2ly", line 2964, in main
    voices = convert (filename, options)
  File "/usr/bin/musicxml2ly", line 2874, in convert
    (voices, staff_info) = get_all_voices (parts)
  File "/usr/bin/musicxml2ly", line 2583, in get_all_voices
    all_voices = voices_in_part_in_parts (parts)
  File "/usr/bin/musicxml2ly", line 2573, in voices_in_part_in_parts
    voices = voices_in_part (p)
  File "/usr/bin/musicxml2ly", line 2560, in voices_in_part
    part.interpret ()
  File "/usr/share/lilypond/2.12.3/python/", line 676, in interpret
    int (attributes_dict.get ('divisions').get_text ()))
ValueError: invalid literal for int() with base 10: '4.0'

But I don't really care, as I'm interested in getting MIDI stuff (a basic PDF file can be created OK with Audiveris' own Score-PrintPDF option), though this will have the mistakes present in the scan, so it is not that useful...

But now let's go for the real thing: getting a MIDI file and playing it. Midi-Store MIDI will create a MIDI file, which then can be uploaded to a MIDI sequencer, in my case Rosegarden. Without any modification, the MIDI is more or less OK, though obviously the mistakes that were present in the scan are there, so a crucial part will be the ability to edit the score manually.

So, if we open the score editor in Rosegarden, we see that the notes are in there (the third guitar's first three bars don't show up when opening the editor in multitrack mode, but they are there if we open the editor just for the guitar's part), but they are all messed up (different clef for 3rd and 4th guitar, different key, different time signature,...).

How difficult is to change it back to a recognizable score?

Well, I did try to adjust it without looking at the Rosegarden documentation, but it was certainly not that trivial, so I decided to follow one of the Rosegarden tutorials to learn the basics, and then come back to this Joropó score, to try to adjust it properly. I followed the following Rosegarden tutorial, but even after reading that, it looks like doing all the modifications to get a score similar to the original one would be quite costly. A simpler task would be just to modify the MIDI file to "sound" like the original, but it is certainly going to take also some time to do it right.

So, if your music is in a PDF generated by a music notation software and you are happy to pay for it, then certainly PDFtoMusic Pro does a much better job. But how can Audiveris handle scanned music? As with PDFtoMusic Pro, I tried to scan the first page of but I'm afraid that the result speaks for itself...:

So, I'm afraid that so far Audiveris is work in progress, but cannot really compete with PDFtoMusic Pro. I will continue my quest to see if I can find some way of making MIDI files out of scanned music sheets, but for the moment it looks like either they are PDF files produced with music notation software (in which case I can use PDFtoMusic Pro) or I will have to type them by hand...

Monday, 4 October 2010

First Steps with Music Software in Linux

There are not (so far) many musical things that I want to do in the computer, but playing a MIDI file; mixing it with my own recording of playing one of the parts, and exporting it all into an audio file is one of them, so today I set some time to learn the basics.  In the computer I did this I'm running Ubutu 10.04, and I have followed the instructions in the Ubuntu Studio pages to quickly install all the software that I need (and much more), and to configure it.

With that in place, I rebooted the PC into the RealTime Kernel, and then I started by running JACK, Qsynth and Rosegarden. (Qsynth needs some soundfonts, and I downloaded some into the file /usr/share/sounds/sf2/FluidR3_GM.sf2. I don't remember where I got them from, but you can get them, for example, at This looks like:

Then, it is time to check that all connectivity is working OK. The "Connect" button in the JACK interface will show the connections available. In my case:

This shows that the "General MIDI Device" in Rosegarden is connected to FLUID Synth (QSynth is just the GUI for fluidsynth), so all should be OK.

Then in Rosegarden we load a MIDI file, and select the instruments you want, modify the volume, reverb, etc. It took me a while (perhaps it should have been faster if I had read the documentation) to figure out how to select different instruments for each track, but it is actually quite easy. Just select one of the tracks:

Then say which instrument number you are going to use for that track:

My confusion for a while was to trust the name of what it would appear in the "Instrument" description, before I actually configured each instrument sound. So, the description in the "Instrument" field can be wrong. I guess it keeps whatever it was used last time with whatever synthesizer, so don't trust it, and instead, once you have selected the instrument number, you should specify the sound to be used for that instrument number:

With that in place (you will see that now the Instrument description in the track parameters section above reflects the change made here), you can now use Rosegarden to play the MIDI file with whatever instruments you choose.

So now we just have to record it into an audio file. This is done quite easily by just routing the output of Qsynth to Ardour. First, we will need to create at least one track with Ardour (here in the example: Audio1):

 Then, in JACK we will have to connect the output from Qsynth to Ardour:

And lastly is just a matter of recording (first we have to select in which track we will be recording, in this case Audio 1,and then proceed with the "main" recording):

Adding other audio tracks (from an external mic, another software, etc.) would follow similar steps in Ardour. The last step is just to "export session to audiofile".

A  basic MIDI-performance obtained in this way can be found at vimeo

Thursday, 30 September 2010

Back to student life...

Well, not really! From the student life I will only get the studying part... partying, dating, drinking, etc. are out...

This week we have started all the classes at the Conservatoire. I'm not sure I will be able to follow all of them, but I'm certainly going to try. In total I have 6.5 "contact" hours per week:

Guitar: 1
Ensemble: 1
Harmony: 2
Choral: 1.5
Chamber music: 1

Except for Chamber music, I had already one lesson of each of these.

In the guitar class, we have started directly with the Leo Brouwer simple studies n. VI and VII (they don't look particularly difficult, though they will need some time).

In the ensemble class, we started with two pieces for guitar quartet by Andrew Forrest (Joropó and Barcarowl), which can be downloaded at:

In harmony class, we started with the basic theory on triads, cadences, etc. I was hoping I could follow the Walter Piston book, and skip the lessons, but it looks like the teacher is going to follow a slightly different method, so I guess I will better attend as many lessons as possible.

In choral music class, we started by singing two pieces: "Adoramus Te" by W.A. Mozart and "Hymne a la Nuit" de J.P. Rameau.

Overall I don't think is going to be as difficult as I was expecting, though my intonation skills could certainly be improved, so I will have to devote some time to ear-training stuff. I would gladly do without the choral music class, but I guess at least it will be a way of forcing myself to not forget these ear-training exercises...

Let's see how I can cope with all this extra work in my already quite-busy life...

(Update: 4-Nov-2010)

It was a bit too much, so I decided to drop one of the courses: Chamber Music, which I will to do next year. My current plan is to do the remaining four years to the professional degree in six years (perhaps it is also a bit ambitious, since the higher the year, the higher the number of courses to take), but it is a bit more realistic. Let's see how it goes...

Friday, 24 September 2010

Building A Virtual Cluster with Xen

Not new stuff, but today I was looking for some information on this, and I just realized that I didn't have a link to this article which I posted some years ago at the ClusterMonkey site. I'm not sure how many things would need updating, but the ideas still hold.

The article is in two parts:



Friday, 30 July 2010

Sphere Online Judge (SPOJ) progress

SPOJ is one of the main sites that host programming challenges. The problems in SPOJ are for the most part much easier than the ones in Project Euler (which I have also posted about, although I have neglected it for several months...), so they are quite nice to try to solve in new programming languages. At this moment there are >10K users who have submitted at least one correct solution to one of the programs, so the place is very active. The graph below will show my progress in SPOJ, although I don't really intend to compete. I will only look for problems that look interesting to develop my programming skills.

Tuesday, 20 July 2010

Debugging Fortran+MPI codes with "The Portland Group" debugger (pgdbg)

I have to admit that I didn't get the Fox algorithm in my previous post correct the very first time, so I had to do some parallel debugging. In the past, I have used gdb, and the Intel Debugger, and now it was time to try pgdbg (The Portland Group debugger). Next in line is TotalView.

At our institution the Portland compiler's version installed is 10.5, but this had some issues with my current workstation linux distribution (Ubuntu 10.04 64 bits), so I installed the current latest version: 10.6. Installation was very simple with their install script, and once the license file was in place it was time to try it.

Compilation of the code can be done with the included MPICH1, with the following command:

angelv@vaso:~/fox$ pgf90 -o fox -Mmpi=mpich1 -g fox.f90

Since we will be using ssh to connect to the other processors (actually just a number of processes all running in my local workstation), we need to first get the ssh security sorted out (tips from the "PGI Tools Guide" documentation, page. 90). We generate the ssh keys with a passphrase (and copy them to the authorized keys):

$ ssh-keygen -t dsa

$ cd $HOME/.ssh
$ cp authorized_keys

And then, from a new terminal we will just have to do the following, and enter the passphrase just once, and all subsequent ssh connections will be passwordless:

$ eval `ssh-agent -s`
$ ssh-add

With this in place, we can run our code with the included MPICH1 version:

angelv@vaso:~/fox$ mpirun -stdin -np 4 ./fox

In order to run it with the debugger, we just add the option -dbg=pgdbg:

angelv@vaso:~/fox$ mpirun -stdin -dbg=pgdbg -np 4 ./fox

The following image shows a moment during the debugging session, where 4 processes have been created, and we are at the end of the first stage in the Fox algorithm. The window in the bottom shows how you can easily see the values of variables (whole matrixes included, which can be indexed according to Fortran syntax) for all (or a selection of) processes involved in the computation.

I need to try it for a longer period, but overall it looks like a very usable parallel debugger. The Portland Group has a video demo of the debugger here.

Monday, 19 July 2010

Fox algorithm for matrix multiplication in parallel with Fortran90+MPI

I'm now re-reading the book "Parallel Programming with MPI" by Peter S. Pacheco and doing some of the exercises in there. An interesting one is the Programming Assignment n.1 in page. 133, which involves fully implementing the Fox parallel algorithm for multiplying matrixes (see for example).

Below is Fortran90 code (version 1... some things need to be improved...) that does it (a better looking source code is at Pastebin, although I'm not sure for how long it will stay there...)

  include "mpif.h"

  INTEGER :: procs, rank, error

  INTEGER :: g_order, g_side, my_g_row, my_g_column, my_g_rank, comm, row_comm, col_comm, block_mpi_t
  INTEGER :: rows
  REAL, DIMENSION(:,:), ALLOCATABLE :: matrixA, matrixB, matrixC, localA, tempA, localB, localC
  CALL MPI_Init ( error )
  CALL MPI_Comm_size ( MPI_COMM_WORLD, procs, error )
  CALL MPI_Comm_rank ( MPI_COMM_WORLD, rank, error )
  CALL Read_Matrix(rank,error,rows,matrixA,matrixB,matrixC)
  CALL Setup_grid(g_order,g_side,error,comm,my_g_rank,my_g_row,my_g_column,row_comm,col_comm)
  CALL Distribute_matrixes(g_side,rows,block_mpi_t,error,matrixA,matrixB,localA,localB,localC,my_g_row,my_g_column)
  CALL Perform_Fox_Algorithm(my_g_row,my_g_column,g_order,localA,tempA,localB,localC,row_comm,col_comm,status,error)
  CALL Print_Last_Result(rank, my_g_rank, g_side, comm, procs, error, status, localC, matrixC, tempA)

  CALL MPI_Finalize (error)

  SUBROUTINE Print_Last_Result(rank, my_g_rank, g_side, comm, procs, error, status, localC, matrixC, tempA)
    INTEGER, INTENT(IN) :: rank, my_g_rank, g_side, comm, procs
    INTEGER, INTENT(OUT) :: error
    REAL, DIMENSION(:,:), INTENT(IN) :: localC
    REAL, DIMENSION(:,:), INTENT(OUT) :: matrixC, tempA

    INTEGER :: grow,gcol,i
    INTEGER,DIMENSION(2) :: coordinates

    IF (rank .EQ. 0 .AND. my_g_rank .NE. 0) PRINT*, "Houston, we have a problem with I/O"

    CALL MPI_SEND(localC,g_side*g_side,MPI_REAL,0,0,comm,error)

    IF (rank .EQ. 0) THEN
       DO i=1,procs
          CALL MPI_Recv(tempA,g_side*g_side,MPI_REAL,MPI_ANY_SOURCE,0,comm,status,error)
          CALL MPI_Cart_coords(comm,status(MPI_Source),2,coordinates,error)
          grow = coordinates(1)
          gcol = coordinates(2)
          matrixC(grow*g_side+1:(grow+1)*g_side,gcol*g_side+1:(gcol+1)*g_side) = tempA
       END DO
    PRINT*,"============ Matrix multiplication in parallel ==================="
    CALL Print_Matrix(matrixC)
    END IF
  END SUBROUTINE Print_Last_Result

  SUBROUTINE Print_Matrix(M)
    REAL, DIMENSION(:,:) :: M
    INTEGER :: side,row,column

    side = SIZE(M,1)

    DO row=1,side
       DO column=1,side-1
          WRITE(*,'(F10.3)',ADVANCE='NO'), M(row,column)
       END DO
       WRITE(*,'(F10.3)'), M(row,side)
    END DO

  SUBROUTINE Perform_Fox_Algorithm(my_g_row,my_g_column,g_order,localA,tempA,localB,localC,row_comm,col_comm,status,error)
    INTEGER, INTENT(IN) :: my_g_row,my_g_column,g_order,row_comm,col_comm
    INTEGER, INTENT(OUT) :: error
    REAL, DIMENSION(:,:), INTENT(IN) :: localA
    REAL, DIMENSION(:,:), INTENT(INOUT) :: tempA,localB,localC

    INTEGER :: source,destination,stage,bcast_root

    localC = 0
    source = MOD(my_g_row + 1,g_order)
    destination = MOD(my_g_row + g_order - 1,g_order)

    DO stage=0,g_order-1
       bcast_root = MOD(my_g_row + stage,g_order)

       IF (my_g_column .EQ. bcast_root) THEN
          CALL MPI_BCAST(localA,g_side*g_side,MPI_REAL,bcast_root,row_comm,error)
          CALL Matrix_Multiply(localA,localB,localC)
          CALL MPI_BCAST(tempA,g_side*g_side,MPI_REAL,bcast_root,row_comm,error)
          CALL Matrix_Multiply(tempA,localB,localC)
       END IF
       CALL MPI_Sendrecv_replace(localB,g_side*g_side,MPI_REAL,destination,0,source,0,col_comm,status,error)
    END DO
  END SUBROUTINE Perform_Fox_Algorithm

  SUBROUTINE Matrix_Multiply(A,B,C)
    INTEGER :: side,row,column

    side = SIZE(A,1)

    DO row=1,side
       DO column=1,side
          C(row,column) = C(row,column) + SUM(A(row,:)*B(:,column))
       END DO
    END DO
  END SUBROUTINE Matrix_Multiply

  SUBROUTINE Distribute_matrixes(g_side,rows,block_mpi_t,error,matrixA,matrixB,localA,localB,localC,my_g_row,my_g_column)
    INTEGER, INTENT(IN) :: g_side,rows,my_g_row,my_g_column
    INTEGER, INTENT(OUT) :: block_mpi_t,error
    REAL, DIMENSION(:,:), ALLOCATABLE, INTENT(OUT) :: localA,localB,localC
    REAL, DIMENSION(:,:), INTENT(INOUT) :: matrixA,matrixB

    CALL MPI_TYPE_VECTOR(g_side,g_side,rows,MPI_REAL,block_mpi_t,error)
    CALL MPI_TYPE_COMMIT(block_mpi_t, error)

    CALL MPI_BCAST(matrixA,rows*rows,MPI_REAL,0,MPI_COMM_WORLD,error)
    CALL MPI_BCAST(matrixB,rows*rows,MPI_REAL,0,MPI_COMM_WORLD,error)

    localA = matrixA(my_g_row*g_side+1:(my_g_row+1)*g_side,my_g_column*g_side+1:(my_g_column+1)*g_side)
    localB = matrixB(my_g_row*g_side+1:(my_g_row+1)*g_side,my_g_column*g_side+1:(my_g_column+1)*g_side)
  END SUBROUTINE Distribute_matrixes

  SUBROUTINE Setup_grid(g_order,g_side,error,comm,my_g_rank,my_g_row,my_g_column,row_comm,col_comm)
    INTEGER, INTENT(OUT) :: g_order, g_side,error,comm,my_g_rank,my_g_row,my_g_column,row_comm,col_comm

    INTEGER,DIMENSION(2) :: dimensions,coordinates
    LOGICAL,DIMENSION(2) :: wrap_around,free_coords

    g_order = SQRT(REAL(procs))
    g_side = rows / g_order

    dimensions = g_order
    wrap_around = .TRUE.
    CALL MPI_Cart_create(MPI_COMM_WORLD,2,dimensions,wrap_around,.TRUE.,comm,error)
    CALL MPI_Comm_rank ( comm, my_g_rank, error )
    CALL MPI_Cart_coords(comm,my_g_rank,2,coordinates,error)
    my_g_row = coordinates(1)
    my_g_column = coordinates(2)

    free_coords(1) = .FALSE.
    free_coords(2) = .TRUE.
    CALL MPI_Cart_sub(comm,free_coords,row_comm,error)

    free_coords(1) = .TRUE.
    free_coords(2) = .FALSE.
    CALL MPI_Cart_sub(comm,free_coords,col_comm,error)

  SUBROUTINE Read_Matrix(rank,error,rows,matrixA,matrixB,matrixC)
    INTEGER, INTENT(IN) :: rank
    INTEGER, INTENT(OUT) :: rows,error
    INTEGER :: i
    REAL, DIMENSION(:,:), ALLOCATABLE, INTENT(OUT) :: matrixA,matrixB,matrixC

    IF (rank .EQ. 0) THEN
       READ*, rows
       DO i=1,rows
          READ*, matrixA(i,:)
       END DO
       DO i=1,rows
          READ*, matrixB(i,:)
       END DO

       !! Calculate the matrix multiplication for comparison                                                                                                                          
       matrixC = 0
       CALL Matrix_Multiply(matrixA,matrixB,matrixC)

       PRINT*, "=======MATRIX A=========="
       CALL Print_Matrix(matrixA)
       PRINT*, "=======MATRIX B=========="
       CALL Print_Matrix(matrixB)
       PRINT*, "=======MATRIX C=========="
       CALL Print_Matrix(matrixC)
    END IF


And given the following matrixes:

angelv@vaso:~/fox$ cat
3.4 4.2 2.1 9.3 5.6 7.8 3.4 6.3
6.6 7.6 2.1 7.2 3.5 7.9 2.0 8.3
8.6 4.1 7.5 3.5 7.1 9.6 4.1 8.1
5.8 4.7 8.6 3.6 7.4 4.3 1.2 0.2
9.9 5.3 8.9 3.7 6.5 9.7 3.6 5.3
8.2 4.7 9.3 3.8 6.7 8.4 4.7 6.4
6.8 8.0 2.8 8.9 9.8 3.2 6.8 8.7
6.4 7.6 8.3 4.3 6.5 4.1 3.9 3.8

3.5 5.7 8.6 4.6 7.5 9.0 2.8 6.0
8.9 3.4 7.3 3.3 5.3 8.2 3.6 3.9
8.3 0.8 2.1 1.6 4.7 6.4 3.4 8.8
7.3 4.3 3.3 2.7 4.5 6.6 9.3 5.9
6.7 6.7 8.0 2.8 3.7 5.7 7.7 8.3
7.8 4.0 8.9 2.9 1.4 0.8 7.8 8.2
3.4 7.1 5.7 9.0 6.1 7.9 8.9 7.7
2.6 4.1 6.9 8.2 9.3 5.0 7.1 4.6

We can compile it and run it as:

angelv@vaso:~/fox$ pgf90 -Mmpi=mpich -o fox fox.f90

angelv@vaso:~/fox$ mpirun -stdin -np 4 ./fox
 =======MATRIX A==========
     3.400     4.200     2.100     9.300     5.600     7.800     3.400     6.300
     6.600     7.600     2.100     7.200     3.500     7.900     2.000     8.300
     8.600     4.100     7.500     3.500     7.100     9.600     4.100     8.100
     5.800     4.700     8.600     3.600     7.400     4.300     1.200     0.200
     9.900     5.300     8.900     3.700     6.500     9.700     3.600     5.300
     8.200     4.700     9.300     3.800     6.700     8.400     4.700     6.400
     6.800     8.000     2.800     8.900     9.800     3.200     6.800     8.700
     6.400     7.600     8.300     4.300     6.500     4.100     3.900     3.800
 =======MATRIX B==========
     3.500     5.700     8.600     4.600     7.500     9.000     2.800     6.000
     8.900     3.400     7.300     3.300     5.300     8.200     3.600     3.900
     8.300     0.800     2.100     1.600     4.700     6.400     3.400     8.800
     7.300     4.300     3.300     2.700     4.500     6.600     9.300     5.900
     6.700     6.700     8.000     2.800     3.700     5.700     7.700     8.300
     7.800     4.000     8.900     2.900     1.400     0.800     7.800     8.200
     3.400     7.100     5.700     9.000     6.100     7.900     8.900     7.700
     2.600     4.100     6.900     8.200     9.300     5.000     7.100     4.600
 =======MATRIX C==========
   260.900   194.020   272.070   178.530   210.450   236.380   297.220   275.730
   274.180   199.380   307.390   197.010   245.450   266.250   285.240   277.610
   311.840   232.300   352.690   225.580   277.280   303.160   320.440   360.720
   247.510   147.520   219.820   111.300   167.610   225.640   198.500   256.890
   327.930   227.120   350.150   209.450   269.700   313.690   306.850   365.810
   318.490   224.600   336.210   216.270   271.960   310.980   311.220   361.910
   319.570   268.880   357.800   255.450   309.740   359.100   362.840   349.110
   288.990   190.670   279.080   175.760   235.560   291.560   257.210   301.530
 ============ Matrix multiplication in parallel ===================
   260.900   194.020   272.070   178.530   210.450   236.380   297.220   275.730
   274.180   199.380   307.390   197.010   245.450   266.250   285.240   277.610
   311.840   232.300   352.690   225.580   277.280   303.160   320.440   360.720
   247.510   147.520   219.820   111.300   167.610   225.640   198.500   256.890
   327.930   227.120   350.150   209.450   269.700   313.690   306.850   365.810
   318.490   224.600   336.210   216.270   271.960   310.980   311.220   361.910
   319.570   268.880   357.800   255.450   309.740   359.100   362.840   349.110
   288.990   190.670   279.080   175.760   235.560   291.560   257.210   301.530

Thursday, 8 July 2010

Classical Guitar Forum

About a month ago I joined the Delcamp Classical Guitar Forum, and so far I've found it amazing. This one is the Forum in English, though there are also sites in French, Italian and Spanish.

There are a lot of people participating (composers, guitar players, luthiers), the conversations are very well kept to the subject (with well organized topics), and the people are really helpful and friendly.

And the resources are also very good. In it you can find information about technique, many recordings of common repertoire pieces, plenty of music scores, etc.

All in all, this is by far the best place in the Internet for sharing guitar experiences with other musicians. If you happen to stumble here and have any interest in Classical Guitar, make sure you visit the Delcamp Forum.

Thanks Jean-François Delcamp for starting this out!

Saturday, 3 July 2010

Opera Mail vs. Thunderbird 3.1 vs. GMail

During the last seven months I've been using the Opera Web browser and its integrated Mail Reader, but the experience started to become a bit painful, so I am going back to Firefox/Thunderbird.

Some of the things that pushed me to do this were:
  • With the Opera browser I was not aware of anything like FoxyProxy, which is very useful for me when accessing my company intranet.
  • My web bank page did not show properly (perhaps their fault?) with Opera.
  • Had issues with logging in my favourite forum with Opera (oddly enough not in all computers).
  • The full editor of blogger does not display properly in Opera.
  • I didn't notice Opera browser being particularly faster than Firefox.
  • Opera Mail did not let me put my own tags to messages, which was a bit annoying sometimes.
  • Opera Mail did not let me configure my mail servers with not standard ports, which was annoying if I wanted to access my company servers through SSH tunnels.
  • In a few occasions Opera Mail just stopped working, and after restarting all the messages would be gone (the local copy only), so it had to download them again from Google Mail.
  • Lately Opera Mail started to think that I had millions of messages and would never finish synchronizing with Gmail. After restarting it, it would recognize again properly that I have only thousands.

So, I decided to try the new Thunderbird 3.1 for e-mail, and if it was up to the job, then go back to the Firefox/Thunderbird tandem. So, I collected the things that I need from a mail reader and did a comparison of Thunderbird, Opera Mail, and Gmail. I'm not interested in doing a comparison of all their features, only the ones that I tend to use. I haven't looked in any depth at the documentation of any of the readers, so some of the things here might be wrong. If you find that any of this information is incorrect or that there are better ways of doing something, please let me know, I'd love to hear it. Thunderbird did now manage to do everything I needed, and I don't get the problems mentioned above (plus it is open source software), so I'm now migrating to Firefox/Thunderbird... for how long?

Here it is a list of things that I want my mail reader to do:

  • No folders, use tags instead
Folders are a nuisance, much better tags. You can apply several tags to the same message, and when searching for a particular tag is like you put a message in different folders at the same time.

- GMail: tags are called labels, and you can easily create arbitrary new labels and assign as many as you want to each message.

- Thunderbird: you can also create Tags and assign as many as you want to each message.

- Opera Mail: as far as I know, only the prebuilt tags.

  • Don't mark messages as read automatically

- GMail apparently marks a message as read as soon as you open it. I don't like this at all. I don't even want to mark a message as read after a given set of time. For me "unread" means that I still have to do something with it, so I want to mark it as "read" only when I decide.

- Thunderbird. No problem. In Preferences, Advanced, unmark "Automatically mark messages as read", and I have to mark them as read explicitly.

- Opera Mail. This was also easily configurable.

  • Efficient searching and filtering of messages

This is probably the most important feature for me, and for basic searches I think the three readers are very good. For "incremental searches" I think Gmail is the winner, but Thunderbird 3.1 second.

By incremental search I mean a situation like this: I want to find all mails that have something to do with "braemore statements", and once I found these, I want to find which ones have the word "winter" in the body.

- GMail. This is very good for incremental searches. Type "braemore statements" and a list of mail appears, then type "winter" and press again and bingo.

- Thunderbird. If I type all the words of an incremental search in the "Search all messages" box (thus not making it really incremental), then I also get the result, but ideally one would only type "braemore statements", then click on "Open as list" and then do a "Filter these messages", including the Body, but this does not work well and it seems to hang forever. (Filtering by body does seem to work at least in the Inbox, but it looks like it is really slow, while the Ctrl+K search is really fast. Go figure....).

- Opera Mail. The search option was very fast and quite good in general. For incremental searches it is very good. Type "braemore statements" in the general search box, then one the mails have been found, type "winter" in the quick filter, and bingo! This is what I was hoping it would work in Thunderbird as well, but for some reason it is not just yet.

  • Good keyboard shortcuts
-  Gmail apparently does have shortcuts, but I doubt that they are comparable to Thunderbird or Opera in this respect.

  • Make "read" messages disappear from view but keep them available
- GMail: I'm sure there has to be a way to do this, but it was not obvious.

- Thunderbird. In my Inbox I just click on the "Unread" quick filter and bingo. I only see the "unread" ones but all the rest are somewhere in there. Much cleaner view.

- Opera Mail also does this without any issues.

  • Good display of HTML messages
- Obviously no trouble with this in Gmail, and Thunderbird does it very well also. With Opera Mail I had trouble with quite a few HTML formatted messages. I don't like HTML messages, but people insist on sending them, so I guess I have to be able to read them.

  • Possibility of deleting attachments
- I could not find how to do this in either Gmail or OperaMail. With Thunderbird is just one of the options of the attachment itself.
  • Different personalities 
 This is useful for sending personal mails, or mails to newsgroups, etc. with different mail addresses and/or signature.

- Gmail: I don't think this is possible at all.

- Thunderbird:  it has support for different personalities, and I have used it in the past. Very convenient. This, together with the possibility of using non-conventional ports was very useful for me. Different personalities with Google Mail as SMTP server does not seem to work (Gmail at the end sends the message with you gmail address, and that's all), but when using non-standard port for my SMTP server, I can send (regardless of where I am) mail through my company SMTP server (via ssh tunnels). This allows me to fake my e-mail address (to avoid spam) when sending messages to newsgroups, and to send mail to internal mailing lists at our company (which rejects mails if coming from an outside address).

- OperaMail. I think this is only possible if you have different mail accounts, so you can select which one to use every time you send a message, but I don't think it is possible having, for instance, different signatures associated with the same mail account.

  • Automatic filtering of messages
OperaMail has something that is very useful: The automatic views of mails sent to newsgroups. With Thunderbird is not so automatic, but it does require little work: one can create Search Folders (Ctrl+Shift+F), where one can create complicated searches, and then save it as a search folder. For instance, I created a search folder called "Direct", where I only see mail that is directed personally to me, another search folder where I only see mail that has a particular mailing list address in the "To or CC" fields, etc. This is similar to the old folder mentality, but the messages don't go into any folder, they all can stay in the same folder, and this is just different filtered views of it. Much more powerful and convenient.