When I started my studies in the Conservatoire (about three months ago), I thought that my Choir class would be quite a chore... but it has turned out to be quite enjoyable, actually. On Wednesday 22nd we took part in the yearly Christmas concert, and a video of our performance is at: http://vimeo.com/18144039
The quality of the video is horrible, but not bad given the circumstances: it was shot by a relative of mine with no prior experience with video cameras, from a bad back seat, and while she was taking care of three small children...
Friday, 24 December 2010
Sunday, 19 December 2010
Classical guitar progress logging (Dec'10)
The recording for this month is still quite rough on the edges, but here it goes: http://vimeo.com/17986724
The music is a guitar trio by Andrew Forrest: "Based on Basie", available at http://www.forrestguitarensembles.co.uk/trios.html (where you can also get the PDF of the score: http://www.forrestguitarensembles.co.uk/scores/trios/T0015BasedonBasie.pdf).
I used OMR software to get a MIDI file of the music. I play the first guitar, and the computer plays the rest. The web page says that this piece should be for grades 4-5, and I'm just starting grade 3, so there is still plenty of room for improvement...
Technical issue
I found an issue with my Jack+Rosegarden+Qsynth setting: I realized that the computer was out of tune (which could be easily seen by using the "Free Music Instrument Tuner", and playing the "concert A", which should be 440Hz.
With help from some guys at the Ubuntu Studio mailing list, I figured out that the problem was with sampling rates in Jack and Qsynth. When both were at the same sampling rate (I tried with 48000), then everything was OK.
Based on Basie by Andrew Forrest from Angel de Vicente on Vimeo.
The music is a guitar trio by Andrew Forrest: "Based on Basie", available at http://www.forrestguitarensembles.co.uk/trios.html (where you can also get the PDF of the score: http://www.forrestguitarensembles.co.uk/scores/trios/T0015BasedonBasie.pdf).
I used OMR software to get a MIDI file of the music. I play the first guitar, and the computer plays the rest. The web page says that this piece should be for grades 4-5, and I'm just starting grade 3, so there is still plenty of room for improvement...
Technical issue
I found an issue with my Jack+Rosegarden+Qsynth setting: I realized that the computer was out of tune (which could be easily seen by using the "Free Music Instrument Tuner", and playing the "concert A", which should be 440Hz.
With help from some guys at the Ubuntu Studio mailing list, I figured out that the problem was with sampling rates in Jack and Qsynth. When both were at the same sampling rate (I tried with 48000), then everything was OK.
Thursday, 18 November 2010
Recording anything that gets played in your computer
I knew it was possible, but I had forgotten the details, so here they go...
Sometimes you might want to record audio played by an application in your computer (system sounds, music in a webpage, etc... basically anything that the computer can output to speakers/headphones). Recording that is actually very easy... In Ubuntu 10.04 I got the instructions from https://wiki.ubuntu.com/PulseAudio
I decided to try recording a song from Spotify (which is running inside Wine in my PC) using Audacity. The only extra software that I needed was PulseAudio Volume Control, and that was also the only one that I needed to configure.
When you start playing a song in Spotify and start recording in Audacity, it probably does not record anything, because Audacity will try to record from the "wrong" source. To route the sound properly, you just need to open the PulseAudio Volume Control and (while recording, otherwise this option is not available) change the option "ALSA Capture from" to "Monitor of Internal Audio Analog Stereo", as shown below, and Audacity will record them properly:
Sometimes you might want to record audio played by an application in your computer (system sounds, music in a webpage, etc... basically anything that the computer can output to speakers/headphones). Recording that is actually very easy... In Ubuntu 10.04 I got the instructions from https://wiki.ubuntu.com/PulseAudio
I decided to try recording a song from Spotify (which is running inside Wine in my PC) using Audacity. The only extra software that I needed was PulseAudio Volume Control, and that was also the only one that I needed to configure.
When you start playing a song in Spotify and start recording in Audacity, it probably does not record anything, because Audacity will try to record from the "wrong" source. To route the sound properly, you just need to open the PulseAudio Volume Control and (while recording, otherwise this option is not available) change the option "ALSA Capture from" to "Monitor of Internal Audio Analog Stereo", as shown below, and Audacity will record them properly:
Monday, 15 November 2010
New Al Zimmermann's Programming Contest
Another one of the famous Al Zimmermann programming contests has just started. The contest runs until 12 Feb 2011, so there is plenty of time to compete. These contests are very competitive and the level is quite high, so it is difficult to stay in the top. Given that, and that I can only devote a limited amount of time to this, I will just try to compete "a little". For me, the goal will be to end within the first 10% of participants. Right now there are 125 contestants, so I should try to get in rank 12 or better. I quickly wrote a basic program and got in rank 33. Now it is time to improve it and see if I can do better than that over time. I will chart my progress below...
Update (14/2/2011): Oh well, the contest ended two days ago, and in the end I didn't get very far at all. My final position was 110, which with 362 contestants sets me at rank around 30.39%. I lost steam in the final weeks of the contest, when I realized that I had run out of easy ideas, and real work was not letting me try harder ones... Let's see if I can do better in the next contest (which is already set to start on Jul 9th).
Saturday, 13 November 2010
Editing videos from a Pentax Optio M40 camera
In a previous post I have a link to a Vimeo video, which was created by:
Today I decided to find a solution to this, so that I could edit the clip with Kdenlive, which is right now my preferred video editing software. The solution seems to use an intermediate software to create the clips to be imported to Kdenlive: Avidemux.
I just load the AVI file from the camera in Avidemux, select the starting and end points of the clip I want to use, and press the "Save" button. A screenshot can be seen here:
Then this clip can be imported into Kdenlive, without any synchronization problems. With Kdenlive we can now add a title, add effects, etc. and if we want a decent quality we can render it as the template MPEG-4 10000k 2-pass (which translates to:
f=mp4 acodec=libmp3lame ab=128k ar=44100 vcodec=mpeg4 minrate=0 b=10000k aspect=%dar mbd=2 trellis=1 mv4=1 pass=2).
The resulting file has perfect synchronization and can be uploaded to Vimeo without any problems and with much better quality than the one generated in my previous post. Here is the resulting video.
- recording a video with my Pentax Optio M40 camera
- cutting the beginning and the end of the video with the PiTiVi software
Today I decided to find a solution to this, so that I could edit the clip with Kdenlive, which is right now my preferred video editing software. The solution seems to use an intermediate software to create the clips to be imported to Kdenlive: Avidemux.
I just load the AVI file from the camera in Avidemux, select the starting and end points of the clip I want to use, and press the "Save" button. A screenshot can be seen here:
Then this clip can be imported into Kdenlive, without any synchronization problems. With Kdenlive we can now add a title, add effects, etc. and if we want a decent quality we can render it as the template MPEG-4 10000k 2-pass (which translates to:
f=mp4 acodec=libmp3lame ab=128k ar=44100 vcodec=mpeg4 minrate=0 b=10000k aspect=%dar mbd=2 trellis=1 mv4=1 pass=2).
The resulting file has perfect synchronization and can be uploaded to Vimeo without any problems and with much better quality than the one generated in my previous post. Here is the resulting video.
Friday, 12 November 2010
File Version Control
I'm a fan of version control systems, but for some reason I end up not using them as regularly as I should, and then I tend to forget how to do the most common operations, so this time I'm logging it here.
I do all my editing from inside Emacs, so I will be using VC, which allows me to use a version control system from within Emacs. VC works with many different version control systems, but for the moment I will use it with a very simple one: RCS. Later on, perhaps I will need a more sophisticated one, since RCS does not work over a network, and it only works at the level of individual files, but for the moment this is all I need.
The integration Emacs-RCS is very tight, and the basics for checking-in, checking-out are just performed with the same command: C-x v v. The first time we do this command, Emacs will offer to create the RCS directory, if it doesn't exist, and it will register (and unlock) the file as version 1.1. To edit the file again, just type C-x v v again to lock the file. When done and when we register the file again, the *VC-log* buffer will open, so that we can add a log entry. We continue this way, so that different versions of the file will be registered.
So now, some of the most common things that I need to do:
* Be able to see what has been changed in the file:
- When editing the file: C-x v =
- When writing the log entry: C-c C-d in the *VC-log* window (after saving the file being edited)
* Compare different versions:
Perhaps the most flexible one is just to type C-x v ~ REVISION, which allows us to view any previouos revision of the file, and then we can compare any two versions with something like ediff-buffers
* See history of changes:
C-x v l gives us the revision control change history. In a different buffer, we will have a list of all the revisions made to the file, together with the text of the log entries. This buffer is also very convenient to do some useful operations:
- With p and n, we can go the the lines corresponding to the previous or the next revision respectively.
- With e we can modify the log entry text.
- With f we can visit the revision indicated at the current line (like C-x v ~ above).
- With d see the diff between the revision at the current line and the next earlier revision.
* Undo changes:
C-x v u revert the file to the revision from which you started editing.
Later on I will add to my daily version control fix how to work with branches, but for the moment this is 90% of what I really need from a CVS.
I do all my editing from inside Emacs, so I will be using VC, which allows me to use a version control system from within Emacs. VC works with many different version control systems, but for the moment I will use it with a very simple one: RCS. Later on, perhaps I will need a more sophisticated one, since RCS does not work over a network, and it only works at the level of individual files, but for the moment this is all I need.
The integration Emacs-RCS is very tight, and the basics for checking-in, checking-out are just performed with the same command: C-x v v. The first time we do this command, Emacs will offer to create the RCS directory, if it doesn't exist, and it will register (and unlock) the file as version 1.1. To edit the file again, just type C-x v v again to lock the file. When done and when we register the file again, the *VC-log* buffer will open, so that we can add a log entry. We continue this way, so that different versions of the file will be registered.
So now, some of the most common things that I need to do:
* Be able to see what has been changed in the file:
- When editing the file: C-x v =
- When writing the log entry: C-c C-d in the *VC-log* window (after saving the file being edited)
* Compare different versions:
Perhaps the most flexible one is just to type C-x v ~ REVISION, which allows us to view any previouos revision of the file, and then we can compare any two versions with something like ediff-buffers
* See history of changes:
C-x v l gives us the revision control change history. In a different buffer, we will have a list of all the revisions made to the file, together with the text of the log entries. This buffer is also very convenient to do some useful operations:
- With p and n, we can go the the lines corresponding to the previous or the next revision respectively.
- With e we can modify the log entry text.
- With f we can visit the revision indicated at the current line (like C-x v ~ above).
- With d see the diff between the revision at the current line and the next earlier revision.
* Undo changes:
C-x v u revert the file to the revision from which you started editing.
Later on I will add to my daily version control fix how to work with branches, but for the moment this is 90% of what I really need from a CVS.
Tuesday, 9 November 2010
Classical guitar progress logging (Nov'10)
Today I'm starting my guitar progress logging, my plan being to record and upload every month a piece that I'm learning, whatever its status, to keep a record of my progress, my struggles, etc.
We start with the Étude Simple n.VI by Leo Brouwer, which can be seen at http://vimeo.com/16656488
My camera doesn't have an input for an external microphone, so I have to rely in its poor quality built-in microphone. (Later on I hope I will be able to learn how to easily and accurately sinchronize video recorded with the camera and audio recorded directly to the computer via microphone, but until then...)
Technical stuff: after editing it with PiTiVi, I render it with the options: VGA(640x480), Audio Output: CD; Container: FFmpeg AVI format muxer; Audio Codec: TwoLAME mp2 encoder; Video Codec: FFmpeg MPEG-4 part 2 encoder
Other codecs gave me problems and the video could not be converted by the Vimeo Servers.
We start with the Étude Simple n.VI by Leo Brouwer, which can be seen at http://vimeo.com/16656488
Étude Simple n.VI by Leo Brouwer from Angel de Vicente on Vimeo.
My camera doesn't have an input for an external microphone, so I have to rely in its poor quality built-in microphone. (Later on I hope I will be able to learn how to easily and accurately sinchronize video recorded with the camera and audio recorded directly to the computer via microphone, but until then...)
Technical stuff: after editing it with PiTiVi, I render it with the options: VGA(640x480), Audio Output: CD; Container: FFmpeg AVI format muxer; Audio Codec: TwoLAME mp2 encoder; Video Codec: FFmpeg MPEG-4 part 2 encoder
Other codecs gave me problems and the video could not be converted by the Vimeo Servers.
Monday, 8 November 2010
Free classical guitar lessons on the internet
Free classical guitar lessons on the internet
Lessons given by Jean-François Delcamp
(Professor at the Brest Conservatory of Music, Dance and Drama - France)
The lessons will begin in November 2010 and finish in June 2011.
These lessons correspond to the first three years at the conservatory
The conditions for registering are available on the forum
http://www.delcamp.us/
More details at: http://www.delcamp.us/viewtopic.php?f=41&t=54022#p587821
Lessons given by Jean-François Delcamp
(Professor at the Brest Conservatory of Music, Dance and Drama - France)
The lessons will begin in November 2010 and finish in June 2011.
These lessons correspond to the first three years at the conservatory
The conditions for registering are available on the forum
http://www.delcamp.us/
More details at: http://www.delcamp.us/viewtopic.php?f=41&t=54022#p587821
Thursday, 4 November 2010
The best video editor in Linux? OpenShot vs. Kdenlive? The jury is still out.
I'm starting to create videos more or less frequently, and the software that I tried so far, while good, was not working that well. My video editing needs are simple: put together a number of sequences, mix them with audio, apply some effects and transitions (fade in, fade out, etc.), and add some titles, credits, etc., and export it all to different formats: for Internet, DVD, etc.; and obviously, all within Linux.
When I start creating videos more regularly, perhaps I change my mind about the software I've used, but so far the fight is between OpenShot and Kdenlive. The video editing software that I have used:
Pitivi, was behaving properly, it looked stable and easy to use, but transition effects are not yet implemented, only fades.
Open Movie Editor, the interface does not look very polished, although it was quite OK for some of the videos I did, but the main problem for me was that I didn't find a way to zoom in or zoom out in the timeline. This was a problem when I had very short clips (perhaps just a few seconds), because it would be difficult to select them, in order to move them around or cut them, etc.
Kdenlive, this is probably the most complete editor of them all, and it has all those features that I want and many more, but in the past it crashed on me many times. Today I'm trying it again, with a new version (0.7.7.1), which apparently squashed many bugs, and it looks very promising. If I manage to run this version without crashes, this is probably the one that will win my heart. A screenshot of the video I'm editing right now:
OpenShot is the new kid in the block, and the interface is quite clean and it is feature rich. So far, my only gripe is that if I apply effects to consecutive video clips, sometimes artifacts are created in the audio, but I think this also happens with Kdenlive, so I need to investigate. The screenshot of the same project with Openshot:
There are a few more programs to try, (see http://en.wikipedia.org/wiki/Comparison_of_video_editing_software), but I think one of these two is going to be more than enough for my needs.
When I start creating videos more regularly, perhaps I change my mind about the software I've used, but so far the fight is between OpenShot and Kdenlive. The video editing software that I have used:
Pitivi, was behaving properly, it looked stable and easy to use, but transition effects are not yet implemented, only fades.
Open Movie Editor, the interface does not look very polished, although it was quite OK for some of the videos I did, but the main problem for me was that I didn't find a way to zoom in or zoom out in the timeline. This was a problem when I had very short clips (perhaps just a few seconds), because it would be difficult to select them, in order to move them around or cut them, etc.
Kdenlive, this is probably the most complete editor of them all, and it has all those features that I want and many more, but in the past it crashed on me many times. Today I'm trying it again, with a new version (0.7.7.1), which apparently squashed many bugs, and it looks very promising. If I manage to run this version without crashes, this is probably the one that will win my heart. A screenshot of the video I'm editing right now:
OpenShot is the new kid in the block, and the interface is quite clean and it is feature rich. So far, my only gripe is that if I apply effects to consecutive video clips, sometimes artifacts are created in the audio, but I think this also happens with Kdenlive, so I need to investigate. The screenshot of the same project with Openshot:
There are a few more programs to try, (see http://en.wikipedia.org/wiki/Comparison_of_video_editing_software), but I think one of these two is going to be more than enough for my needs.
Friday, 29 October 2010
Youtube videos to MP3 (or mpeg)
Today I wanted to download the music of a YouTube video, in order to show it to some friends off-line. It turns out that this is quite easy (for reference, this is using Ubuntu 10.04 LTS). (Information consulted: http://hubpages.com/hub/Youtube-to-MP3-on-Ubuntu-Linux and http://lookherefirst.wordpress.com/2007/12/15/how-to-convert-flv-files-to-mpg-or-mp3/).
The first step is to get the aptly named script youtube-dl. The version available in Ubuntu 10.04 is 2010.04.04, but this does not work properly anymore, and it gives the following error when trying to download a video:
If we download the latest version (2010.10.24) from http://bitbucket.org/rg3/youtube-dl/wiki/Home, and just copy the file to /usr/bin/youtube-dl, then we have no trouble and the video (in .flv format) is saved:
Then, in order to convert it to an MP3 you will need ffmpeg and the libmp3lame0 library (and in Ubuntu 10.04, perhaps also in others, the package libavcodec-unstripped-52, so that ffmpeg cand find libmp3lame). With this in place, a small script will take care of everything:
As an example:
Despite its name, we can also use the script as-is to convert Youtube videos for example to mpeg format:
The first step is to get the aptly named script youtube-dl. The version available in Ubuntu 10.04 is 2010.04.04, but this does not work properly anymore, and it gives the following error when trying to download a video:
angelv~$ youtube-dl http://www.youtube.com/watch?v=BTwhEPiv0U8
[youtube] Setting language
[youtube] BTwhEPiv0U8: Downloading video info webpage
[youtube] BTwhEPiv0U8: Extracting video information
ERROR: format not available for video
angelv~$
If we download the latest version (2010.10.24) from http://bitbucket.org/rg3/youtube-dl/wiki/Home, and just copy the file to /usr/bin/youtube-dl, then we have no trouble and the video (in .flv format) is saved:
angelv~$ youtube-dl http://www.youtube.com/watch?v=BTwhEPiv0U8
[youtube] Setting language
[youtube] BTwhEPiv0U8: Downloading video webpage
[youtube] BTwhEPiv0U8: Downloading video info webpage
[youtube] BTwhEPiv0U8: Extracting video information
[download] Destination: BTwhEPiv0U8.flv
[download] 100.0% of 7.73M at 59.74k/s ETA 00:00
angelv~$ ls -lt *flv
-rw-r--r-- 1 angelv dialout 8100631 2010-10-29 10:01 BTwhEPiv0U8.flv
angelv~$
Then, in order to convert it to an MP3 you will need ffmpeg and the libmp3lame0 library (and in Ubuntu 10.04, perhaps also in others, the package libavcodec-unstripped-52, so that ffmpeg cand find libmp3lame). With this in place, a small script will take care of everything:
#!/bin/bash
x=$RANDOM.flv
youtube-dl --output=$x "$1"
ffmpeg -i $x -ar 44100 -ab 160k -ac 2 "$2"
rm $x
As an example:
angelv~/temp$ yt2mp3 http://www.youtube.com/watch?v=BTwhEPiv0U8 chiclana.mp3
[youtube] Setting language
[youtube] BTwhEPiv0U8: Downloading video webpage
[youtube] BTwhEPiv0U8: Downloading video info webpage
[youtube] BTwhEPiv0U8: Extracting video information
[download] Destination: 17660.flv
[download] 100.0% of 7.73M at 59.97k/s ETA 00:00
FFmpeg version SVN-r0.5.1-4:0.5.1-1ubuntu1, Copyright (c) 2000-2009 Fabrice Bellard, et al.
configuration: --extra-version=4:0.5.1-1ubuntu1 --prefix=/usr --enable-avfilter --enable-avfilter-lavf --enable-vdpau --enable-bzlib --enable-libgsm --enable-libschroedinger --e\
nable-libspeex --enable-libtheora --enable-libvorbis --enable-pthreads --enable-zlib --disable-stripping --disable-vhook --enable-runtime-cpudetect --enable-gpl --enable-postproc \
--enable-swscale --enable-x11grab --enable-libdc1394 --enable-shared --disable-static
libavutil 49.15. 0 / 49.15. 0
libavcodec 52.20. 1 / 52.20. 1
libavformat 52.31. 0 / 52.31. 0
libavdevice 52. 1. 0 / 52. 1. 0
libavfilter 0. 4. 0 / 0. 4. 0
libswscale 0. 7. 1 / 0. 7. 1
libpostproc 51. 2. 0 / 51. 2. 0
built on Mar 4 2010 12:41:55, gcc: 4.4.3
Seems stream 0 codec frame rate differs from container frame rate: 49.99 (4999/100) -> 25.00 (25/1)
Input #0, flv, from '17660.flv':
Duration: 00:03:24.44, start: 0.000000, bitrate: 253 kb/s
Stream #0.0: Video: h264, yuv420p, 320x240 [PAR 1:1 DAR 4:3], 253 kb/s, 25 tbr, 1k tbn, 49.99 tbc
Stream #0.1: Audio: aac, 22050 Hz, stereo, s16
Output #0, mp3, to 'chiclana.mp3':
Stream #0.0: Audio: libmp3lame, 44100 Hz, stereo, s16, 160 kb/s
Stream mapping:
Stream #0.1 > #0.0
Press [q] to stop encoding
size= 4003kB time=204.93 bitrate= 160.0kbits/s
video:0kB audio:4003kB global headers:0kB muxing overhead 0.000781%
angelv~/temp$ ls -lt *mp3
-rw-r--r-- 1 angelv dialout 4098644 2010-10-29 10:34 chiclana.mp3
angelv~/temp$ Despite its name, we can also use the script as-is to convert Youtube videos for example to mpeg format:
angelv@vaso:~$ yt2mp3 http://www.youtube.com/watch?v=McdD9Ng4VnY funny.mpg
angelv@pilas:~$ youtube-dl --extract-audio --audio-format=mp3 http://www.youtube.com/watch?v=suTKOmmoBtw[youtube] Setting language[youtube] suTKOmmoBtw: Downloading video webpage[youtube] suTKOmmoBtw: Downloading video info webpage[youtube] suTKOmmoBtw: Extracting video information[download] Destination: suTKOmmoBtw.flv[download] 100.0% of 13.89M at 80.60k/s ETA 00:00[avconv] Destination: suTKOmmoBtw.mp3angelv@pilas:~$ Monday, 18 October 2010
From music sheet to MIDI (fourth round)
Time for SmartScore:
the Joropó piece, which comes from a PDF file generated with a music notation software (oh, well, if you look at the original score, you will see that this is far from perfect... not so Smart, after all!):
Given the previous performance, I couldn't expect too much for the old scanned document at http://www2.kb.dk/elib/noder/rischel/RiBS0848.pdf, but for completeness sake here it goes (although is almost useless):
And what about the scanned copy of one of my music books? Well, it gets confused with some of the bars, and it certainly would require some manual tweaking.
As per PhotoScore, MIDI files cannot be exported in the demo version, so I cannot test how well the music would be encoded.
So far, from the different softwares that I have tried, I think the ones that I would to try a bit better are SharpEye and PhotoScore
the Joropó piece, which comes from a PDF file generated with a music notation software (oh, well, if you look at the original score, you will see that this is far from perfect... not so Smart, after all!):
Given the previous performance, I couldn't expect too much for the old scanned document at http://www2.kb.dk/elib/noder/rischel/RiBS0848.pdf, but for completeness sake here it goes (although is almost useless):
And what about the scanned copy of one of my music books? Well, it gets confused with some of the bars, and it certainly would require some manual tweaking.
As per PhotoScore, MIDI files cannot be exported in the demo version, so I cannot test how well the music would be encoded.
So far, from the different softwares that I have tried, I think the ones that I would to try a bit better are SharpEye and PhotoScore
Friday, 15 October 2010
Private Repository with Fedora 13 (part 1)
At my work we are considering ways to efficiently allow users to install/remove software in their Fedora workstations, but without requiring the root password, and in a controlled way (which basically means allowing them to install/remove only some packages and not others).
So I set up to try the following:
So I start by downloading the 32bits Fedora 13 Desktop Edition, and installing it in a VirtualBox Virtual Machine. The Live System comes up OK inside VirtualBox, but for some reason, the "Install to Hard Drive" option does not seem to work... I'm not the first one to have this problem, and setting the "Base Memory" for the Virtual Machine to 1GB seems to solve the problem, and the installation to hard disk starts OK. Apparently after installation we don't need so much memory, so I change it back to 512MB and Fedora13 boots without any trouble:
So, we do the same for another virtual machine, the one that will act as the private repository. Next, we need to be able to boot both machines at the same time and configure networking so that they will see each other. After reading the VirtualBox instructions I decide configure each machine to have two adapters, one with NAT, and the other one as an internal network (which they will use to see each other):
And inside the guests, I configure eth1 manually with the addresses 192.168.0.1 and 192.168.0.2
Then we add two lines to the file /etc/hosts in both guests:
And both machines can see each other without trouble:
Since we haven't changed any configuration regarding the Fedora repositories, right now both machines will get all the software upgrades from the standard repositories, and only root can do them.
So now, we have to be able to create a local repository in the machine repo and use it as an additional repository in machine ws, but one from which a regular user can install software.
In order to create a local repository, we follow more or less these instructions:
But we cannot contact the web server at http://repo, since the firewall is preventing it. We open it by running the command system-config-firewall and selecting "http" as trusted. Then, from ws there is no problem to load the default Apache webpage.
Then we create the local directories:
In that directory we copy the following RPM xzgv-0.9.1-2.fc13.i686.rpm and then run the createrepo command:
Then we test in ws whether it is working. First we create the file SIE.repo:
(Modify to whatever metadata_expire suits you: see http://forums.fedoraforum.org/showthread.php?t=216398)
And we see that a simple install would work:
But when I really try to install something, then I see that we get problems with permissions. By default Apache simply does not list the .rpm files, and we cannot download them directly (if we rename a .rpm file to a file without extension, then there is no problem...). This has something to do with SELinux! Temporarily running "setenforce 0" solves the problem (see http://www.crypt.gen.nz/selinux/disable_selinux.html).
Installing something as root is then no problem. In order to be able to run it as a regular user, I create a script:
and add the following in the /etc/sudoers of ws (see sudoers documentation):
This is not ideal. The user can install but cannot remove the package. We should find a better way to make sure that a regular user can do all the usual tasks with yum, but only if they affect a package coming from our local repository (search packages, install them, remove them, be informed automatically of any updates...), but for the moment we leave it like this. Later on, we have to have a good read of PackageKit to see if it provides better control for this (it uses PolicyKit).
So now we have (an alpha version!) most of the staff in place, and we should just document how to create a RPM package out of our local software, and how to make sure that users get notified of updates to our software. This will have to go in another post...
So I set up to try the following:
- a Fedora13 workstation which downloads software from the standard repositories, to which regular users have no permission
- to create a private repository in another local server, in which we can have all the software local to our workplace, and from which users can install/remove software without requiring administration privileges
So I start by downloading the 32bits Fedora 13 Desktop Edition, and installing it in a VirtualBox Virtual Machine. The Live System comes up OK inside VirtualBox, but for some reason, the "Install to Hard Drive" option does not seem to work... I'm not the first one to have this problem, and setting the "Base Memory" for the Virtual Machine to 1GB seems to solve the problem, and the installation to hard disk starts OK. Apparently after installation we don't need so much memory, so I change it back to 512MB and Fedora13 boots without any trouble:
So, we do the same for another virtual machine, the one that will act as the private repository. Next, we need to be able to boot both machines at the same time and configure networking so that they will see each other. After reading the VirtualBox instructions I decide configure each machine to have two adapters, one with NAT, and the other one as an internal network (which they will use to see each other):
And inside the guests, I configure eth1 manually with the addresses 192.168.0.1 and 192.168.0.2
Then we add two lines to the file /etc/hosts in both guests:
[root@ws ~]# head -n 2 /etc/hosts
192.168.0.2 repo
192.168.0.1 ws
And both machines can see each other without trouble:
Since we haven't changed any configuration regarding the Fedora repositories, right now both machines will get all the software upgrades from the standard repositories, and only root can do them.
So now, we have to be able to create a local repository in the machine repo and use it as an additional repository in machine ws, but one from which a regular user can install software.
In order to create a local repository, we follow more or less these instructions:
yum install httpd
chkconfig --levels 235 httpd on
/etc/init.d/httpd start
But we cannot contact the web server at http://repo, since the firewall is preventing it. We open it by running the command system-config-firewall and selecting "http" as trusted. Then, from ws there is no problem to load the default Apache webpage.
Then we create the local directories:
mkdir /var/www/html/yum/SIE/fedora-13/i386
In that directory we copy the following RPM xzgv-0.9.1-2.fc13.i686.rpm and then run the createrepo command:
createrepo /var/www/html/yum/SIE/fedora-13/i386
Then we test in ws whether it is working. First we create the file SIE.repo:
(Modify to whatever metadata_expire suits you: see http://forums.fedoraforum.org/showthread.php?t=216398)
And we see that a simple install would work:
But when I really try to install something, then I see that we get problems with permissions. By default Apache simply does not list the .rpm files, and we cannot download them directly (if we rename a .rpm file to a file without extension, then there is no problem...). This has something to do with SELinux! Temporarily running "setenforce 0" solves the problem (see http://www.crypt.gen.nz/selinux/disable_selinux.html).
Installing something as root is then no problem. In order to be able to run it as a regular user, I create a script:
and add the following in the /etc/sudoers of ws (see sudoers documentation):
ALL ALL=(ALL) NOPASSWD: /usr/bin/SIE.install
This is not ideal. The user can install but cannot remove the package. We should find a better way to make sure that a regular user can do all the usual tasks with yum, but only if they affect a package coming from our local repository (search packages, install them, remove them, be informed automatically of any updates...), but for the moment we leave it like this. Later on, we have to have a good read of PackageKit to see if it provides better control for this (it uses PolicyKit).
So now we have (an alpha version!) most of the staff in place, and we should just document how to create a RPM package out of our local software, and how to make sure that users get notified of updates to our software. This will have to go in another post...
Wednesday, 13 October 2010
From music sheet to MIDI (third round)
Today is the time to try another piece of software: PhotoScore Ultimate 6
As with the others, we start with an easy one, the Joropó piece, which comes from a PDF file generated with a music notation software. And PhotoScore Ultimate 6 gets this:
A perfect performance. It is a pity that I could not save it to a MIDI file to try to load it with Rosegarden, since this was the Demo version, and that feature is not present.
The old scanned document at http://www2.kb.dk/elib/noder/rischel/RiBS0848.pdf does not get recognized very well with this program either:
And for my scanned copy of my book, PhotoScore does a quite decent job, although not better than SharpEye:
It is a pity that the demo version cannot export to MIDI, because it would be nice to be able to compare PhotoScore's generated MIDI to that generated by SharpEye.
As with the others, we start with an easy one, the Joropó piece, which comes from a PDF file generated with a music notation software. And PhotoScore Ultimate 6 gets this:
A perfect performance. It is a pity that I could not save it to a MIDI file to try to load it with Rosegarden, since this was the Demo version, and that feature is not present.
The old scanned document at http://www2.kb.dk/elib/noder/rischel/RiBS0848.pdf does not get recognized very well with this program either:
And for my scanned copy of my book, PhotoScore does a quite decent job, although not better than SharpEye:
It is a pity that the demo version cannot export to MIDI, because it would be nice to be able to compare PhotoScore's generated MIDI to that generated by SharpEye.
Sunday, 10 October 2010
Saturday, 9 October 2010
From music sheet to MIDI (second round)
Yesterday I tried PDFtoMusic Pro and Audiveris for getting MIDI files out of music sheets. Today I'm trying another software, which looks much more promising: SharpEye.
So, let's start with the easy one, the Joropó piece, which comes from a PDF file generated with a music notation software. SharpEye gets this:
Not a single mistake for the first seven bars, for all the guitars. Not bad at all. And the generated MIDI file can be loaded into Rosegarden without any trouble.
How does SharpEye cope with the old scanned document at http://www2.kb.dk/elib/noder/rischel/RiBS0848.pdf? Well, the side-by-side view of the original document and the music recognized shows that there is still plenty of room for improvement, although it does work much better than Audiveris.
But it is true that the quality of the original is a bit poor, so let's be a bit nicer with the poor silicon brain, and let's compare Audiveris and SharpEye with a nicely scanned piece (I scan a random piece in one of my books at 600 dpi, though put it back to 300 dpi when transforming it to JPG (for Audiveris) and TIFF (for SharpEye)), and the first 9 bars of the scanned document look like:
What Audiveris gets is this:
You can see that many of the notes are correct, but this would definitely require some time to tidy it up. Worse is the fact that the MIDI created out of this seems to be broken, and when trying to load it in Rosegarden it simply refuses to do it, showing the message: "Attempt to read past MIDI file end"
SharpEye does a much better job:
For the first nine bars, it looks like SharpEye has done an excellent job. Actually, for the whole piece (24 bars), I counted only three mistakes, which is very good, but we shouldn't get too carried away... When the created MIDI file is imported into Rosegarden, the first nine bars look like this:
A lot of mistakes now... More or less OK to recognize the piece, but given the quality inside SharpEye, one would like to get a better MIDI file...
The quest for the perfect sheet-to-MIDI tool is still on!
So, let's start with the easy one, the Joropó piece, which comes from a PDF file generated with a music notation software. SharpEye gets this:
Not a single mistake for the first seven bars, for all the guitars. Not bad at all. And the generated MIDI file can be loaded into Rosegarden without any trouble.
How does SharpEye cope with the old scanned document at http://www2.kb.dk/elib/noder/rischel/RiBS0848.pdf? Well, the side-by-side view of the original document and the music recognized shows that there is still plenty of room for improvement, although it does work much better than Audiveris.
But it is true that the quality of the original is a bit poor, so let's be a bit nicer with the poor silicon brain, and let's compare Audiveris and SharpEye with a nicely scanned piece (I scan a random piece in one of my books at 600 dpi, though put it back to 300 dpi when transforming it to JPG (for Audiveris) and TIFF (for SharpEye)), and the first 9 bars of the scanned document look like:
What Audiveris gets is this:
You can see that many of the notes are correct, but this would definitely require some time to tidy it up. Worse is the fact that the MIDI created out of this seems to be broken, and when trying to load it in Rosegarden it simply refuses to do it, showing the message: "Attempt to read past MIDI file end"
SharpEye does a much better job:
A lot of mistakes now... More or less OK to recognize the piece, but given the quality inside SharpEye, one would like to get a better MIDI file...
The quest for the perfect sheet-to-MIDI tool is still on!
Friday, 8 October 2010
Let's get serious about ear-training (seconds)
I seriously need to train my ear, and my ear-training companion is going to be GNU Solfege. The idea is to try (daily or very nearly) one of the many tests that are available with that software and move to the next one when I consistently (three times in a row) can get a higher than 90% accuracy in each one. So, let's start from the beginning, and plot here the performance data for seconds:
From music sheet to MIDI (originally from PDFs to MIDI)
There is a LOT of music out there... thousands and thousands of pieces, and not much time to practice all of them, so it would be very nice to be able to get a PDF file, convert it to MIDI and quickly have an idea of how it is supposed to be, and then perhaps use that MIDI file in order to practice. So I have been trying two different software tools to do it. The first one is PDFtoMusic Pro, which is not available for Linux and is not free, and the other one Audiveris, which is Open Source Software and will happily work in Linux.
In order to test each software, I start with the piece Joropó (for four guitars), which can be downloaded from http://www.forrestguitarensembles.co.uk/quartets.html
PDFtoMusic Pro
I don't have Windows, but I installed this in my Ubuntu 10.04 machine with Wine 1.1.42, and it runs without any (apparent) trouble.
I had no trouble at all to load the file, and to play directly inside it, without exporting it to MIDI, but actually I'm more interested in being able to export it to a MIDI file, so that later I can tweak it from inside Rosegarden or similar.
Exporting the file (or only selected pages) to MIDI is trivial, and when loaded in Rosegarden, there is no problem with playback. If we want to edit the file with the Notation Editor in Rosegarden, the result is pretty good, as can be seen from the first measures:
In order to get the same notation as the original score, we would have to tweak some of the sharps and turn them into flats, connect some notes with ligados, and change the clef of the fourth guitar, but those are quite small things.
If we make changes to the score and then want to print the music, then the result is also quite good from within Rosegarden. In this case, I just changed the clef of the fourth guitar, and printed it from within Rosegarden (which uses Lilypond as the printing engine), and the result was quite acceptable:
So, with PDFs that are generated with a music notation software, PDFtoMusic Pro works very well, but it cannot handle scanned music. For instance, when trying to load the file http://www2.kb.dk/elib/noder/rischel/RiBS0848.pdf it will come up with the message:
Audiveris
Audiveris is different from PDFtoMusic Pro (besides being Open Source), in which it is actually an OMR software and it will scan any document for music, even if it hasn't been generated by a music notation software.
I start Audiveris like:
Audiveris can only handle one page at at time, so we decide to go for the first page, which I save as a PDF (with evince), but the result when loaded is just not there:
So, I convert it to png first, but then Audiveris cannot even load it:
With JPG all looks much better:
And all is fine now:
So far we have only loaded the image. If we now want to perform the OMR step, we just go to the menu Step-Score, and we get the scanned music. If you look carefully below, there are some mistakes, but most of it is in the right place.
Now, in theory one can export this to MusicXML format and use something like Lilypond to create a printable score, but something is not working. If I export it to MusicXML (Score-Export), and then try to use musicxml2ly I get the following errors:
But I don't really care, as I'm interested in getting MIDI stuff (a basic PDF file can be created OK with Audiveris' own Score-PrintPDF option), though this will have the mistakes present in the scan, so it is not that useful...
But now let's go for the real thing: getting a MIDI file and playing it. Midi-Store MIDI will create a MIDI file, which then can be uploaded to a MIDI sequencer, in my case Rosegarden. Without any modification, the MIDI is more or less OK, though obviously the mistakes that were present in the scan are there, so a crucial part will be the ability to edit the score manually.
So, if we open the score editor in Rosegarden, we see that the notes are in there (the third guitar's first three bars don't show up when opening the editor in multitrack mode, but they are there if we open the editor just for the guitar's part), but they are all messed up (different clef for 3rd and 4th guitar, different key, different time signature,...).
How difficult is to change it back to a recognizable score?
Well, I did try to adjust it without looking at the Rosegarden documentation, but it was certainly not that trivial, so I decided to follow one of the Rosegarden tutorials to learn the basics, and then come back to this Joropó score, to try to adjust it properly. I followed the following Rosegarden tutorial, but even after reading that, it looks like doing all the modifications to get a score similar to the original one would be quite costly. A simpler task would be just to modify the MIDI file to "sound" like the original, but it is certainly going to take also some time to do it right.
So, if your music is in a PDF generated by a music notation software and you are happy to pay for it, then certainly PDFtoMusic Pro does a much better job. But how can Audiveris handle scanned music? As with PDFtoMusic Pro, I tried to scan the first page of http://www2.kb.dk/elib/noder/rischel/RiBS0848.pdf but I'm afraid that the result speaks for itself...:
So, I'm afraid that so far Audiveris is work in progress, but cannot really compete with PDFtoMusic Pro. I will continue my quest to see if I can find some way of making MIDI files out of scanned music sheets, but for the moment it looks like either they are PDF files produced with music notation software (in which case I can use PDFtoMusic Pro) or I will have to type them by hand...
In order to test each software, I start with the piece Joropó (for four guitars), which can be downloaded from http://www.forrestguitarensembles.co.uk/quartets.html
PDFtoMusic Pro
I don't have Windows, but I installed this in my Ubuntu 10.04 machine with Wine 1.1.42, and it runs without any (apparent) trouble.
I had no trouble at all to load the file, and to play directly inside it, without exporting it to MIDI, but actually I'm more interested in being able to export it to a MIDI file, so that later I can tweak it from inside Rosegarden or similar.
Exporting the file (or only selected pages) to MIDI is trivial, and when loaded in Rosegarden, there is no problem with playback. If we want to edit the file with the Notation Editor in Rosegarden, the result is pretty good, as can be seen from the first measures:
In order to get the same notation as the original score, we would have to tweak some of the sharps and turn them into flats, connect some notes with ligados, and change the clef of the fourth guitar, but those are quite small things.
If we make changes to the score and then want to print the music, then the result is also quite good from within Rosegarden. In this case, I just changed the clef of the fourth guitar, and printed it from within Rosegarden (which uses Lilypond as the printing engine), and the result was quite acceptable:
So, with PDFs that are generated with a music notation software, PDFtoMusic Pro works very well, but it cannot handle scanned music. For instance, when trying to load the file http://www2.kb.dk/elib/noder/rischel/RiBS0848.pdf it will come up with the message:
Audiveris
Audiveris is different from PDFtoMusic Pro (besides being Open Source), in which it is actually an OMR software and it will scan any document for music, even if it hasn't been generated by a music notation software.
I start Audiveris like:
angelv@vaso:~/audiveris$ java -Xmx256M -jar dist/audiveris-3.4.jar
Audiveris can only handle one page at at time, so we decide to go for the first page, which I save as a PDF (with evince), but the result when loaded is just not there:
So, I convert it to png first, but then Audiveris cannot even load it:
angelv@vaso:~/temp$ convert -density 300 first_page.pdf first_page.png
[...]
omr.sheet.picture.PictureLoader.loadFile(PictureLoader.java:101) -- INFO: Loading image from /home/angelv/temp/first_page.png ...
omr.sheet.picture.PictureLoader$Listener.imageComplete(PictureLoader.java:366) -- INFO: Image loading complete
omr.step.SheetSteps$LoadTask.doit(SheetSteps.java:489) -- WARNING: Exception [java.lang.RuntimeException: Unsupported pixel size:16]
omr.step.Step.doStepRange(Step.java:354) -- WARNING: Processing aborted [omr.step.StepException: java.lang.RuntimeException: Unsupported pixel size:16]
With JPG all looks much better:
convert -density 300 first_page.pdf first_page.jpg
And all is fine now:
So far we have only loaded the image. If we now want to perform the OMR step, we just go to the menu Step-Score, and we get the scanned music. If you look carefully below, there are some mistakes, but most of it is in the right place.
Now, in theory one can export this to MusicXML format and use something like Lilypond to create a printable score, but something is not working. If I export it to MusicXML (Score-Export), and then try to use musicxml2ly I get the following errors:
angelv@vaso:~/temp$ musicxml2ly first_page.xml
Reading MusicXML from first_page.xml ...
Traceback (most recent call last):
File "/usr/bin/musicxml2ly", line 2969, in
main()
File "/usr/bin/musicxml2ly", line 2964, in main
voices = convert (filename, options)
File "/usr/bin/musicxml2ly", line 2874, in convert
(voices, staff_info) = get_all_voices (parts)
File "/usr/bin/musicxml2ly", line 2583, in get_all_voices
all_voices = voices_in_part_in_parts (parts)
File "/usr/bin/musicxml2ly", line 2573, in voices_in_part_in_parts
voices = voices_in_part (p)
File "/usr/bin/musicxml2ly", line 2560, in voices_in_part
part.interpret ()
File "/usr/share/lilypond/2.12.3/python/musicxml.py", line 676, in interpret
int (attributes_dict.get ('divisions').get_text ()))
ValueError: invalid literal for int() with base 10: '4.0'
angelv@vaso:~/temp$
But I don't really care, as I'm interested in getting MIDI stuff (a basic PDF file can be created OK with Audiveris' own Score-PrintPDF option), though this will have the mistakes present in the scan, so it is not that useful...
But now let's go for the real thing: getting a MIDI file and playing it. Midi-Store MIDI will create a MIDI file, which then can be uploaded to a MIDI sequencer, in my case Rosegarden. Without any modification, the MIDI is more or less OK, though obviously the mistakes that were present in the scan are there, so a crucial part will be the ability to edit the score manually.
So, if we open the score editor in Rosegarden, we see that the notes are in there (the third guitar's first three bars don't show up when opening the editor in multitrack mode, but they are there if we open the editor just for the guitar's part), but they are all messed up (different clef for 3rd and 4th guitar, different key, different time signature,...).
How difficult is to change it back to a recognizable score?
Well, I did try to adjust it without looking at the Rosegarden documentation, but it was certainly not that trivial, so I decided to follow one of the Rosegarden tutorials to learn the basics, and then come back to this Joropó score, to try to adjust it properly. I followed the following Rosegarden tutorial, but even after reading that, it looks like doing all the modifications to get a score similar to the original one would be quite costly. A simpler task would be just to modify the MIDI file to "sound" like the original, but it is certainly going to take also some time to do it right.
So, if your music is in a PDF generated by a music notation software and you are happy to pay for it, then certainly PDFtoMusic Pro does a much better job. But how can Audiveris handle scanned music? As with PDFtoMusic Pro, I tried to scan the first page of http://www2.kb.dk/elib/noder/rischel/RiBS0848.pdf but I'm afraid that the result speaks for itself...:
So, I'm afraid that so far Audiveris is work in progress, but cannot really compete with PDFtoMusic Pro. I will continue my quest to see if I can find some way of making MIDI files out of scanned music sheets, but for the moment it looks like either they are PDF files produced with music notation software (in which case I can use PDFtoMusic Pro) or I will have to type them by hand...
Monday, 4 October 2010
First Steps with Music Software in Linux
There are not (so far) many musical things that I want to do in the computer, but playing a MIDI file; mixing it with my own recording of playing one of the parts, and exporting it all into an audio file is one of them, so today I set some time to learn the basics. In the computer I did this I'm running Ubutu 10.04, and I have followed the instructions in the Ubuntu Studio pages to quickly install all the software that I need (and much more), and to configure it.
With that in place, I rebooted the PC into the RealTime Kernel, and then I started by running JACK, Qsynth and Rosegarden. (Qsynth needs some soundfonts, and I downloaded some into the file /usr/share/sounds/sf2/FluidR3_GM.sf2. I don't remember where I got them from, but you can get them, for example, at http://musescore.org/handbook/soundfont). This looks like:
Then, it is time to check that all connectivity is working OK. The "Connect" button in the JACK interface will show the connections available. In my case:
This shows that the "General MIDI Device" in Rosegarden is connected to FLUID Synth (QSynth is just the GUI for fluidsynth), so all should be OK.
Then in Rosegarden we load a MIDI file, and select the instruments you want, modify the volume, reverb, etc. It took me a while (perhaps it should have been faster if I had read the documentation) to figure out how to select different instruments for each track, but it is actually quite easy. Just select one of the tracks:
Then say which instrument number you are going to use for that track:
My confusion for a while was to trust the name of what it would appear in the "Instrument" description, before I actually configured each instrument sound. So, the description in the "Instrument" field can be wrong. I guess it keeps whatever it was used last time with whatever synthesizer, so don't trust it, and instead, once you have selected the instrument number, you should specify the sound to be used for that instrument number:
With that in place (you will see that now the Instrument description in the track parameters section above reflects the change made here), you can now use Rosegarden to play the MIDI file with whatever instruments you choose.
So now we just have to record it into an audio file. This is done quite easily by just routing the output of Qsynth to Ardour. First, we will need to create at least one track with Ardour (here in the example: Audio1):
Then, in JACK we will have to connect the output from Qsynth to Ardour:
And lastly is just a matter of recording (first we have to select in which track we will be recording, in this case Audio 1,and then proceed with the "main" recording):
Adding other audio tracks (from an external mic, another software, etc.) would follow similar steps in Ardour. The last step is just to "export session to audiofile".
A basic MIDI-performance obtained in this way can be found at vimeo
With that in place, I rebooted the PC into the RealTime Kernel, and then I started by running JACK, Qsynth and Rosegarden. (Qsynth needs some soundfonts, and I downloaded some into the file /usr/share/sounds/sf2/FluidR3_GM.sf2. I don't remember where I got them from, but you can get them, for example, at http://musescore.org/handbook/soundfont). This looks like:
Then, it is time to check that all connectivity is working OK. The "Connect" button in the JACK interface will show the connections available. In my case:
This shows that the "General MIDI Device" in Rosegarden is connected to FLUID Synth (QSynth is just the GUI for fluidsynth), so all should be OK.
Then in Rosegarden we load a MIDI file, and select the instruments you want, modify the volume, reverb, etc. It took me a while (perhaps it should have been faster if I had read the documentation) to figure out how to select different instruments for each track, but it is actually quite easy. Just select one of the tracks:
Then say which instrument number you are going to use for that track:
My confusion for a while was to trust the name of what it would appear in the "Instrument" description, before I actually configured each instrument sound. So, the description in the "Instrument" field can be wrong. I guess it keeps whatever it was used last time with whatever synthesizer, so don't trust it, and instead, once you have selected the instrument number, you should specify the sound to be used for that instrument number:
With that in place (you will see that now the Instrument description in the track parameters section above reflects the change made here), you can now use Rosegarden to play the MIDI file with whatever instruments you choose.
So now we just have to record it into an audio file. This is done quite easily by just routing the output of Qsynth to Ardour. First, we will need to create at least one track with Ardour (here in the example: Audio1):
Then, in JACK we will have to connect the output from Qsynth to Ardour:
And lastly is just a matter of recording (first we have to select in which track we will be recording, in this case Audio 1,and then proceed with the "main" recording):
Adding other audio tracks (from an external mic, another software, etc.) would follow similar steps in Ardour. The last step is just to "export session to audiofile".
A basic MIDI-performance obtained in this way can be found at vimeo
Thursday, 30 September 2010
Back to student life...
Well, not really! From the student life I will only get the studying part... partying, dating, drinking, etc. are out...
This week we have started all the classes at the Conservatoire. I'm not sure I will be able to follow all of them, but I'm certainly going to try. In total I have 6.5 "contact" hours per week:
Guitar: 1
Ensemble: 1
Harmony: 2
Choral: 1.5
Chamber music: 1
Except for Chamber music, I had already one lesson of each of these.
In the guitar class, we have started directly with the Leo Brouwer simple studies n. VI and VII (they don't look particularly difficult, though they will need some time).
In the ensemble class, we started with two pieces for guitar quartet by Andrew Forrest (Joropó and Barcarowl), which can be downloaded at: http://www.forrestguitarensembles.co.uk/quartets.html
In harmony class, we started with the basic theory on triads, cadences, etc. I was hoping I could follow the Walter Piston book, and skip the lessons, but it looks like the teacher is going to follow a slightly different method, so I guess I will better attend as many lessons as possible.
In choral music class, we started by singing two pieces: "Adoramus Te" by W.A. Mozart and "Hymne a la Nuit" de J.P. Rameau.
Overall I don't think is going to be as difficult as I was expecting, though my intonation skills could certainly be improved, so I will have to devote some time to ear-training stuff. I would gladly do without the choral music class, but I guess at least it will be a way of forcing myself to not forget these ear-training exercises...
Let's see how I can cope with all this extra work in my already quite-busy life...
(Update: 4-Nov-2010)
It was a bit too much, so I decided to drop one of the courses: Chamber Music, which I will to do next year. My current plan is to do the remaining four years to the professional degree in six years (perhaps it is also a bit ambitious, since the higher the year, the higher the number of courses to take), but it is a bit more realistic. Let's see how it goes...
This week we have started all the classes at the Conservatoire. I'm not sure I will be able to follow all of them, but I'm certainly going to try. In total I have 6.5 "contact" hours per week:
Guitar: 1
Ensemble: 1
Harmony: 2
Choral: 1.5
Chamber music: 1
Except for Chamber music, I had already one lesson of each of these.
In the guitar class, we have started directly with the Leo Brouwer simple studies n. VI and VII (they don't look particularly difficult, though they will need some time).
In the ensemble class, we started with two pieces for guitar quartet by Andrew Forrest (Joropó and Barcarowl), which can be downloaded at: http://www.forrestguitarensembles.co.uk/quartets.html
In harmony class, we started with the basic theory on triads, cadences, etc. I was hoping I could follow the Walter Piston book, and skip the lessons, but it looks like the teacher is going to follow a slightly different method, so I guess I will better attend as many lessons as possible.
In choral music class, we started by singing two pieces: "Adoramus Te" by W.A. Mozart and "Hymne a la Nuit" de J.P. Rameau.
Overall I don't think is going to be as difficult as I was expecting, though my intonation skills could certainly be improved, so I will have to devote some time to ear-training stuff. I would gladly do without the choral music class, but I guess at least it will be a way of forcing myself to not forget these ear-training exercises...
Let's see how I can cope with all this extra work in my already quite-busy life...
(Update: 4-Nov-2010)
It was a bit too much, so I decided to drop one of the courses: Chamber Music, which I will to do next year. My current plan is to do the remaining four years to the professional degree in six years (perhaps it is also a bit ambitious, since the higher the year, the higher the number of courses to take), but it is a bit more realistic. Let's see how it goes...
Friday, 24 September 2010
Building A Virtual Cluster with Xen
Not new stuff, but today I was looking for some information on this, and I just realized that I didn't have a link to this article which I posted some years ago at the ClusterMonkey site. I'm not sure how many things would need updating, but the ideas still hold.
The article is in two parts:
Part_1
Part_2
The article is in two parts:
Part_1
Part_2
Friday, 30 July 2010
Sphere Online Judge (SPOJ) progress
SPOJ is one of the main sites that host programming challenges. The problems in SPOJ are for the most part much easier than the ones in Project Euler (which I have also posted about, although I have neglected it for several months...), so they are quite nice to try to solve in new programming languages. At this moment there are >10K users who have submitted at least one correct solution to one of the programs, so the place is very active. The graph below will show my progress in SPOJ, although I don't really intend to compete. I will only look for problems that look interesting to develop my programming skills.
Tuesday, 20 July 2010
Debugging Fortran+MPI codes with "The Portland Group" debugger (pgdbg)
I have to admit that I didn't get the Fox algorithm in my previous post correct the very first time, so I had to do some parallel debugging. In the past, I have used gdb, and the Intel Debugger, and now it was time to try pgdbg (The Portland Group debugger). Next in line is TotalView.
At our institution the Portland compiler's version installed is 10.5, but this had some issues with my current workstation linux distribution (Ubuntu 10.04 64 bits), so I installed the current latest version: 10.6. Installation was very simple with their install script, and once the license file was in place it was time to try it.
Compilation of the code can be done with the included MPICH1, with the following command:
Since we will be using ssh to connect to the other processors (actually just a number of processes all running in my local workstation), we need to first get the ssh security sorted out (tips from the "PGI Tools Guide" documentation, page. 90). We generate the ssh keys with a passphrase (and copy them to the authorized keys):
And then, from a new terminal we will just have to do the following, and enter the passphrase just once, and all subsequent ssh connections will be passwordless:
With this in place, we can run our code with the included MPICH1 version:
In order to run it with the debugger, we just add the option -dbg=pgdbg:
The following image shows a moment during the debugging session, where 4 processes have been created, and we are at the end of the first stage in the Fox algorithm. The window in the bottom shows how you can easily see the values of variables (whole matrixes included, which can be indexed according to Fortran syntax) for all (or a selection of) processes involved in the computation.
I need to try it for a longer period, but overall it looks like a very usable parallel debugger. The Portland Group has a video demo of the debugger here.
At our institution the Portland compiler's version installed is 10.5, but this had some issues with my current workstation linux distribution (Ubuntu 10.04 64 bits), so I installed the current latest version: 10.6. Installation was very simple with their install script, and once the license file was in place it was time to try it.
Compilation of the code can be done with the included MPICH1, with the following command:
angelv@vaso:~/fox$ pgf90 -o fox -Mmpi=mpich1 -g fox.f90
Since we will be using ssh to connect to the other processors (actually just a number of processes all running in my local workstation), we need to first get the ssh security sorted out (tips from the "PGI Tools Guide" documentation, page. 90). We generate the ssh keys with a passphrase (and copy them to the authorized keys):
$ ssh-keygen -t dsa
$ cd $HOME/.ssh
$ cp id_dsa.pub authorized_keys
And then, from a new terminal we will just have to do the following, and enter the passphrase just once, and all subsequent ssh connections will be passwordless:
$ eval `ssh-agent -s`
$ ssh-add
With this in place, we can run our code with the included MPICH1 version:
angelv@vaso:~/fox$ mpirun -stdin fox.in -np 4 ./fox
In order to run it with the debugger, we just add the option -dbg=pgdbg:
angelv@vaso:~/fox$ mpirun -stdin fox.in -dbg=pgdbg -np 4 ./fox
The following image shows a moment during the debugging session, where 4 processes have been created, and we are at the end of the first stage in the Fox algorithm. The window in the bottom shows how you can easily see the values of variables (whole matrixes included, which can be indexed according to Fortran syntax) for all (or a selection of) processes involved in the computation.
I need to try it for a longer period, but overall it looks like a very usable parallel debugger. The Portland Group has a video demo of the debugger here.
Monday, 19 July 2010
Fox algorithm for matrix multiplication in parallel with Fortran90+MPI
I'm now re-reading the book "Parallel Programming with MPI" by Peter S. Pacheco and doing some of the exercises in there. An interesting one is the Programming Assignment n.1 in page. 133, which involves fully implementing the Fox parallel algorithm for multiplying matrixes (see for example).
Below is Fortran90 code (version 1... some things need to be improved...) that does it (a better looking source code is at Pastebin, although I'm not sure for how long it will stay there...)
And given the following matrixes:
We can compile it and run it as:
Below is Fortran90 code (version 1... some things need to be improved...) that does it (a better looking source code is at Pastebin, although I'm not sure for how long it will stay there...)
PROGRAM FOX
IMPLICIT NONE
include "mpif.h"
INTEGER :: procs, rank, error
INTEGER, DIMENSION(MPI_STATUS_SIZE) :: status
INTEGER :: g_order, g_side, my_g_row, my_g_column, my_g_rank, comm, row_comm, col_comm, block_mpi_t
INTEGER :: rows
REAL, DIMENSION(:,:), ALLOCATABLE :: matrixA, matrixB, matrixC, localA, tempA, localB, localC
CALL MPI_Init ( error )
CALL MPI_Comm_size ( MPI_COMM_WORLD, procs, error )
CALL MPI_Comm_rank ( MPI_COMM_WORLD, rank, error )
CALL Read_Matrix(rank,error,rows,matrixA,matrixB,matrixC)
CALL Setup_grid(g_order,g_side,error,comm,my_g_rank,my_g_row,my_g_column,row_comm,col_comm)
CALL Distribute_matrixes(g_side,rows,block_mpi_t,error,matrixA,matrixB,localA,localB,localC,my_g_row,my_g_column)
CALL Perform_Fox_Algorithm(my_g_row,my_g_column,g_order,localA,tempA,localB,localC,row_comm,col_comm,status,error)
CALL Print_Last_Result(rank, my_g_rank, g_side, comm, procs, error, status, localC, matrixC, tempA)
CALL MPI_Finalize (error)
CONTAINS
SUBROUTINE Print_Last_Result(rank, my_g_rank, g_side, comm, procs, error, status, localC, matrixC, tempA)
INTEGER, INTENT(IN) :: rank, my_g_rank, g_side, comm, procs
INTEGER, INTENT(OUT) :: error
INTEGER, DIMENSION(:), INTENT(OUT) :: status
REAL, DIMENSION(:,:), INTENT(IN) :: localC
REAL, DIMENSION(:,:), INTENT(OUT) :: matrixC, tempA
INTEGER :: grow,gcol,i
INTEGER,DIMENSION(2) :: coordinates
IF (rank .EQ. 0 .AND. my_g_rank .NE. 0) PRINT*, "Houston, we have a problem with I/O"
CALL MPI_SEND(localC,g_side*g_side,MPI_REAL,0,0,comm,error)
IF (rank .EQ. 0) THEN
matrixC=0
DO i=1,procs
CALL MPI_Recv(tempA,g_side*g_side,MPI_REAL,MPI_ANY_SOURCE,0,comm,status,error)
CALL MPI_Cart_coords(comm,status(MPI_Source),2,coordinates,error)
grow = coordinates(1)
gcol = coordinates(2)
matrixC(grow*g_side+1:(grow+1)*g_side,gcol*g_side+1:(gcol+1)*g_side) = tempA
END DO
PRINT*,"============ Matrix multiplication in parallel ==================="
CALL Print_Matrix(matrixC)
END IF
END SUBROUTINE Print_Last_Result
SUBROUTINE Print_Matrix(M)
REAL, DIMENSION(:,:) :: M
INTEGER :: side,row,column
side = SIZE(M,1)
DO row=1,side
DO column=1,side-1
WRITE(*,'(F10.3)',ADVANCE='NO'), M(row,column)
END DO
WRITE(*,'(F10.3)'), M(row,side)
END DO
END SUBROUTINE Print_Matrix
SUBROUTINE Perform_Fox_Algorithm(my_g_row,my_g_column,g_order,localA,tempA,localB,localC,row_comm,col_comm,status,error)
INTEGER, INTENT(IN) :: my_g_row,my_g_column,g_order,row_comm,col_comm
INTEGER, INTENT(OUT) :: error
INTEGER, DIMENSION(:), INTENT(OUT) :: status
REAL, DIMENSION(:,:), INTENT(IN) :: localA
REAL, DIMENSION(:,:), INTENT(INOUT) :: tempA,localB,localC
INTEGER :: source,destination,stage,bcast_root
localC = 0
source = MOD(my_g_row + 1,g_order)
destination = MOD(my_g_row + g_order - 1,g_order)
DO stage=0,g_order-1
bcast_root = MOD(my_g_row + stage,g_order)
IF (my_g_column .EQ. bcast_root) THEN
CALL MPI_BCAST(localA,g_side*g_side,MPI_REAL,bcast_root,row_comm,error)
CALL Matrix_Multiply(localA,localB,localC)
ELSE
CALL MPI_BCAST(tempA,g_side*g_side,MPI_REAL,bcast_root,row_comm,error)
CALL Matrix_Multiply(tempA,localB,localC)
END IF
CALL MPI_Sendrecv_replace(localB,g_side*g_side,MPI_REAL,destination,0,source,0,col_comm,status,error)
END DO
END SUBROUTINE Perform_Fox_Algorithm
SUBROUTINE Matrix_Multiply(A,B,C)
REAL, DIMENSION(:,:), INTENT(IN) :: A,B
REAL, DIMENSION(:,:), INTENT(OUT) :: C
INTEGER :: side,row,column
side = SIZE(A,1)
DO row=1,side
DO column=1,side
C(row,column) = C(row,column) + SUM(A(row,:)*B(:,column))
END DO
END DO
END SUBROUTINE Matrix_Multiply
SUBROUTINE Distribute_matrixes(g_side,rows,block_mpi_t,error,matrixA,matrixB,localA,localB,localC,my_g_row,my_g_column)
INTEGER, INTENT(IN) :: g_side,rows,my_g_row,my_g_column
INTEGER, INTENT(OUT) :: block_mpi_t,error
REAL, DIMENSION(:,:), ALLOCATABLE, INTENT(OUT) :: localA,localB,localC
REAL, DIMENSION(:,:), INTENT(INOUT) :: matrixA,matrixB
CALL MPI_TYPE_VECTOR(g_side,g_side,rows,MPI_REAL,block_mpi_t,error)
CALL MPI_TYPE_COMMIT(block_mpi_t, error)
CALL MPI_BCAST(matrixA,rows*rows,MPI_REAL,0,MPI_COMM_WORLD,error)
CALL MPI_BCAST(matrixB,rows*rows,MPI_REAL,0,MPI_COMM_WORLD,error)
ALLOCATE(localA(g_side,g_side),tempA(g_side,g_side),localB(g_side,g_side),localC(g_side,g_side))
localA = matrixA(my_g_row*g_side+1:(my_g_row+1)*g_side,my_g_column*g_side+1:(my_g_column+1)*g_side)
localB = matrixB(my_g_row*g_side+1:(my_g_row+1)*g_side,my_g_column*g_side+1:(my_g_column+1)*g_side)
END SUBROUTINE Distribute_matrixes
SUBROUTINE Setup_grid(g_order,g_side,error,comm,my_g_rank,my_g_row,my_g_column,row_comm,col_comm)
INTEGER, INTENT(OUT) :: g_order, g_side,error,comm,my_g_rank,my_g_row,my_g_column,row_comm,col_comm
INTEGER,DIMENSION(2) :: dimensions,coordinates
LOGICAL,DIMENSION(2) :: wrap_around,free_coords
g_order = SQRT(REAL(procs))
g_side = rows / g_order
dimensions = g_order
wrap_around = .TRUE.
CALL MPI_Cart_create(MPI_COMM_WORLD,2,dimensions,wrap_around,.TRUE.,comm,error)
CALL MPI_Comm_rank ( comm, my_g_rank, error )
CALL MPI_Cart_coords(comm,my_g_rank,2,coordinates,error)
my_g_row = coordinates(1)
my_g_column = coordinates(2)
free_coords(1) = .FALSE.
free_coords(2) = .TRUE.
CALL MPI_Cart_sub(comm,free_coords,row_comm,error)
free_coords(1) = .TRUE.
free_coords(2) = .FALSE.
CALL MPI_Cart_sub(comm,free_coords,col_comm,error)
END SUBROUTINE Setup_grid
SUBROUTINE Read_Matrix(rank,error,rows,matrixA,matrixB,matrixC)
INTEGER, INTENT(IN) :: rank
INTEGER, INTENT(OUT) :: rows,error
INTEGER :: i
REAL, DIMENSION(:,:), ALLOCATABLE, INTENT(OUT) :: matrixA,matrixB,matrixC
IF (rank .EQ. 0) THEN
READ*, rows
CALL MPI_BCAST(rows,1,MPI_REAL,0,MPI_COMM_WORLD,error)
ALLOCATE(matrixA(rows,rows),matrixB(rows,rows),matrixC(rows,rows))
DO i=1,rows
READ*, matrixA(i,:)
END DO
DO i=1,rows
READ*, matrixB(i,:)
END DO
!! Calculate the matrix multiplication for comparison
matrixC = 0
CALL Matrix_Multiply(matrixA,matrixB,matrixC)
PRINT*, "CALCULATED IN PROCESS 0"
PRINT*, "=======MATRIX A=========="
CALL Print_Matrix(matrixA)
PRINT*, "=======MATRIX B=========="
CALL Print_Matrix(matrixB)
PRINT*, "=======MATRIX C=========="
CALL Print_Matrix(matrixC)
ELSE
CALL MPI_BCAST(rows,1,MPI_REAL,0,MPI_COMM_WORLD,error)
ALLOCATE(matrixA(rows,rows),matrixB(rows,rows))
END IF
END SUBROUTINE Read_Matrix
END PROGRAM FOXAnd given the following matrixes:
angelv@vaso:~/fox$ cat fox.in
8
3.4 4.2 2.1 9.3 5.6 7.8 3.4 6.3
6.6 7.6 2.1 7.2 3.5 7.9 2.0 8.3
8.6 4.1 7.5 3.5 7.1 9.6 4.1 8.1
5.8 4.7 8.6 3.6 7.4 4.3 1.2 0.2
9.9 5.3 8.9 3.7 6.5 9.7 3.6 5.3
8.2 4.7 9.3 3.8 6.7 8.4 4.7 6.4
6.8 8.0 2.8 8.9 9.8 3.2 6.8 8.7
6.4 7.6 8.3 4.3 6.5 4.1 3.9 3.8
3.5 5.7 8.6 4.6 7.5 9.0 2.8 6.0
8.9 3.4 7.3 3.3 5.3 8.2 3.6 3.9
8.3 0.8 2.1 1.6 4.7 6.4 3.4 8.8
7.3 4.3 3.3 2.7 4.5 6.6 9.3 5.9
6.7 6.7 8.0 2.8 3.7 5.7 7.7 8.3
7.8 4.0 8.9 2.9 1.4 0.8 7.8 8.2
3.4 7.1 5.7 9.0 6.1 7.9 8.9 7.7
2.6 4.1 6.9 8.2 9.3 5.0 7.1 4.6
We can compile it and run it as:
angelv@vaso:~/fox$ pgf90 -Mmpi=mpich -o fox fox.f90angelv@vaso:~/fox$ mpirun -stdin fox.in -np 4 ./fox
CALCULATED IN PROCESS 0
=======MATRIX A==========
3.400 4.200 2.100 9.300 5.600 7.800 3.400 6.300
6.600 7.600 2.100 7.200 3.500 7.900 2.000 8.300
8.600 4.100 7.500 3.500 7.100 9.600 4.100 8.100
5.800 4.700 8.600 3.600 7.400 4.300 1.200 0.200
9.900 5.300 8.900 3.700 6.500 9.700 3.600 5.300
8.200 4.700 9.300 3.800 6.700 8.400 4.700 6.400
6.800 8.000 2.800 8.900 9.800 3.200 6.800 8.700
6.400 7.600 8.300 4.300 6.500 4.100 3.900 3.800
=======MATRIX B==========
3.500 5.700 8.600 4.600 7.500 9.000 2.800 6.000
8.900 3.400 7.300 3.300 5.300 8.200 3.600 3.900
8.300 0.800 2.100 1.600 4.700 6.400 3.400 8.800
7.300 4.300 3.300 2.700 4.500 6.600 9.300 5.900
6.700 6.700 8.000 2.800 3.700 5.700 7.700 8.300
7.800 4.000 8.900 2.900 1.400 0.800 7.800 8.200
3.400 7.100 5.700 9.000 6.100 7.900 8.900 7.700
2.600 4.100 6.900 8.200 9.300 5.000 7.100 4.600
=======MATRIX C==========
260.900 194.020 272.070 178.530 210.450 236.380 297.220 275.730
274.180 199.380 307.390 197.010 245.450 266.250 285.240 277.610
311.840 232.300 352.690 225.580 277.280 303.160 320.440 360.720
247.510 147.520 219.820 111.300 167.610 225.640 198.500 256.890
327.930 227.120 350.150 209.450 269.700 313.690 306.850 365.810
318.490 224.600 336.210 216.270 271.960 310.980 311.220 361.910
319.570 268.880 357.800 255.450 309.740 359.100 362.840 349.110
288.990 190.670 279.080 175.760 235.560 291.560 257.210 301.530
============ Matrix multiplication in parallel ===================
260.900 194.020 272.070 178.530 210.450 236.380 297.220 275.730
274.180 199.380 307.390 197.010 245.450 266.250 285.240 277.610
311.840 232.300 352.690 225.580 277.280 303.160 320.440 360.720
247.510 147.520 219.820 111.300 167.610 225.640 198.500 256.890
327.930 227.120 350.150 209.450 269.700 313.690 306.850 365.810
318.490 224.600 336.210 216.270 271.960 310.980 311.220 361.910
319.570 268.880 357.800 255.450 309.740 359.100 362.840 349.110
288.990 190.670 279.080 175.760 235.560 291.560 257.210 301.530
angelv@vaso:~/fox$
Subscribe to:
Posts (Atom)