Youtube videos to MP3 (or mpeg)

Today I wanted to download the music of a YouTube video, in order to show it to some friends off-line. It turns out that this is quite easy (for reference, this is using Ubuntu 10.04 LTS). (Information consulted: http://hubpages.com/hub/Youtube-to-MP3-on-Ubuntu-Linux and http://lookherefirst.wordpress.com/2007/12/15/how-to-convert-flv-files-to-mpg-or-mp3/).

The first step is to get the aptly named script youtube-dl. The version available in Ubuntu 10.04 is 2010.04.04, but this does not work properly anymore, and it gives the following error when trying to download a video:

angelv~$ youtube-dl http://www.youtube.com/watch?v=BTwhEPiv0U8
[youtube] Setting language
[youtube] BTwhEPiv0U8: Downloading video info webpage
[youtube] BTwhEPiv0U8: Extracting video information
ERROR: format not available for video
angelv~$


If we download the latest version (2010.10.24) from http://bitbucket.org/rg3/youtube-dl/wiki/Home, and just copy the file to /usr/bin/youtube-dl, then we have no trouble and the video (in .flv format) is saved:

angelv~$ youtube-dl http://www.youtube.com/watch?v=BTwhEPiv0U8
[youtube] Setting language
[youtube] BTwhEPiv0U8: Downloading video webpage
[youtube] BTwhEPiv0U8: Downloading video info webpage
[youtube] BTwhEPiv0U8: Extracting video information
[download] Destination: BTwhEPiv0U8.flv
[download] 100.0% of 7.73M at   59.74k/s ETA 00:00

angelv~$ ls -lt *flv
-rw-r--r-- 1 angelv dialout 8100631 2010-10-29 10:01 BTwhEPiv0U8.flv
angelv~$


Then, in order to convert it to an MP3 you will need ffmpeg and the libmp3lame0 library (and in Ubuntu 10.04, perhaps also in others, the package libavcodec-unstripped-52, so that ffmpeg cand find libmp3lame). With this in place, a small script will take care of everything:

#!/bin/bash
x=$RANDOM.flv
youtube-dl --output=$x "$1"
ffmpeg -i $x -ar 44100 -ab 160k -ac 2 "$2"
rm $x


As an example:

angelv~/temp$ yt2mp3 http://www.youtube.com/watch?v=BTwhEPiv0U8 chiclana.mp3
[youtube] Setting language
[youtube] BTwhEPiv0U8: Downloading video webpage
[youtube] BTwhEPiv0U8: Downloading video info webpage
[youtube] BTwhEPiv0U8: Extracting video information
[download] Destination: 17660.flv
[download] 100.0% of 7.73M at   59.97k/s ETA 00:00
FFmpeg version SVN-r0.5.1-4:0.5.1-1ubuntu1, Copyright (c) 2000-2009 Fabrice Bellard, et al.
  configuration: --extra-version=4:0.5.1-1ubuntu1 --prefix=/usr --enable-avfilter --enable-avfilter-lavf --enable-vdpau --enable-bzlib --enable-libgsm --enable-libschroedinger --e\
nable-libspeex --enable-libtheora --enable-libvorbis --enable-pthreads --enable-zlib --disable-stripping --disable-vhook --enable-runtime-cpudetect --enable-gpl --enable-postproc \
--enable-swscale --enable-x11grab --enable-libdc1394 --enable-shared --disable-static
  libavutil     49.15. 0 / 49.15. 0
  libavcodec    52.20. 1 / 52.20. 1
  libavformat   52.31. 0 / 52.31. 0
  libavdevice   52. 1. 0 / 52. 1. 0
  libavfilter    0. 4. 0 /  0. 4. 0
  libswscale     0. 7. 1 /  0. 7. 1
  libpostproc   51. 2. 0 / 51. 2. 0
  built on Mar  4 2010 12:41:55, gcc: 4.4.3

Seems stream 0 codec frame rate differs from container frame rate: 49.99 (4999/100) -> 25.00 (25/1)
Input #0, flv, from '17660.flv':
  Duration: 00:03:24.44, start: 0.000000, bitrate: 253 kb/s
    Stream #0.0: Video: h264, yuv420p, 320x240 [PAR 1:1 DAR 4:3], 253 kb/s, 25 tbr, 1k tbn, 49.99 tbc
    Stream #0.1: Audio: aac, 22050 Hz, stereo, s16
Output #0, mp3, to 'chiclana.mp3':
    Stream #0.0: Audio: libmp3lame, 44100 Hz, stereo, s16, 160 kb/s
Stream mapping:
  Stream #0.1 > #0.0
Press [q] to stop encoding
size=    4003kB time=204.93 bitrate= 160.0kbits/s
video:0kB audio:4003kB global headers:0kB muxing overhead 0.000781%
 
angelv~/temp$ ls -lt *mp3
-rw-r--r-- 1 angelv dialout 4098644 2010-10-29 10:34 chiclana.mp3
angelv~/temp$ 

Despite its name, we can also use the script as-is to convert Youtube videos for example to mpeg format:

angelv@vaso:~$ yt2mp3 http://www.youtube.com/watch?v=McdD9Ng4VnY funny.mpg

UPDATE (10-Apr-2013): With Ubuntu 12.10, youtube-dl works perfect right now to download YouTube videos, and it even has an option to extract automatically the audio, so what I was achieving with this post is now a no-brainer:


angelv@pilas:~$ youtube-dl --extract-audio --audio-format=mp3 http://www.youtube.com/watch?v=suTKOmmoBtw
[youtube] Setting language
[youtube] suTKOmmoBtw: Downloading video webpage
[youtube] suTKOmmoBtw: Downloading video info webpage
[youtube] suTKOmmoBtw: Extracting video information
[download] Destination: suTKOmmoBtw.flv
[download] 100.0% of 13.89M at   80.60k/s ETA 00:00
[avconv] Destination: suTKOmmoBtw.mp3
angelv@pilas:~$ 

From music sheet to MIDI (fourth round)

Time for SmartScore:

the Joropó piece, which comes from a PDF file generated with a music notation software (oh, well, if you look at the original score, you will see that this is far from perfect... not so Smart, after all!):

Given the previous performance, I couldn't expect too much for the old scanned document at http://www2.kb.dk/elib/noder/rischel/RiBS0848.pdf, but for completeness sake here it goes (although is almost useless):

And what about the scanned copy of one of my music books? Well, it gets confused with some of the bars, and it certainly would require some manual tweaking.

As per PhotoScore, MIDI files cannot be exported in the demo version, so I cannot test how well the music would be encoded.

So far, from the different softwares that I have tried, I think the ones that I would to try a bit better are SharpEye and PhotoScore

Private Repository with Fedora 13 (part 1)

At my work we are considering ways to efficiently allow users to install/remove software in their Fedora workstations, but without requiring the root password, and in a controlled way (which basically means allowing them to install/remove only some packages and not others).

So I set up to try the following:

• a Fedora13 workstation which downloads software from the standard repositories, to which regular users have no permission
• to create a private repository in another local server, in which we can have all the software local to our workplace, and from which users can install/remove software without requiring administration privileges

So I start by downloading the 32bits Fedora 13 Desktop Edition, and installing it in a VirtualBox Virtual Machine. The Live System comes up OK inside VirtualBox, but for some reason, the "Install to Hard Drive" option does not seem to work... I'm not the first one to have this problem, and setting the "Base Memory" for the Virtual Machine to 1GB seems to solve the problem, and the installation to hard disk starts OK. Apparently after installation we don't need so much memory, so I change it back to 512MB and Fedora13 boots without any trouble:

So, we do the same for another virtual machine, the one that will act as the private repository. Next, we need to be able to boot both machines at the same time and configure networking so that they will see each other. After reading the VirtualBox instructions I decide configure each machine to have two adapters, one with NAT, and the other one as an internal network (which they will use to see each other):

And inside the guests, I configure eth1 manually with the addresses 192.168.0.1 and 192.168.0.2

 Then we add two lines to the file /etc/hosts in both guests:


[root@ws ~]# head -n 2 /etc/hosts
192.168.0.2 repo
192.168.0.1 ws


And both machines can see each other without trouble:

Since we haven't changed any configuration regarding the Fedora repositories, right now both machines will get all the software upgrades from the standard repositories, and only root can do them.

So now, we have to be able to create a local repository in the machine repo and use it as an additional repository in machine ws, but one from which a regular user can install software.

In order to create a local repository, we follow more or less these instructions:

yum install httpd
chkconfig --levels 235 httpd on
/etc/init.d/httpd start


But we cannot contact the web server at http://repo, since the firewall is preventing it. We open it by running the command  system-config-firewall and selecting "http" as trusted. Then, from ws there is no problem to load the default Apache webpage.

Then we create the local directories:

mkdir /var/www/html/yum/SIE/fedora-13/i386


In that directory we copy the following RPM xzgv-0.9.1-2.fc13.i686.rpm and then run the createrepo command:

createrepo /var/www/html/yum/SIE/fedora-13/i386


Then we test in ws whether it is working. First we create the file SIE.repo:

(Modify to whatever metadata_expire suits you: see http://forums.fedoraforum.org/showthread.php?t=216398)

And we see that a simple install would work:

But when I really try to install something, then I see that we get problems with permissions. By default Apache simply does not list the .rpm files, and we cannot download them directly (if we rename a .rpm file to a file without extension, then there is no problem...). This has something to do with SELinux! Temporarily running "setenforce 0" solves the problem (see http://www.crypt.gen.nz/selinux/disable_selinux.html).

Installing something as root is then no problem. In order to be able to run it as a regular user, I create a script:

and add the following in the /etc/sudoers of ws (see sudoers documentation):

ALL ALL=(ALL) NOPASSWD: /usr/bin/SIE.install


This is not ideal. The user can install but cannot remove the package. We should find a better way to make sure that a regular user can do all the usual tasks with yum, but only if they affect a package coming from our local repository (search packages, install them, remove them, be informed automatically of any updates...), but for the moment we leave it like this. Later on, we have to have a good read of PackageKit to see if it provides better control for this (it uses PolicyKit).

So now we have (an alpha version!) most of the staff in place, and we should just document how to create a RPM package out of our local software, and how to make sure that users get notified of updates to our software. This will have to go in another post...

Ear-training: fourths and fifths

(A continuation of the Ear-training thread).

From music sheet to MIDI (third round)

Today is the time to try another piece of software: PhotoScore Ultimate 6

As with the others, we start with an easy one, the Joropó piece, which comes from a PDF file generated with a music notation software. And PhotoScore Ultimate 6 gets this:

A perfect performance. It is a pity that I could not save it to a MIDI file to try to load it with Rosegarden, since this was the Demo version, and that feature is not present.

The old scanned document at http://www2.kb.dk/elib/noder/rischel/RiBS0848.pdf does not get recognized very well with this program either:

And for my scanned copy of my book, PhotoScore does a quite decent job, although not better than SharpEye:

 It is a pity that the demo version cannot export to MIDI, because it would be nice to be able to compare PhotoScore's generated MIDI to that generated by SharpEye.

Ear-training: thirds

(A continuation of the Ear-training thread).

From music sheet to MIDI (second round)

Yesterday I tried PDFtoMusic Pro and Audiveris for getting MIDI files out of music sheets. Today I'm trying another software, which looks much more promising: SharpEye.

So, let's start with the easy one, the Joropó piece, which comes from a PDF file generated with a music notation software. SharpEye gets this:

Not a single mistake for the first seven bars, for all the guitars. Not bad at all. And the generated MIDI file can be loaded into Rosegarden without any trouble.

How does SharpEye cope with the old scanned document at http://www2.kb.dk/elib/noder/rischel/RiBS0848.pdf? Well, the side-by-side view of the original document and the music recognized shows that there is still plenty of room for improvement, although it does work much better than Audiveris.

But it is true that the quality of the original is a bit poor, so let's be a bit nicer with the poor silicon brain, and let's compare Audiveris and SharpEye with a nicely scanned piece (I scan a random piece in one of my books at 600 dpi, though put it back to 300 dpi when transforming it to JPG (for Audiveris) and TIFF (for SharpEye)), and the first 9 bars of the scanned document look like:

What Audiveris gets is this:

You can see that many of the notes are correct, but this would definitely require some time to tidy it up. Worse is the fact that the MIDI created out of this seems to be broken, and when trying to load it in Rosegarden it simply refuses to do it, showing the message: "Attempt to read past MIDI file end"

SharpEye does a much better job:

For the first nine bars, it looks like SharpEye has done an excellent job. Actually, for the whole piece (24 bars), I counted only three mistakes, which is very good, but we shouldn't get too carried away... When the created MIDI file is imported into Rosegarden, the first nine bars look like this:

A lot of mistakes now... More or less OK to recognize the piece, but given the quality inside SharpEye, one would like to get a better MIDI file...

The quest for the perfect sheet-to-MIDI tool is still on!

Let's get serious about ear-training (seconds)

I seriously need to train my ear, and my ear-training companion is going to be GNU Solfege. The idea is to try (daily or very nearly) one of the many tests that are available with that software and move to the next one when I consistently (three times in a row) can get a higher than 90% accuracy in each one. So, let's start from the beginning, and plot here the performance data for  seconds:

From music sheet to MIDI (originally from PDFs to MIDI)

There is a LOT of music out there... thousands and thousands of pieces, and not much time to practice all of them, so it would be very nice to be able to get a PDF file, convert it to MIDI and quickly have an idea of how it is supposed to be, and then perhaps use that MIDI file in order to practice. So I have been trying two different software tools to do it. The first one is PDFtoMusic Pro, which is not available for Linux and is not free, and the other one  Audiveris, which is Open Source Software and will happily work in Linux.

In order to test each software, I start with the piece Joropó (for four guitars), which can be downloaded from http://www.forrestguitarensembles.co.uk/quartets.html

PDFtoMusic Pro

I don't have Windows, but I installed this in my Ubuntu 10.04 machine with Wine 1.1.42, and it runs without any (apparent) trouble.

I had no trouble at all to load the file, and to play directly inside it, without exporting it to MIDI, but actually I'm more interested in being able to export it to a MIDI file, so that later I can tweak it from inside Rosegarden or similar.

Exporting the file (or only selected pages) to MIDI is trivial, and when loaded in Rosegarden, there is no problem with playback. If we want to edit the file with the Notation Editor in Rosegarden, the result is pretty good, as can be seen from the first measures:

In order to get the same notation as the original score, we would have to tweak some of the sharps and turn them into flats, connect some notes with ligados, and change the clef of the fourth guitar, but those are quite small things.

If we make changes to the score and then want to print the music, then the result is also quite good from within Rosegarden. In this case, I just changed the clef of the fourth guitar, and printed it from within Rosegarden (which uses Lilypond as the printing engine), and the result was quite acceptable:

So, with PDFs that are generated with a music notation software, PDFtoMusic Pro works very well, but it cannot handle scanned music. For instance, when trying to load the file http://www2.kb.dk/elib/noder/rischel/RiBS0848.pdf it will come up with the message:

Audiveris

Audiveris is different from PDFtoMusic Pro (besides being Open Source), in which it is actually an OMR software and it will scan any document for music, even if it hasn't been generated by a music notation software.

I start Audiveris like:

angelv@vaso:~/audiveris$ java -Xmx256M -jar dist/audiveris-3.4.jar


Audiveris can only handle one page at at time, so we decide to go for the first page, which I save as a PDF (with evince), but the result when loaded is just not there:

So, I convert it to png first, but then Audiveris cannot even load it:

angelv@vaso:~/temp$ convert -density 300 first_page.pdf first_page.png

[...]

omr.sheet.picture.PictureLoader.loadFile(PictureLoader.java:101) -- INFO: Loading image from /home/angelv/temp/first_page.png ...
 omr.sheet.picture.PictureLoader$Listener.imageComplete(PictureLoader.java:366) -- INFO: Image loading complete
 omr.step.SheetSteps$LoadTask.doit(SheetSteps.java:489) -- WARNING: Exception [java.lang.RuntimeException: Unsupported pixel size:16]
 omr.step.Step.doStepRange(Step.java:354) -- WARNING: Processing aborted [omr.step.StepException: java.lang.RuntimeException: Unsupported pixel size:16]


With JPG all looks much better:

convert -density 300 first_page.pdf first_page.jpg


And all is fine now:

So far we have only loaded the image. If we now want to perform the OMR step, we just go to the menu Step-Score, and we get the scanned music. If you look carefully below, there are some mistakes, but most of it is in the right place.

Now, in theory one can export this to MusicXML format and use something like Lilypond to create a printable score, but something is not working. If I export it to MusicXML (Score-Export), and then try to use musicxml2ly I get the following errors:


angelv@vaso:~/temp$ musicxml2ly first_page.xml
Reading MusicXML from first_page.xml ...
Traceback (most recent call last):
  File "/usr/bin/musicxml2ly", line 2969, in
    main()
  File "/usr/bin/musicxml2ly", line 2964, in main
    voices = convert (filename, options)
  File "/usr/bin/musicxml2ly", line 2874, in convert
    (voices, staff_info) = get_all_voices (parts)
  File "/usr/bin/musicxml2ly", line 2583, in get_all_voices
    all_voices = voices_in_part_in_parts (parts)
  File "/usr/bin/musicxml2ly", line 2573, in voices_in_part_in_parts
    voices = voices_in_part (p)
  File "/usr/bin/musicxml2ly", line 2560, in voices_in_part
    part.interpret ()
  File "/usr/share/lilypond/2.12.3/python/musicxml.py", line 676, in interpret
    int (attributes_dict.get ('divisions').get_text ()))
ValueError: invalid literal for int() with base 10: '4.0'
angelv@vaso:~/temp$



But I don't really care, as I'm interested in getting MIDI stuff (a basic PDF file can be created OK with Audiveris' own Score-PrintPDF option), though this will have the mistakes present in the scan, so it is not that useful...

But now let's go for the real thing: getting a MIDI file and playing it. Midi-Store MIDI will create a MIDI file, which then can be uploaded to a MIDI sequencer, in my case Rosegarden. Without any modification, the MIDI is more or less OK, though obviously the mistakes that were present in the scan are there, so a crucial part will be the ability to edit the score manually.

So, if we open the score editor in Rosegarden, we see that the notes are in there (the third guitar's first three bars don't show up when opening the editor in multitrack mode, but they are there if we open the editor just for the guitar's part), but they are all messed up (different clef for 3rd and 4th guitar, different key, different time signature,...).

How difficult is to change it back to a recognizable score?

Well, I did try to adjust it without looking at the Rosegarden documentation, but it was certainly not that trivial, so I decided to follow one of the Rosegarden tutorials to learn the basics, and then come back to this Joropó score, to try to adjust it properly. I followed the following Rosegarden tutorial, but even after reading that, it looks like doing all the modifications to get a score similar to the original one would be quite costly. A simpler task would be just to modify the MIDI file to "sound" like the original, but it is certainly going to take also some time to do it right.

So, if your music is in a PDF generated by a music notation software and you are happy to pay for it, then certainly PDFtoMusic Pro does a much better job. But how can Audiveris handle scanned music? As with PDFtoMusic Pro, I tried to scan the first page of http://www2.kb.dk/elib/noder/rischel/RiBS0848.pdf but I'm afraid that the result speaks for itself...:

So, I'm afraid that so far Audiveris is work in progress, but cannot really compete with PDFtoMusic Pro. I will continue my quest to see if I can find some way of making MIDI files out of scanned music sheets, but for the moment it looks like either they are PDF files produced with music notation software (in which case I can use PDFtoMusic Pro) or I will have to type them by hand...

First Steps with Music Software in Linux

There are not (so far) many musical things that I want to do in the computer, but playing a MIDI file; mixing it with my own recording of playing one of the parts, and exporting it all into an audio file is one of them, so today I set some time to learn the basics.  In the computer I did this I'm running Ubutu 10.04, and I have followed the instructions in the Ubuntu Studio pages to quickly install all the software that I need (and much more), and to configure it.

With that in place, I rebooted the PC into the RealTime Kernel, and then I started by running JACK, Qsynth and Rosegarden. (Qsynth needs some soundfonts, and I downloaded some into the file /usr/share/sounds/sf2/FluidR3_GM.sf2. I don't remember where I got them from, but you can get them, for example, at http://musescore.org/handbook/soundfont). This looks like:

Then, it is time to check that all connectivity is working OK. The "Connect" button in the JACK interface will show the connections available. In my case:

This shows that the "General MIDI Device" in Rosegarden is connected to FLUID Synth (QSynth is just the GUI for fluidsynth), so all should be OK.

Then in Rosegarden we load a MIDI file, and select the instruments you want, modify the volume, reverb, etc. It took me a while (perhaps it should have been faster if I had read the documentation) to figure out how to select different instruments for each track, but it is actually quite easy. Just select one of the tracks:

Then say which instrument number you are going to use for that track:

My confusion for a while was to trust the name of what it would appear in the "Instrument" description, before I actually configured each instrument sound. So, the description in the "Instrument" field can be wrong. I guess it keeps whatever it was used last time with whatever synthesizer, so don't trust it, and instead, once you have selected the instrument number, you should specify the sound to be used for that instrument number:

With that in place (you will see that now the Instrument description in the track parameters section above reflects the change made here), you can now use Rosegarden to play the MIDI file with whatever instruments you choose.

So now we just have to record it into an audio file. This is done quite easily by just routing the output of Qsynth to Ardour. First, we will need to create at least one track with Ardour (here in the example: Audio1):

 Then, in JACK we will have to connect the output from Qsynth to Ardour:

And lastly is just a matter of recording (first we have to select in which track we will be recording, in this case Audio 1,and then proceed with the "main" recording):

Adding other audio tracks (from an external mic, another software, etc.) would follow similar steps in Ardour. The last step is just to "export session to audiofile".

A  basic MIDI-performance obtained in this way can be found at vimeo