Running a command/script in all machines in a Condor pool

Sadly, all the machines in our Condor pool do not have exactly the same configuration, so it is sometimes convenient to run a command/script  in all the machines (mostly to find misbehaving ones). Since there is apparently no built-in way of doing this with Condor, I just wrote some quick Perl scripts to do it (in this case to find which machines cannot run Octave.

cycle_all_condor.pl will take a constraint string and a basic Condor command file. For example, I will run it like:

[angelv]$ ./cycle_all_condor.pl 'Arch == "X86_64"' basic.file


in order to find all the 64 bits machines in our pool, and will use basic.file as the prefix to the resulting Condor sumbit file. The resulting condor submit file is just sent to standard output (which can be redirected to a file, and then submitted with Condor).

The scripts are very simple, and probably written in a very poor Perl, but they do the job. Here they go:

[angelv]$ cat cycle_all_condor.pl
#! /usr/bin/perl                                                                                                                                          

$constraints = @ARGV[0];
$basic_cmd_file = @ARGV[1];

system "cat $basic_cmd_file";

open (MACHINES, "condor_status -const '$constraints' -format \"%s\n\" Machine | sort -u |");

while (<MACHINES>) {
    chomp;
    print "Requirements = Machine == \"$_\" \n";
    print "Queue \n"
}

close (MACHINES);


[angelv]$ cat basic.file
Executable = octave.sh
Universe = vanilla
output = output.$(Process)
error = error.$(Process)
log = log.condor
notification = NEVER
getenv = True



[angelv]$ cat octave.sh
#!/bin/sh
uname -a
octave --help



[angelv]$ cat results.pl
#!/usr/bin/perl                                                                                                                                            
@files = <error*>;
foreach $file (@files) {
    if (-s $file) {
        ($name,$ext) = split(/\./,$file);
        print "$file \n";
        system "cat $file output.$ext \n";
    }
}

SAGE + Chromium in my display wall

I have continued the configuration of my display wall (see my previous post).

Until now I had managed to get OpenGL applications working OK with Chromium, but only Chromium for the type of display wall that I envisage we need is not sufficient:

• the main problem I found is that I cannot run two Chromium applications at the same time. Perhaps this is doable, but I didn't find how to do it.
• applications cannot be moved around the display wall (there is a resizeable option with Chromium, but when I used it I got all sort of artifacts in the running application).

I also managed to get SAGE working fine, and the image viewer and the video viewer work fine (and you can easily move windows, rotate them, resize them, etc.), but OpenGL applications would need either tweaking the source code or either run them under the GL library wrapper (which according to its developer it has to be considered beta and is not recommended for production walls).

So I decided to check whether SAGE and Chromium can work fine, and I found that there is no problem with it. Something that I would like to work out is that I can still only run one application with Chromium. Other than that, the stuff started with SAGE can be moved, resized and rotated very easily, which is nice. The Chromium applications have a fixed position and size when I start them, but you can easily decide where and how big to start the application (see the two sizes of the Google Earth application in the video).

Running 32bits applications in Chromium while running 64bits SAGE is no problem (I just need to run the gl_32.sh script as seen in the previous post before launching the 32 bits app, and once it has started run gl_64.sh to get the library links back to normal).

For the size and the position of the Chromium applications, one can easily create a script with these data as input, and which creates a configuration file for Chromium which is used to launch the given application. Moving or resizing the application once it has started seems to give trouble.

A demo of this new setting can be found here.

Just finished my entry exams for 3rd grade.

Yesterday I finished my entry exams for 3rd grade at out local conservatory...

Today they will tell me the marks... I'm obviously not sure, but I think that I will get a 6.75 (out of 10), which is not great but it is not bad either. For 3rd year there is only one place, but two candidates, so passing is not enough, I have to outdo the other guy...

The exam has 3 main parts:
A: music interpretation with the guitar (for these I had to play the three pieces that I mentioned in a previous post) (60% of the final mark)
B: with four subparts: rhythm, music theory, intonation and dictation. (25%, where each subpart is marked up to (respectively): 4,2,2,2)
C: playing a small piece at first sight (15%)

And you have to pass each of the 3 main parts in order to get a final mark. Preparing the interpretation part was fun, but the theory not so much, since I was assuming that it would be simpler, and only started looking at it seriously a week before the exam. This meant that I had lots to put in my head, but did not have the time to practice, analyze, or simply study it properly, so I put in my TO DO list: reading of Harmony for the Guitar (or something similar).

In a few hours the marks will be published, but I know that the two parts that need working a lot are dictation and intonation, so regardless of the exam results I will have to do exercises to improve these. So far, my best companion for ear training will be GNU Solfege.

Will report back...

A DAY LATER....

Good stuff: I got an overall mark of 7.2 (out of 10) and the other guy got a 6.7, so I assume I will be a student again (and hopefully there is another place for the other guy as well, he plays well and he looks passionate about the guitar, so I think he deserves it too).

Bad stuff: I think I got the worst note of all candidates (at all levels and with all instruments) in intonation. I need to practice this a lot.

Graham Scan in Haskell

The last exercise in chapter 3 of Real World Haskell is quite more difficult than the previous ones, so it deserves a post with code. I have not used any function not yet seen at that point in the book, so the code is not very pretty, and I have followed quite closely the algorithm for Graham Scan as explained in Introduction to Algorithms, which can also be seen here.

When I learn a bit more about Haskell syntax I will try to pretify it, but for the moment here it goes in all its uglyness:


data Direction = LeftTurn
    | RightTurn
    | Straight
      deriving (Eq, Show)

turn (x1,y1) (x2,y2) (x3,y3)
     | cross_product == 0 = Straight
     | cross_product > 0 = LeftTurn
     | cross_product < 0 = RightTurn
     where
       cross_product = (x2 - x1)*(y3 - y1) - (y2 - y1)*(x3 - x1)

miny [x] = x
miny ((x1,y1):(x2,y2):xs)
     | y1 < y2 = miny ((x1,y1):xs)
     | y2 < y1 = miny ((x2,y2):xs)
     | x1 < x2 = miny ((x1,y1):xs)
     | x2 < x1 = miny ((x2,y2):xs)
     | otherwise = error "repeated points"

removepoint _ [] = []
removepoint (x,y) ((x1,y1):xs)
    | x == x1 && y == y1 = xs
    | otherwise = (x1,y1):removepoint (x,y) xs

sort_merge_polar origin [] = []
sort_merge_polar origin [x] = [x]
sort_merge_polar (x1,y1) lista =
    merge (sort_merge_polar (x1,y1) (take midpoint lista)) (sort_merge_polar (x1,y1) (drop midpoint lista))
        where
          midpoint = div (length lista) 2
          merge [] [] = []
          merge [] lisb = lisb
          merge lisa [] = lisa
          merge ((x2,y2):xs) ((x3,y3):ys)
              | turn (x1,y1) (x2,y2) (x3,y3) == LeftTurn = (x2,y2) : merge xs ((x3,y3):ys)
              | turn (x1,y1) (x2,y2) (x3,y3) == RightTurn = (x3,y3) : merge ((x2,y2):xs) ys
              | turn (x1,y1) (x2,y2) (x3,y3) == Straight = further_point : merge xs ys
              where further_point
                        | abs(x2 - x1) > abs(x3 - x1) = (x2,y2)
                        | abs(x2 - x1) < abs(x3 - x1) = (x3,y3)
                        | abs(y2 - y1) > abs(y3 - y1) = (x2,y2)
                        | abs(y2 - y1) < abs(y3 - y1) = (x3,y3)
                        | otherwise = error "repeated points.."

graham_scan (pa:pb:pc:ps) = result
    where
      p0 = miny (pa:pb:pc:ps)
      sorted_list = sort_merge_polar p0 (removepoint p0 (pa:pb:pc:ps))
      p1 = head sorted_list
      p2 = head (tail sorted_list)
      result = iterate_graham [p2,p1,p0] (tail (tail sorted_list))
      iterate_graham convex_hull [] = reverse(convex_hull)
      iterate_graham (top:ntop:xs) (pi:ps)
          | turn ntop top pi == RightTurn = iterate_graham (ntop:xs) (pi:ps)
          | turn ntop top pi == LeftTurn = iterate_graham (pi:top:ntop:xs) ps
          | turn ntop top pi == Straight = error "didn't I get rid of straight points relative to p0??"

graham_scan _ = error "we need at least three points"


As a basic test, if we compute the Convex Hull of the following points, we get the correct result. Other online solutions seem to choke on inputs like this, probably due to an incorrect implementation of the Graham Scan algorithm.

*Main> graham_scan [(10,0), (10,1),(-10,1),(-10,0),(-7,0),(-10,2),(-10,3),(-4,1),(-2,2),(-12,1)]
[(-10,0),(10,0),(10,1),(-10,3),(-12,1)]
*Main>

Getting ready for the classical guitar 3rd grade entry exam

The 3rd grade classical guitar entry exam is taking place on Monday 11th. The most important part of the exam is to play three musical pieces (one of them from memory). Yesterday I decided to record myself and this is the result (this was as close a setting to the exam as possible, i.e. no sound "post-production", no second chances, etc.). There are some clear mistakes, and during the exam I will get a bit nervous and there will probably be some more. I have one week to make them sound a little bit better...

I plan to play them in the following order (from easier to more difficult):
Kleynjans, Arpége, Op.77
Carcassi, Study n.7, Op. 60
Bach, Bourree BWV 996

Wish me good luck! :-)

Tracking swimming performance.

I'm slowly getting back into shape, so I have decided to start logging my progress in the swimming pool. Today I did (medium effort) 1500m freestyle in 32:17 minutes, though the goal is to get it in 28:20 (2X the World Record). The following charts will show the progress...

IRC client in Emacs: Riece

I have never used IRC, but in certain areas it seems like an interesting resource (right now I'm learning Haskell, and the Haskell IRC community #haskell seems quite active), so I've decided to give it a go. After looking at a number of IRC clients I have decided to try Riece, which is an IRC client for Emacs. It certainly fits well with my working environment and I can leave the connection open and reconnect from a different computer via a terminal and ssh. I connect via the freenode servers, and this is how it looks like.

I connect to chat.freenode.net, and the most used commands are in this list.

Adding SAGE to my display wall

As a continuation to my work on a tiled display wall, here there are the steps taken to install SAGE: "a graphics streaming architecture for supporting collaborative scientific visualization environments with potentially hundreds of megapixels of contiguous display resolution". So far, the basics are installed correctly, some demo applications are working OK, plus imageviewer and VLC, though I'm having trouble with the VNCViewer... (I'll keep working on this).

We dowload the current source version sage3-03-24-09.tgz.

Before trying to compile SAGE, we need to make sure we get the prerequisites OK, which can be found at the SAGE Forum.

For SDL, we install via Synaptic Package Manager:libsdl1.2-dev; for readline, libreadline6-dev

And we will also need the packages libmagickwand-dev, portaudio19-dev, python-wxgtk2.8, libglew1.5-dev, python-numpy

QUANTA
======

We download the current version.


angelv@vaiven:~/Downloads$ tar -zxvf QUANTA-1.0.tar.gz
angelv@vaiven:~/Downloads$ cd QUANTA_1.0/
angelv@vaiven:~/Downloads/QUANTA_1.0$ mkdir QUANTA_build
angelv@vaiven:~/Downloads/QUANTA_1.0$ cd QUANTA_build/


(to get cmake and ccmake, I install the package cmake-curses-gui)


angelv@vaiven:~/Downloads/QUANTA_1.0/QUANTA_build$ ccmake ../


But the default version of g++ (4.4) will give errors like:
/home/angelv/Downloads/QUANTA_1.0/src/QUANTAdb_c.cxx:367: error: ‘memcpy’ was not declared in this scope

Since correcting this would need changing the source code, we go the easy route, and we are going to use a previous version of the compiler 4.2

So, in the cmake options I change:


CMAKE_CXX_COMPILER /usr/bin/g++-4.2
CMAKE_CXX_FLAGS -fPIC
QUANTA_BIN_TYPE 64

angelv@vaiven:~/Downloads/QUANTA_1.0/QUANTA_build$ make
angelv@vaiven:~/Downloads/QUANTA_1.0/QUANTA_build$ sudo make install


SAGE
====

angelv@vaiven:~/sage3.0$


Modify .bashrc and start a new shell


angelv@vaiven:~$ tail -n 4 .bashrc
export SAGE_DIRECTORY=$HOME/sage3.0
export PATH=${SAGE_DIRECTORY}/bin:${PATH}
export LD_LIBRARY_PATH=${SAGE_DIRECTORY}/lib:${LD_LIBRARY_PATH}


In src/Makefile:

QUANTA_DIR=/usr/local/include
QUANTA_CFLAGS=-I${QUANTA_DIR}/QUANTA -I${QUANTA_DIR}


(tips from http://www.vislab.uq.edu.au/research/optiportal/software/jaunty.html)
both in imgToDxt.cpp and imageviewer.cpp in the app/FileViewer/ImageViewer/ directory we change


#if MagickLibVersion >= 0x645
#define MagickGetImagePixels MagickGetAuthenticPixels
#endif


to

#if 0
#if MagickLibVersion >= 0x645
#define MagickGetImagePixels MagickGetAuthenticPixels
#endif
#endif


Now edit the sage3.0/config.mk file. Change the line:
ARCHITECTURE=$(shell uname -p)
to
ARCHITECTURE=$(shell uname -m)

Then, if running on amd64 architecture, change (at about line 97):
XLIBS=-L/usr/X11R6/lib64 -lGLU -lGL -lXmu -lXi -lXext -lX11
to
XLIBS=-L/usr/lib -lGLU -lGL -lXmu -lXi -lXext -lX11


angelv@vaiven:~/sage3.0$ make install (do not run make first!)


Now check the script named sage. The line in it which reads:
if [ `uname` == "Linux" ]
should be changed to:
if [ `uname` = "Linux" ]


Configuration of SAGE
---------------------

angelv@vaiven:~/sage3.0/bin$ head -n 1 /home/angelv/sage3.0/bin/fsManager.conf
fsManager vaiven 192.168.1.1 161.72.201.73


in fsManger.conf

tileConfiguration stdtile-2.angelv.conf



cat stdtile-2.angelv.conf:
TileDisplay
Dimensions 2 1
Mullions 0.625 0.625 0.625 0.625
Resolution 1280 2048
PPI 90
Machines 2

DisplayNode
Name dwall-11
IP 192.168.1.11
Monitors 1 (1,0)

DisplayNode
Name dwall-12
IP 192.168.1.12
Monitors 1 (0,0)



cat tileNodes.list
192.168.1.1
192.168.1.11
192.168.1.12


In the render nodes (so far dwall-11 and dwall-12) I do:

angelv@dwall-12:~$ sudo ln -s /home/angelv/sage3.0/lib/libquanta.so /usr/lib/


Test that it is working correctly:
(from the readme.txt file in sage distribution)

6. Test Whether SAGE is Working Properly
        In $SAGE_DIRECTORY/bin
        (1) Execute "fsManager". Tiled display becomes black if it runs correctly.
                Users can specify a configuration file name as command-line argument.
                Otherwise "fsManager.conf" is used.
        (2) Open another terminal and execute "render"
        (3) Open another terminal and execute "fsConsole"
        (4) press TAB key twice. fsConsole commands are listed.
        (5) ? or help command gives you short description for each command
        (6) If you type initial character of a command and press TAB key,
                the command is completed. Then, press TAB key again. You can see
                the description of the command.
        (7) move 0 1000 0 : move the app window

And I find no problems.

Test with the graphical interface
---------------------------------

angelv@vaiven:~/sage3.0/bin$ ./sage


after clicking start I get both monitors black and the interface to SAGE connection. I try to add a new server, and put the ports as default, host name "vaiven" and IP "192.168.1.1", but then when checking the info, it always tells me that the server is not running. Neverthelles, I put "connect as" angelv, then press connect. A message comes out saying: "Connection to sage server failed. Chat will be unavailable". Click OK and the SAGE UI comes out. I try to run "atlantis" but nothing happnes. By going to "Advanced Mode" in the SAGE Launcher window, and tab "Application Launcher" I see that it complains that cannot find atlantis. Sure enough, if I try to run atlantis like SAGE does it from the command line, then it does not find atlantis either:


angelv@vaiven:~/sage3.0/bin$ ssh 127.0.0.1 atlantis
bash: atlantis: command not found
angelv@vaiven:~/sage3.0/bin$


To solve it move the lines about SAGE binaries and libraries to the beginning of the .bashrc file (tip thanks to http://sayspy.blogspot.com/2006/10/anyone-know-how-to-get-sshbash-to-use.html)


angelv@vaiven:~$ head -n 10 .bashrc
# ~/.bashrc: executed by bash(1) for non-login shells.
# see /usr/share/doc/bash/examples/startup-files (in the package bash-doc)
# for examples

export SAGE_DIRECTORY=$HOME/sage3.0
export PATH=${SAGE_DIRECTORY}/bin:${PATH}
export LD_LIBRARY_PATH=${SAGE_DIRECTORY}/lib:${LD_LIBRARY_PATH}

# If not running interactively, don't do anything
[ -z "$PS1" ] && return
angelv@vaiven:~$


And now atlantis works without any trouble.

Applications that I want:
------------------------

* View images:

In the SAGE Launcher -> Advanced Settings, I open the Settings for the File Server, and type as Library Root /home/angelv/sage3.0/bin/file_library. Then, after starting SAGE, in the SAGE UI I go to Options-> File Library nad I Choose the "local" file library. Then, with Browse I select any image in my computer. Automatically is incorporated to the library, and I can select the images by (in the File Library - local user interface) opening the menu Files -> Image at the left. Clicking in Show will show the image in the display wall, and then I can resize it, move it, etc. from within the SAGE UI.

* Videos:

See http://www.vislab.uq.edu.au/research/optiputer/software/vlc/index.html


sudo apt-get install libhal-dev libmad0-dev libavcodec-dev libavformat-dev libswscale-dev libpostproc-dev libtwolame-dev liba52-dev libfribidi-dev libgcrypt-dev


Compile it with ./GO and copy it to the sage bin directory with
/bin/cp -r build/bin/vlc $SAGE_DIRECTORY/bin

Then, in the Advanced Mode -> File Server Settings -> File Types, type: vlc --vout sage
then, when selecting a video from the File Library, it displays without problems, with the sound going in the head node speakers. (I'm not sure if it is going also to the other nodes, since I don't have loudspeakers in them).

* Sharing the desktop

Install vnc4server xvnc4viewer

I run a VNC server


angelv@vaiven:~/sage3.0/bin$ vncserver -geometry 800x600


then connecting to it with the default vncviewer is no problem


angelv@vaiven:~/sage3.0/bin$ vncviewer vaiven:1


but if I try to run it with the provided VNC:


angelv@vaiven:~/sage3.0/bin$ ./VNCViewer 127.0.0.1 1 800 600 sagesage
inet_addr 127.0.0.1 = 16777343
VNC server supports protocol version 3.8 (viewer 3.3)
here
*** buffer overflow detected ***: ./VNCViewer terminated
======= Backtrace: =========




So, part is working, but I still need to iron out some problems. To be continued...