Settings for a Tiled Display Wall

As I wrote in a previous post I built a tiled display wall from "scratch". I haven't tried to streamline the process, so there is plenty of room for improvement, but I write here all the notes I took while building it.

My test setting involves 3 PCs and six monitors (two per PC). The "head" node has two network cards, one connected to the "outside" world and the other one connected to a Gigabit router. The "render" nodes only have one network card, connected to the router. The three PC's have NVidia cards.

I decide to give a try to the 9.10 Ubuntu distribution (64 bits), so the first thing is to install Karmic Koala in the head node (vaiven). After an installation without any issues, it is time to configure a few things. First, the network cards. eth1 is identified as the outside card, so I change the DHCP setting for it, and assign it a fixed IP address at our institution. To eth0 I also assign a fixed address (now for the internal network), with IP 192.168.1.1, netmask 255.255.255.0 (no gateway, DNS, etc.). In /etc/hosts I change the line

127.0.1.1 vaiven
to

192.168.1.1 vaiven


I do all the recommended updates (as per 18/5/2010), and install openssh-server, synergy, and emacs.

In the first render node I install Karmic Koala with automatic login. I decide that I'm going to call the render nodes as dwall-XY where X will be the row and Y the column. So, the first one is dwall-11, and the IP address (eth0) will be: 192.168.1.11, with gateway 192.168.1.1, DNS servers and search domains as per eth0 in vaiven.

After the initial installation I want to change some important things:

* In order to access the Internet from the render nodes (hints from http://www.cyberciti.biz/tips/linux-as-router-for-dsl-t1-line-etc.html), in vaiven I modify the file /etc/sysctl.conf so that it contains the line

net.ipv4.ip_forward = 1


And I also modify /etc/rc.local to change the firewall policies:

angelv@vaiven:~$ tail /etc/rc.local
## In order to enable or disable this script just change the execution# bits.
## By default this script does nothing.
iptables --table nat --append POSTROUTING --out-interface eth1 -j MASQUERADE
iptables --append FORWARD --in-interface eth0 -j ACCEPT

exit 0
angelv@vaiven:~$


I restart vaiven, and test from dwall-11 that I can access the Internet. Then I also install all recommended updates (with the Update Manager), and I install openssh-server, synergy and emacs.

* A poor man's KVM

I only have one keyboard and a mouse, so I don't want to get up and change the USB connections all the time, and I don't want to buy a KVM, so I make sure that synergy is installed in both vaiven and dwall-11. I create the following config file in my home directory in vaiven

angelv@vaiven:~$ cat synergy.conf
section: screens
vaiven:
dwall-11:
end
section: links
vaiven:
left = dwall-11
dwall-11:
right = vaiven
end
angelv@vaiven:~$


I make sure that /etc/hosts in both machines have the lines
angelv@dwall-11:~$ cat /etc/hosts
127.0.0.1 localhost
192.168.1.1 vaiven
192.168.1.11 dwall-11[...]


And then I run:

angelv@vaiven:~$ synergys --config synergy.conf
angelv@dwall-11:~$ synergyc vaiven


With this I can move the mouse and use the keyboard in both computers without changing the USB connections.

I restart both computers and verify that dwall-11 logins automatically, that once synergys and synergyc are started I can move the mouse to dwall-11 and back to vaiven, and that I can access the Internet without problems from dwall-11.

One problem that I found with synergy is that some keys in my keyboard don't register (like th "ñ"), but for most purposes it works perfect.

* Share home folders in the cluster.

I install in vaiven nfs-kernel-server and modify the following files:


angelv@vaiven:~$ tail -n 5 /etc/exports
# Example for NFSv4:
# /srv/nfs4 gss/krb5i(rw,sync,fsid=0,crossmnt,no_subtree_check)
# /srv/nfs4/homes gss/krb5i(rw,sync,no_subtree_check)
#
/home 192.168.1.0/255.255.255.0(rw,sync)

angelv@vaiven:~$ tail -n 10 /etc/hosts.deny

# You may wish to enable this to ensure any programs that don't
# validate looked up hostnames still leave understandable logs. In past
# versions of Debian this has been the default.
# ALL: PARANOID
portmap:ALL
lockd:ALL
mountd:ALL
rquotad:ALL
statd:ALL

angelv@vaiven:~$ tail -n 10 /etc/hosts.allow
# daemon name. Remember that you can only use the keyword "ALL" and IP
# addresses (NOT host or domain names) for the portmapper, as well as for
# rpc.mountd (the NFS mount daemon). See portmap(8) and rpc.mountd(8)
# for further information.
#
portmap: 192.168.1.0/255.255.255.0
lockd: 192.168.1.0/255.255.255.0
mountd: 192.168.1.0/255.255.255.0
rquotad: 192.168.1.0/255.255.255.0
statd: 192.168.1.0/255.255.255.0
angelv@vaiven:~$


In dwall-11 I install nfs-common and modify the following files:


angelv@dwall-11:~$ tail -n 5 /etc/fstab
UUID=c9035fea-00c9-4fac-9c41-56409248ffd6 / ext4 errors=remount-ro 0 1
# swap was on /dev/sda5 during installation
UUID=c00e80b8-af40-45dc-9516-3c0e80449848 none swap sw 0 0
/dev/scd0 /media/cdrom0 udf,iso9660 user,noauto,exec,utf8 0 0
192.168.1.1:/home /home nfs rw,hard,intr 0 0
angelv@dwall-11:~$


* Cluster shell environment
In order to be able to control all the computers from a single terminal I install Omnitty. For this I install in vaiven libncurses5-dev and then follow the instructions at http://omnitty.sourceforge.net/.

I also need to link a library file:

sudo ln -s /usr/local/lib/librote.so* /usr/lib/


In a terminal I change the Edit->Keyboard Shortcuts Help->Contents from F1 to Alt-Shift-F1 (since we will need F1 inside Omnitty) and then I create the file dwall

angelv@vaiven:~$ cat dwall
vaiven
dwall-11
angelv@vaiven:~$


With this, when I start Omnitty, by pressing F5 and then typing @dwall, a connection to all the machines in the cluster will be made.

Then, to avoid typing the password everytime I do:

angelv@vaiven:~$ ssh-keygen -t rsa
angelv@vaiven:~$ cat .ssh/id_rsa.pub >> .ssh/authorized_keys


* Removing Ubuntu panels from view.
When displaying in the complete wall, we don't want to see the Ubuntu panels, so in vaiven I remove the panel on the bottom of the screen, and I change the properties in the top one to Autohide. Since this information is stored in the home directory, when I login again in dwall-11, these settings are replicated in there.

* In dwall-11 and vaiven I download and configure the NVidia drivers (System->Administration->Hardware drivers)

* I don't want the monitors in the display wall to go to sleep or to lock, so in dwall-11 in the Screensaver preferences I uncheck "Activate screensaver ..." and "Lock screen when screensaver..." and inside the Screensaver Preferences in Power Management I put "Never" in "Put display to sleep when inactive".

Chromium

It is now time to start configuring Chromium http://chromium.sourceforge.net/, so I download it from http://sourceforge.net/projects/chromium/files/ (version 1.9). The instructions for installation are at: http://chromium.sourceforge.net/doc/chromium2.html

I create two directories in my home to hold de 32 and 64 bits versions. First we go for the 64 version. I enter in cr-1.9, and to compile it we will need first a few things:


angelv@vaiven:~$ sudo ln -s /usr/bin/make /usr/bin/gmake

angelv@vaiven:~/cr-1.9$ sudo apt-get install libglut3-dev libglut g++ libxmu-headers libxmu-dev libjpeg-dev zlib1g-dev libxi-dev


Then, we just run make.

To compile the 32 bits version, I go into cr-1.9-32 and modify the file options.mk so that it contains the line:


FORCE_32BIT_ABI=1


We will need some extra packages (libc6-dev-i386, g++-multilib, ia32-libs and lib32z1-dev) and then we can just run make.

To make Chromium work fine, we need to create some files (for this I follow my previous post my previous post).


angelv@vaiven:~$ cat enable_cr
#!/bin/bash
cd /home/angelv/cr-1.9/lib/Linux
ln -s libcrfaker.so libGL.so.1
ln -s libcrfaker.so libGL.so

cd /home/angelv/cr-1.9-32/lib/Linux
ln -s libcrfaker.so libGL.so.1
ln -s libcrfaker.so libGL.so

angelv@vaiven:~$ cat disable_cr
#!/bin/bash
cd /home/angelv/cr-1.9/lib/Linux
rm libGL.so.1
rm libGL.so

cd /home/angelv/cr-1.9-32/lib/Linux
rm libGL.so.1
rm libGL.so

angelv@vaiven:~$ tail -n 10 .bashrc
# this, if it's already enabled in /etc/bash.bashrc and /etc/profile
# sources /etc/bash.bashrc).
if [ -f /etc/bash_completion ] && ! shopt -oq posix; then
. /etc/bash_completion
fi


export PATH="/home/angelv/cr-1.9/bin/Linux:$PATH"
export LD_LIBRARY_PATH="/home/angelv/cr-1.9/lib/Linux:/home/angelv/cr-1.9-32/lib/Linux:$LD_LIBRARY_PATH"
export CRMOTHERSHIP="vaiven"

angelv@vaiven:~$ cat .crconfigs
* /home/angelv/first_angel.conf %p

angelv@vaiven:~$ cat first_angel.conf
import sys
sys.path.append('/home/angelv/cr-1.9/mothership/server')
from mothership import *

TILE_WIDTH = 1280
TILE_HEIGHT = 2048

appnode = CRApplicationNode('vaiven')
tilesortspu = SPU('tilesort')
appnode.AddSPU(tilesortspu)
appnode.Conf( 'show_cursor', 1 )
cr = CR()
cr.AddNode(appnode)

renderspu = SPU('render')
renderspu.Conf('fullscreen', 1)
#renderspu.Conf('window_geometry', [0, 0, 512, 512])
renderspu.Conf( 'show_cursor', 1 )

node = CRNetworkNode('dwall-11')
node.AddTile(0, 0, TILE_WIDTH, TILE_HEIGHT)
node.AddSPU(renderspu)
node.AutoStart( ["/usr/bin/ssh",'-x',"dwall-11", "/home/angelv/start-crserver.sh 0.0 /home/angelv/cr-1.9/bin/Linux/crserver vaiven 10000 7000"] )

tilesortspu.AddServer(node, protocol='tcpip', port=7000)

cr.AddNode(node)

demo = sys.argv[1]

appnode.SetApplication(demo)

cr.Go()
angelv@vaiven:~$

angelv@vaiven:~$ cat start-crserver.sh
#!/bin/bash
export PATH="/home/angelv/cr-1.9/bin/Linux:$PATH"
export LD_LIBRARY_PATH="/home/angelv/cr-1.9/lib/Linux:/home/angelv/cr-1.9-32/lib/Linux:$LD_LIBRARY_PATH"

export DISPLAY=:$1
$2 -mothership $3:$4 -port $5
angelv@vaiven:~$


and in dwall-11 we install the necessary drivers:

angelv@dwall-11:~$ sudo apt-get install libgl1-mesa-dev


First, for testing, we try Chromium in just one PC (following its instructions:


% cd mothership/configs
% python crdemo.conf atlantis &
% crserver &
% crappfaker


and all is fine (when trying to run this in Ubuntu 10.04 we got problems like angelv@vaiven:~/cr-1.9/mothership/configs$ CR Warning(vaiven:14640): Render SPU: Display :0.0 doesn't have the necessary visual: RGB, Doublebuffer, Z)


Now it is time to try it properly.


angelv@vaiven:~$ ./enable_cr
angelv@vaiven:~$ glxgears
CR Warning(vaiven:22224): the OpenGL faker was loaded without crappfaker!
Defaulting to an application id of -1!
This won't work if you're debugging a parallel application!
In this case, set the CR_APPLICATION_ID_NUMBER environment
variable to the right thing (see opengl_stub/load.c)
CR Warning(vaiven:22224): Using Chromium configuration for * from /home/angelv/.crconfigs
This is Chromium, Version 1.9
Start a crappfaker on vaiven
Autostart for node dwall-11: ['/usr/bin/ssh', '-x', 'dwall-11', '/home/angelv/start-crserver.sh 0.0 /home/angelv/cr-1.9/bin/Linux/crserver vaiven 10000 7000']
Mothership signalling spawning process 22224
CR Info(dwall-11:7033): Total output dimensions = (1280, 2048)
26 frames in 5.1 seconds
26 frames in 5.1 seconds
26 frames in 5.1 seconds
24 frames in 5.1 seconds
24 frames in 5.1 seconds
27 frames in 5.1 seconds


No problems, although it is very slow. To make it work properly, we need to go to Appearance -> Visual Effects, and select None (I do it for both vaiven and dwall-11). With this we are back in business and glxgears works perfectly.

I install in vaiven stellarium (sudo apt-get install stellarium) and works with Chromium without issues.

I try Google Earth (which I download from http://earth.google.com/). No trouble installing it by running the script (although it complains about the cache everytime I start it). To solve the complaining about the cache I have to do:

angelv@vaiven:~$ sudo chown -R angelv:angelv .config/Google/
angelv@vaiven:~$ grep home .config/Google/GoogleEarthPlus.conf
KMLPath=/home/angelv/.googleearth
CachePath=/home/angelv/.googleearth/Cache


Now, in order to run GoogleEarth with Chromium I have to first make a symbolic link and create a couple of files:


angelv@vaiven:~$ ls -lt /usr/lib/libGL.so.1_32
lrwxrwxrwx 1 root root 29 2010-05-20 13:26 /usr/lib/libGL.so.1_32 -> /usr/lib32/libGL.so.185.18.36
angelv@vaiven:~$

angelv@vaiven:~$ cat gl_32.sh
#!/bin/sh

sudo rm /usr/lib/libGL.so.1
sudo ln -s /usr/lib/libGL.so.1_32 /usr/lib/libGL.so.1

angelv@vaiven:~$ cat gl_64.sh
#!/bin/sh

sudo rm /usr/lib/libGL.so.1
sudo ln -s /usr/lib/libGL.so.1_64 /usr/lib/libGL.so.1
angelv@vaiven:~$


And then it runs without issues (except that the we have to turn off the Atmosphere effect).


angelv@vaiven:~$ sudo ./gl_32.sh
angelv@vaiven:~$ googleearth


Adding render nodes

The idea was to use SystemImager to replicate the render nodes, but I found some problems, so for the time being I do it manually and will try to later on figure out a better way of cloning the render nodes.


In dwall-11 I open Synaptic Package Manager and File-> Save Marking As -> (Save full state, not only changes) -> Save (for instance: dwall.packages)

I install a new node (dwall-12) with the Ubuntu 9.10 CD. And then I proceed as follows:

* Configure the network to be 192.168.1.12 (as per dwall-11)
* Copy /etc/hosts and make sure is the same in all the machines (with all the stuff until now).
* Copy to /etc/fstab the line to share the home
* Get the dwall.packages file. Open Synaptic Manager -> Read Markings (dwall.packages) -> Apply
* Wait and reboot dwall-12

After rebooting I need to change a few things in order to include fully the new node in the display wall:

* Modify the dwall file for Omnity
* Modify the synergy.conf file
* System -> Administration -> Hardware drivers (to activate the Nvidia driver) (and copy the xorg.conf file from vavien to dwall-12)

* Modify first_angel.conf

angelv@vaiven:~$ diff first_angel.conf first_angel.conf.ORIGINAL
21,35d20
< node.AddTile(1280, 0, TILE_WIDTH, TILE_HEIGHT)
< node.AddSPU(renderspu)
< node.AutoStart( ["/usr/bin/ssh",'-x',"dwall-11", "/home/angelv/start-crserver.sh 0.0 /home/angelv/cr-1.9/bin/Linux/crserver vaiven 10000 7000"] )
<
< tilesortspu.AddServer(node, protocol='tcpip', port=7000)
<
< cr.AddNode(node)
<
<
< renderspu = SPU('render')
< renderspu.Conf('fullscreen', 1)
< #renderspu.Conf('window_geometry', [0, 0, 512, 512])
< renderspu.Conf( 'show_cursor', 1 )
<
< node = CRNetworkNode('dwall-12')
38c23
< node.AutoStart( ["/usr/bin/ssh",'-x',"dwall-12", "/home/angelv/start-crserver.sh 0.0 /home/angelv/cr-1.9/bin/Linux/crserver vaiven 10000 7000"] )
---
> node.AutoStart( ["/usr/bin/ssh",'-x',"dwall-11", "/home/angelv/start-crserver.sh 0.0 /home/angelv/cr-1.9/bin/Linux/crserver vaiven 10000 7000"] )
angelv@vaiven:~$


And all works excellently: glxgears, googleearth and stellarium. Googleearth working with the Atmosphere effect does not work (as expected).

Software to see images, and to watch videos

* For video, we can try VLC (as suggested in here), with its built-in wall filter. We install VLC (installation of packages always through omnitty, so that we keep all systems synchronized) and create two scripts:


angelv@vaiven:~$ cat vlc-wall_2x2.sh
#!/bin/sh

# cleanup
killall -9 vlc
ssh dwall-11 killall -9 vlc
ssh dwall-12 killall -9 vlc


vlc --aspect-ratio 5:4 --sout '#standard{scale=1,access=udp,mux=ts,dst=224.0.55.55:1234}' --ttl 1 $@ &
cvlc --novideo --vout dummy udp://@224.0.55.55 &


ssh dwall-12 'DISPLAY=:0 cvlc --aspect-ratio 5:4 --noaudio --vout-filter wall --wall-cols 2 --wall-rows 2 --wall-active 0 --fullscreen --xvideo-xineramascreen 0 udp://@224.0.55.55' &

ssh dwall-12 'DISPLAY=:0 cvlc --aspect-ratio 5:4 --noaudio --vout-filter wall --wall-cols 2 --wall-rows 2 --wall-active 2 --fullscreen --xvideo-xineramascreen 1 udp://@224.0.55.55' &

ssh dwall-11 'DISPLAY=:0 cvlc --aspect-ratio 5:4 --noaudio --vout-filter wall --wall-cols 2 --wall-rows 2 --wall-active 1 --fullscreen --xvideo-xineramascreen 0 udp://@224.0.55.55' &

ssh dwall-11 'DISPLAY=:0 cvlc --aspect-ratio 5:4 --noaudio --vout-filter wall --wall-cols 2 --wall-rows 2 --wall-active 3 --fullscreen --xvideo-xineramascreen 1 udp://@224.0.55.55' &


angelv@vaiven:~$ cat vlc-wall-kill.sh
#!/bin/sh

# cleanup
killall -9 vlc
ssh dwall-11 killall -9 vlc
ssh dwall-12 killall -9 vlc

angelv@vaiven:~$


I also modify the file /etc/rc.local and add the line

route add -net 224.0.0.0 netmask 240.0.0.0 dev eth0


I reboot, I open VLC and do "Reset Preferences" and then to watch a video in the 4 monitors we just run the following (when trying to extend it to three computers it does not work OK. The image is shown, but the scales in each of the monitors is different. Apparently this is a known bug, so we should investigate more):


angelv@vaiven:~$ ./vlc-wall_2x2.sh


and when we are done

angelv@vaiven:~$ ./vlc-wall-kill.sh



* For an image viewer we are going to try the TerraServer Blaster

In order to compile it we install an old version of the g++ compiler (g++-4.2), which also installs libstdc++6-4.2-dev (otherwise, when compiling, it complains about not finding iostream.h), we also install the package libfltk-dev, and then change the compiler line in the Makefile to make use of g++-4.2, and then we just run make, which does its thing without problems:


angelv@vaiven:~/tsb-1.3$ file tsb
tsb: ELF 64-bit LSB executable, x86-64, version 1 (SYSV), dynamically linked (uses shared libs), for GNU/Linux 2.6.15, not stripped
angelv@vaiven:~/tsb-1.3$


After enabling Chromium (/home/angelv/enable_cr) TSB works perfectly. The program comes with a nice demo to verify that all is working, which we can call with:

angelv@vaiven:~/tsb-1.3$ ./tsb config-HiRISE.txt


With this in place, and after installing with the package manager Stellarium, Celestia and Paraview we have all we need for the moment working fine, as can be seen in this demo.