Working with a linux cluster
May 09, 2015
I frequently need to work on a linux cluster. It’s a lot of work to manage the environment on many machines, especially when you don’t have root access. Over time I gradually came up with a working solution.
The cluster I work on has
/home shared over NFS. If I put program and libraries in my
$HOME directory, they can be shared across multiple machine. So when I need a program or a library, I will download its source, manually compile and install it in my
$HOME directory (usually by adding
configure script). Add
~/.bashrc and the manually installed programs will override system programs.
Also, linuxbrew is very useful for my situation. It puts stuff
in $HOME/.linuxbrew, so after installing any package via linuxbrew on one machine, it’s available throughout the cluster.
The following is what I do to setup my work environment on a cluster:
# install rbenv, ruby-build, and manually install ruby $ git clone https://github.com/sstephenson/rbenv.git ~/.rbenv $ echo 'export PATH="$HOME/.rbenv/bin:$PATH"' >> ~/.bashrc $ echo 'eval "$(rbenv init -)"' >> ~/.bashrc $ source ~/.bashrc # refresh bash environment $ git clone https://github.com/sstephenson/ruby-build.git ~/.rbenv/plugins/ruby-build $ rbenv install 2.2.2 $ rbenv global 2.2.2
In my case,
rbenv install 2.2.2 will report build failure, which is due to missing OpenSSL libraries. Ruby is actually built successfully, but without OpenSSL extensions. I had to manually cd to the build directory and run
make install to install ruby by hand.
Without OpenSSL support,
gem cannot download packages. We need to install OpenSSL and rebuild ruby later.
# install linuxbrew $ ruby -e "$(curl -fsSL https://raw.githubusercontent.com/Homebrew/linuxbrew/go/install)" $ echo ‘export PATH="$HOME/.linuxbrew/bin:$PATH"’ >> ~/.bashrc $ source ~/.bashrc # refresh bash environment $ brew install rbenv # so rbenv could be updated by linuxbrew $ brew install openssl $ rbenv install -f 2.2.2 # reinstall ruby with openssl support $ gem install bundler
At the time of writing this post, I cannot install Lua via linuxbrew. So I installed Lua and LuaRocks manually.
# manually build and install Lua $ make -C src clean all SYSCFLAGS="-DLUA_USE_LINUX" SYSLIBS="-Wl,-E -ldl -lreadline -lncurses" $ make install
You can also set the flags in
Now, for LuaRocks to work, you must have
unzip utility. If it’s not installed,
you can do this:
# install unzip for luarocks $ brew tap homebrew/dupes $ brew install unzip
Installing LuaRocks is easy, just download it from here, and do the traditional
configure (don’t forget
Switching between alternative packages
One more thing is how to switch between different versions/implementations of a package. In my case, I need to switch between OpenMPI and MVAPICH2. My solution is to create a
$HOME/packages directory, and install packages into separate
$HOME/packages/<package> subdirectories. In my case, I have
And for each package, I created an activate script setting up its environment:
The activate script has content like this:
# content of $HOME/packages/activate-openmpi-1.8.4 export PATH=$HOME/packages/openmpi-1.8.4/bin:$PATH export LD_LIBRARY_PATH=$HOME/packages/openmpi-1.8.4/lib:$LD_LIBRARY_PATH
To select which package to use, I created
# content of $HOME/packages/bashrc.pkgs source $HOME/packages/activate-openmpi-1.8.4 # enabled # source $HOME/packages/activate-mvapich2-2.1rc2 # disabled
And add following lines to
~/.bashrc for setting up packages:
# add to ~/.bashrc if [ -f ~/packages/bashrc.pkgs ]; then source ~/packages/bashrc.pkgs fi
See this post for additional tips on managing dotfiles.