SHARCNET has many clusters and most clusters
have R installed. The following instructions are used for
setting up Rmpi on the cluster orca. The same instructions can
be used for other clusters by changing the cluster name orca
accordingly. Also a fake user name joe is used. Please change
joe to your own login name.
SHARCNET has its own instructions for installing Rmpi. You can
find it
here.
But it is not complete. Just use it as reference.
Tested clusters with OpenMPI 1.6.2 and R 3.0.1: kraken, orca,
saw, goblin, hound, redfin
Tested clusters with OpenMPI 1.4.2 and R 3.0.1: mako
Tested clusters with issues: brown (run errors), requin (no R)
- Setup R.
Check if the cluster has R by
ls /opt/sharcnet/r
If it shows
2.15.0 2.15.3 3.0.1
then the newest R 3.0.1 is installed.
Create a cluster-specic file in your home bin directory to
launch R. Create the bin directory if it doesn't exist.
mkdir ~/bin
cp /opt/sharcnet/r/3.0.1/bin/R ~/bin/R_orca
Edit the .bash_profile (only need to edit once) by
nano ~/.bash_profile
and add at the end of the file
PATH= $HOME/bin:$PATH; export PATH
export R_LIBS=/work/$USER/$CLUSTER/R_local
Note: You need to create a cluster specific
directory in your working space:
mkdir /work/joe/orca; mkdir /work/joe/orca/R_local
Logout and login to see if you can run R by
R_orca
- Install Rmpi
Download the latest Rmpi from http://www.stats.uwo.ca/faculty/yu/Rmpi/
. For example,
wget
http://www.stats.uwo.ca/faculty/yu/Rmpi/download/dev/Rmpi_0.6-4.tar.gz
Choose a proper OpenMPI distribution
module list
Here OpenMPI 1.6.2 is used. Edit R_orca by
nano ~/bin/R_orca
At the third line, add
PATH="/opt/sharcnet/openmpi/1.6.2/intel/bin:$PATH";
export PATH
LD_LIBRARY_PATH=/opt/sharcnet/openmpi/1.6.2/intel/lib/:/opt/sharcnet/torque/current/lib/
export LD_LIBRARY_PATH
Install Rmpi
R_orca CMD INSTALL Rmpi_0.6-4.tar.gz
--configure-args=--with-mpi=/opt/sharcnet/openmpi/1.6.2/intel
Before using Rmpi, we need to set R_PROFILE since parallel
jobs on SHARCNET are launched via a job scheduler. Edit
R_orca again
nano ~/bin/R_orca
Insert the following line after export LD_LIBRARY_PATH
R_PROFILE=/work/$USER/$CLUSTER/R_local/Rmpi/Rprofile; export
R_PROFILE
Note: If Rmpi needs to be reinstalled or updated, you
must disable the above line (add # in front of it). Similary
if you install other packages, this R_PROFILE must be
disabled. Once installation is done, enable R_PROFILE again.
Test that Rmpi is installed and configured properly
by running R_orca. Rmpi should be loaded automatically (with
one CPU)
mpi.comm.size(0)
- Submit an Rmpi job
To run Rmpi as a batch job, go to your local work directory
cd /work/joe/orca
Run
sqsub -q mpi -r <time> -n <# CPUs> -o out
R_orca CMD BATCH <input_R_file> <output>
For example, to run a file called test.R (see the
following) for one hour on 32 CPUs (one master and 31
slaves) with output to test.out, run
sqsub -q mpi -r 1h -n 32 -o out R_orca CMD BATCH
test.R test.out
Use sqjobs to check your queue and use sqkill to kill your
job.
################ test.R#############
#setup parallel random number generator
mpi.setup.rngstream()
#create your own function(s)
myfun=function(n) mean(rnorm(n))
#transfer your function(s) to all slaves
mpi.bcast.Robj2slave(myfun)
#run the parallel job
output <- mpi.parReplicate(1000,myfun(1000000))
output[1:10]#can save output to a file
#must close all slaves
mpi.close.Rslaves()
mpi.quit()
##################################
Appendix:
If the cluster you are working on doesn't have R, then you
need to install it on your own. Go to
cd /work/joe/orca
Download the newest R source file (.tar.gz) from CRAN to
/work/joe/orca
wget <URL to R-3.x.x.tar.gz>
Untar and compile R
tar -zxf R-3.x.x.tar.gz
cd R-3.x.x; ./configure; make
Then copy R program to your bin directory
cp bin/R ~/bin/R_orca