Instructions to install and run Rmpi on SHARCNET

SHARCNET has many clusters and most clusters have R installed. The following instructions are used for setting up Rmpi on the cluster orca. The same instructions can be used for other clusters by changing the cluster name orca accordingly. Also a fake user name joe is used. Please change joe to your own login name.

SHARCNET has its own instructions for installing Rmpi. You can find it here. But it is not complete. Just use it as reference.

Tested clusters with OpenMPI 1.6.2 and R 3.0.1: kraken, orca, saw, goblin, hound, redfin
Tested clusters with OpenMPI 1.4.2 and R 3.0.1: mako
Tested clusters with issues: brown (run errors), requin (no R)
  1. Setup R.
     Check if the cluster has R by
       ls /opt/sharcnet/r
    If it shows
       2.15.0  2.15.3  3.0.1
    then the newest R 3.0.1 is installed.

    Create a cluster-speci c file in your home bin directory to launch R. Create the bin directory if it doesn't exist.
      mkdir ~/bin
      cp /opt/sharcnet/r/3.0.1/bin/R ~/bin/R_orca

    Edit the .bash_profile (only need to edit once) by
      nano ~/.bash_profile
    and add at the end of the file
      PATH= $HOME/bin:$PATH;  export PATH
      export R_LIBS=/work/$USER/$CLUSTER/R_local

    Note:  You need to create a cluster specific directory in your working space:
      mkdir /work/joe/orca; mkdir /work/joe/orca/R_local

    Logout and login to see if you can run R by
      R_orca

  2. Install Rmpi
    Download the latest Rmpi from http://www.stats.uwo.ca/faculty/yu/Rmpi/ . For example,
      wget http://www.stats.uwo.ca/faculty/yu/Rmpi/download/dev/Rmpi_0.6-4.tar.gz

    Choose a proper OpenMPI distribution
      module list

    Here OpenMPI 1.6.2 is used. Edit R_orca by
      nano ~/bin/R_orca
    At the third line, add
      PATH="/opt/sharcnet/openmpi/1.6.2/intel/bin:$PATH"; export PATH
      LD_LIBRARY_PATH=/opt/sharcnet/openmpi/1.6.2/intel/lib/:/opt/sharcnet/torque/current/lib/
      export LD_LIBRARY_PATH

    Install Rmpi
      R_orca CMD INSTALL Rmpi_0.6-4.tar.gz --configure-args=--with-mpi=/opt/sharcnet/openmpi/1.6.2/intel

    Before using Rmpi, we need to set R_PROFILE since parallel jobs on SHARCNET are launched via a job scheduler. Edit R_orca again
      nano ~/bin/R_orca
    Insert the following line after export LD_LIBRARY_PATH
       R_PROFILE=/work/$USER/$CLUSTER/R_local/Rmpi/Rprofile; export R_PROFILE

    Note: If Rmpi needs to be reinstalled or updated, you must disable the above line (add # in front of it). Similary if you install other packages, this R_PROFILE must be disabled. Once installation is done, enable R_PROFILE again.

    Test that Rmpi is installed and con figured properly  by running R_orca. Rmpi should be loaded automatically (with one CPU)
      mpi.comm.size(0)

  3. Submit an Rmpi job
    To run Rmpi as a batch job, go to your local work directory
      cd /work/joe/orca
    Run
      sqsub -q mpi -r <time> -n <# CPUs> -o out R_orca CMD BATCH <input_R_file> <output>

    For example, to run a file called test.R (see the following)  for one hour on 32 CPUs (one master and 31 slaves) with output to test.out, run
      sqsub -q mpi -r 1h -n 32 -o out R_orca CMD BATCH test.R test.out

    Use sqjobs to check your queue and use sqkill to kill your job.

    ################ test.R#############
    #setup parallel random number generator
    mpi.setup.rngstream()
    #create your own function(s)
    myfun=function(n) mean(rnorm(n))
    #transfer your function(s) to all slaves
    mpi.bcast.Robj2slave(myfun)
    #run the parallel job
    output <- mpi.parReplicate(1000,myfun(1000000))
    output[1:10]#can save output to a file
    #must close all slaves
    mpi.close.Rslaves()
    mpi.quit()
    ##################################

Appendix:

If the cluster you are working on doesn't have R, then you need to install it on your own. Go to
  cd /work/joe/orca

Download the newest R source file (.tar.gz) from CRAN to /work/joe/orca
  wget <URL to R-3.x.x.tar.gz>

Untar and compile R
  tar -zxf R-3.x.x.tar.gz
  cd R-3.x.x;  ./configure; make

Then copy R program to your bin directory
  cp bin/R ~/bin/R_orca