Change Logs for Rmpi

2014-05-20, version 0.6-5 (released to CRAN):

All /dontrun is replaced by /donttest in man pages, suggested by Prof. Brian Ripley.

To close any running slaves left unintentionally by users, the follow codes
     if (mpi.comm.size(1)>0)
        mpi.close.Rslaves()
are added to .onUnload.

Fix a bug in mpi.bcast.Robj2slave and mpi.bcast.data2slave

2014-05-17, version 0.6-4 (released to CRAN):

libmpi.so.1 is loaded first with dlopen, suggested by Don Armstrong.

Due to discontinuation of MPICH2 for Windows, support of Rmpi under Windows's MPICH2 is dropped. Thanks to Prof. Brian Ripley, Rmpi can be compiled under Microsoft MPI. However, mpi.spawn.Rslaves() is disabled since spawning is not implemented in Microsoft MPI.

MPI_Errhandler_set is replaced by MPI_Comm_set_errhandler due to its deprecation.

A MPI type INTELMPI has been added to configure.ac. This allows Rmpi to be compiled with Intel's MPI (variation of MPICH). This modification was supplied by "Boyan Bejanov" <bbejanov at bank banque canada ca>.

All parallel apply functions with the arguments x and fun are replaced with X and FUN. This is in line with R's apply functions.

It has been observed that (un)serialization doesn't release memory properly after use. So the garbage collection (gc()) is used when there is a (un)serialization as along as it is not  nonblocking call related.

A new function mpi.bcast.data2slave is added. It transfers data (a double vector or a matrix)  natively without (un)serilization. It should be used with a huge vector or matrix. It results less memory usage and faster transmission. Notice that data with missing values (NA) are allowed.

A MPI type CRAY has been added to configure.ac. This allows Rmpi to be compiled on a Cray cluster with Cray's own MPICH2. It should be compiled as

  R CMD INSTALL Rmpi_0.6-4.tar.gz --configure-args="  \\
--with-Rmpi-include=/path_to_cray_mpich2_include \\ --with-Rmpi-libpath=/path_to_cray_mpich2_lib \\ --with-Rmpi-type=CRAY" --no-test-load

Please include --no-test-load, otherwise the installation will fail. Use Cray's aprun to launch Rmpi with Rprofile, either interactively or through PBS.

2013-03-26, version 0.6-3 (released to CRAN):

On Mac OS X, there is an error related to loading libmpi.dylib  during MPI_Init, an issue found by "Jan de Leeuw" < deleeuw at stat ucla edu >.  Under open-mpi 1.6.4, loading libmpi.dylib is not required and hence is removed. The same issue was detected by CRAN, as reported by Prof. Brian Ripley.

2013-03-8, version 0.6-2 (released to CRAN):

The dependence of the package rlecuyer for parallel RNG is removed. Instead the package parallel is used. Accordingly, the function mpi.setup.rngstream is simplified with iseed as input.

Due to difficulty to assign global objects .mpi.applyLB and .mpi.parSim, the option apply.seq in mpi.(i)applyLB related functions is disabled. So is the option sim.seq in mpi.parSim.

Add an option ... to mpi.bcast.cmd. This allows an execution like myfun(x,y) on all slaves if mpi.bcast.cmd(myfun, x=x,y=y) is used, where x and y are objects on the master.

mpi.remote.exec has been modified so that mpi.remote.exec(x<-1:10) will save the object x on all slaves.

Add a MPI API mpi.comm.c2f which converts an Rmpi's communicator into an integer that can be used as the communicator in external FORTRAN code, suggested by "Chris Paciorek" <paciorek at stat.berkeley.edu> and "Benjamin Lipshitz" <lipshitz at berkely.edu>.

Add an option all=FALSE to mpi.bcat.Robj2slave. If all=TRUE, all objects created on master will be transmitted to all slaves. Create a new function mpi.bcast.Rfun2slave which transfers  all functions created by master to all slaves.

Core count for Mac OS X is rewritten so that mpi.universe.size will correctly report total number of cores on a single Mac machine. Due to R64 naming on Mac, a script MacR64slaves.sh is created to aid R64 to choose a right slave.

Under Open MPI, add a program check on orted in configure.ac since Rmpi cannot be loaded without orted , an issue found by "Dirk Eddelbuettel" <edd at debian.org>.

Modify mpi.scatter.Robj so that subobj <- lapply(obj,serialize, connection=NULL) is replaced by subobj<-lapply(1:size, function(i) serialize(obj[[i]], NULL)). This results substantial improvement of encode speed. The idea of this modification comes from pdbMPI package.

2012-08-19, version 0.6-1 (released to CRAN):

mpi.bcast.cmd has been rewrittten with two additional options nonblock and sleep. It uses for loop with mpi.send and mpi.recv pair to broadcast a command. The use of mpi.bcast is dropped since it cannot support nonblock calls. If nonblock=TRUE, the receivers will run an infinite repeat loop with nonblock call mpi.iprobe. If there is no message, it will sleep and try again. Accordingly, mpi.spawn.Rslaves has two more options nonblock and sleep. If nonblock=TRUE (default), mpi.spawn.Rslaves() will create slaves with no CPU consumption while waiting. The default value sleep=0.1 (sec) seems using least CPU but still responding to a command quickly. To make slaves more responding, change sleep to a smaller number.

It seems loading libmpi.so with error checking fails Rmpi for Mac. This is because Mac OS  doesn't use .so naming. Loading libmpi.dylib has been added. Grab a binary one here. It is compiled under Lion (64bit only)..

Detection of Mac OS is added to mpi.universe.size in case that Rmpi is running under Open MPI and the default value is 1. In this case, the total number of cores is returned instead of 1. This will be useful when spawning slaves on a single Mac. A machine with hyperthreaded cores is not tested though.

Functions bin.nchar, .mpi.serialize and .mpi.unserialize are deprecated.

.First.Lib replaced by .onLoad, .Last.Lib by .onUnload. Fix a unloading warning message for Mac. It looks like that the option --disable-dlopen is not necessary to install Open MPI 1.6, at least on Debian. This might be R's .onLoad correctly loading dynamic libraries and Open MPI is not required to be compiled with static libraries enabled.

Fix all "no visible binding for" NOTE.

inst/Rslaves.bat is removed. Also .bat is renamed to .cmd. Modify src/Makevars.win so that Rmpi can be compiled under both MPICH2 32 and 64bit.

2012-08-08, version 0.6-0 (released to CRAN):

Rmpi has been successfully compiled under 64bit R 2.14.1 on Windows 7 64bit platform. The latest MPICH2 1.4.1p1 or later is needed.

mpi.gather.Robj has been modified to include ... option. Now one can use simplify=FALSE to tell mpi.gather.Robj not to simplify outputs, suggested by "George Ostrouchov" <ostrouchovg at ornl.gov>.

To compile Rmpi against OpenMPI 1.5 or 1.6, you have to enable static library during compiling OpenMPI:
 .
/configure --disable-dlopen
This will create static libraries in additional to shared libraries (not my own idea; coming from here). Otherwise Rmpi will not be compiled. On the other hand, since OpenMPI 1.5, libmpi.so.0 no longer exists. To accommodate both OpenMPI 1.4 or older and OpenMPI 1.5 or newer,  libmpi.so.0 will be loaded first. If it fails, libmpi.so wil be loaded next.  If either libmpi.so.0 or libmpi.so is not loaded properly, it will generate an error like
mca_paffinity_linux.so: undefined symbol: mca_base_param_reg_int
ERROR: loading failed

Loading libmpi.so with error checking was suggested and patched by "Hugo Mildenberger" <Hugo.Mildenberger at web.de>.

Add –lmpl and –lopa to MPICH2 lib in configure.ac so that Rmpi works under FreeBSD and MPICH(2), suggested by "Rainer Hurling" <rhurlin at gwdg.de>.

An option quiet=FALSE has been added to mpi.comm.spawn and mpi.spawn.Rslaves. If enabled, it avoids printing anything if there 

were no errors. This option was suggested and patched byManuel López-Ibáñez” <manuel.lopez-ibanez at ulb.ac.be>.

Default CC="gcc -std=gnu99" has been added to configure.ac in case R_HOME is not set properly, suggested by "Tony Dale" <Tony.Dale at canterbury.ac.nz>. 
mpi.setup.sprng has been removed. Rmpi will reply only on the package rlecuyer for parallel RNG.

2010-11-30, version 0.5-9 (released to CRAN):

slave.hostinfo is modified so that in the default short form with 8 or more slaves, the first 3 and last 2 slaves are shown.

If slaves are not generated through spawning (mpi.comm.size(0) < mpi.comm.size(comm)), mpi.close.Rslaves is not going to delete any log files.

Modify configure.ac so that Rmpi can be compiled under 64bit openSUSE 11.3, suggested by Martin Morgan  <mtmorgan at fhcrc.org>.

All LAM related commands are called only under LAM , i.e., if .Call("mpidist",PACKAGE="Rmpi") == 2 is true. Thanks to Prof. Brian Ripley and Martin Morgan  <mtmorgan at fhcrc.org> for identifying related bugs.

Add a stream to the master in mpi.setup.rngstream so that master can replicate its own RNG.

Add Rslaves32.bat to work under R-2.12.0 32bit.

mpichhosts has been rewritten to accommondate new changes in MPICH2 v1.2.1. system("smpd -get hosts", intern=TRUE) is used to find MPICH2's hosts information instead of registry query. Due to a bug in R's function system, it returns empty under both Windows 7 32bit and 64bit for R 2.10.1 or lower. This bug is fixed in R 2.11.0. When hosts information is not specified in MPICH2 (default is used), mpichhosts() will find total number of CPUs from registry query.

2010-1-18, version 0.5-8 (released to CRAN):

Add mpi.parMM and mpi.iparMM functions for parallel matrix products. mpi.parMM is a clone of parMM from snow package. It has a load balancing parmeter job.num.  Although parallel matrix product can be achieved by mpi.parApply(A, 1, "%*%", B), mpi.parMM is more efficient and faster. Choose job.num to be 2 to 4 times of total number of slaves to futher speed up of  huge matrix product. mpi.iparMM is a nonblocking version of mpi.parMM without load balancing feature.

Fix a bug in mpi.iapplyLB.

configure.ac has been completely rewriteen by Prof. Brian Ripley. Now it allows --with-Rmpi-include= and --with-Rmpi-libpath= options so that MPI's include and lib paths don't need to have the same root. The original option  --with-Rmpi= is still available. Another feature includes the option --with-Rmpi-type= to specify OPENMPI, LAM, or MPICH. Thanks "Prof. Brian Ripley" <ripley at stats.ox.ac.uk>.

Fix a bug in mpi.comm.dup. Thanks   "George Ostrouchov" <ostrouchovg at ornl.gov>.

Fix "Unknown package(s) ‘serialize’ in Rd xrefs" warning, compiled under 2.10.0 or higher.

2009-2-20, version 0.5-7:

Add nonblocking mpi.iapply, mpi.iapplyLB, mpi.iparApply, mpi.iparLapply, mpi.iparSapply, mpi.iparCapply, mpi.iparRapply, and mpi.iparReplicate functions. In those functions, mpi.iprobe is used while master is waiting for slave results. After mpi.iprobe returns FALSE, master went sleep with duration specified by sleep argument (default 0.001 sec or 1ms). Now, under OpenMPI,  master will use little CPU cycles while waiting for slave results.

Fix a segment fault under OpenMPI 1.3 and Debian 64bit. Thanks "Jeff Squyres" <jsquyres at cisco.com> and "Dirk Eddelbuettel" <edd at debian.org> for the fixing.

2008-12-5, version 0.5-6:

Add C level function mpidist. .Call("mpidist",PACKAGE="Rmpi") returns 1 if it is OpenMPI; 2 if LAM and 3 if MPICH.

Modify .First.lib so that dynamic library will be loaded first. Then in case LAM is detected, lamboot might be used.

Add additinoal detections of OpenMPI at various locations, including /usr/lib/openmpi, suggested by "Dirk Eddelbuettel" <edd at debian.org>.

Modify mpi.parApply so that parallel matrix product, like mpi.parApply(A, 1, "%*%", B), can be used. This was suggested by "Martin Morgan" < mtmorgan at fhcrc org>

Modify mpi.apply so that mpi.scatter.Robj is used instead of mpi.send.Robj. This may result some improvements with large datasets.

Fix a bug in mpi.scatter.Robj and add a new function mpi.scatter.Robj2slave which is more efficient than using mpi.parLapply to distribute components of a list to slaves.

sleep.exe is removed from inst directory to avoid the possibility of virus infection. So is the corresponding line in Rslaves.bat.

Modify Rprofile in inst directory so that it will not generate errors under OpenMPI when leaving slaves.

2007-10-24, version 0.5-5:

dlopen has been used to load libmpi.so explicitly. This is mainly useful for Rmpi under OpenMPI where one might see many error messages:
mca: base: component_find: unable to open osc pt2pt: file not found (ignored)
if libmpi.so is not loaded with RTLD_GLOBAL flag.

In Rslave.sh, [ $3 == "needlog" ]; is changed to [ "$3" = "needlog" ]; This might stop those dreadful error messages:
[: 20: ==: unexpected operator
during spawning R slaves, suggested by "Dirk Eddelbuettel" <edd at debian.org>. Similarly, in configure.ac, 
test "$MPI_ROOT"=="";  is changed to [ [ "$MPI_ROOT" = "" ] ]; Otherwise it will not take --with-mpi argument, reported by "Shintaro KATAYAMA" <katayama at gsc riken jp>.

All objects in C codes are properly initialized, done by "Dirk Eddelbuettel" <edd at debian.org>. This will stop warning messages during compiling on some Linux distributions such as Fedora. 

MPI_Init(&fake_argc, (char ***)(void*)&fake_argv) has been used when compiling Rmpi under MPICH otherwise loading Rmpi in R will result segmentation errors. This was suggested by "Aaron Sethman" <sethmana at mail.nih.gov>. For other systems even with MPICH2, MPI_Init((void *)0,(void *)0); is still used. 

configure.ac has been reorganized. Now it will detect OpenMPI first, then LAM and finally MPICH or MPICH2. It can also detect OpenMPI or LAM on both Fedora 32bit or 64bit systems. Users can still overwrite those detections with --with-mpi argument or MPI_ROOT environment variable.

Modify mpi.spawn.Rslaves so that it will not get "System could not find the path" error during spawning under Windows.  

The original .splitIndices function (copied from snow) did not split indices evenly. As suggested by my colleague Dr. McLeod <aim at stats.uwo.ca>, I used the algorithm in equal.count from lattice package to split indices evenly.

2007-10-01, version 0.5.4:

Transferring in mpi.bcast.cmd has been changed from character to raw. This might add some overhead but it will avoid any issues associated with character transfer. 

All character vector transfers have been rewritten since R 2.6.0 does not allow any modifications under C level. Due to complication of creating multiple temp buffers for storing receiving buffers, only the first element in a character vector (R char vector) is transferred. For sending buffers, #define CHAR2(x) ((char *) CHAR(x))  has been used, suggested by Prof. Ripley <ripley at stats.ox.ac.uk>. In addition character receive in mpi.irecv is disabled. 

Add lib64/lam in search path in configure.ac. This might make Rmpi compile under FC 64bit.

Change libmpi.a to libmpi.so in configure.ac and add detection of openmpi dir so that Rmpi can be compiled under OpenMPI with full functionality. R can be started like R -> library(Rmpi) -> mpi.spawn.Rslaves() without remote nodes. Or R can be started like "mpirun --host hostfile -np 1 R --no-save" and library(Rmpi) -> mpi.spawn.Rslaves() bur R runs in noninteractive mode without command recall. One major problem associated with OpenMPI is that all R slaves run at 100%CPU while they are waiting for master instructions. So the implementation of MPI_BCAST in OpenMPI is different from that in LAM. Hence Rmpi under OpenMPI is not good for coding and testing. Rather it should be used for real computations.

Add AC_PROG_CC in configure.ac to work with SUSE linux, suggested by George Ostrouchov <ostrouchovg at ornl.gov>.

Modify configure.ac so that LAM on Mac OS X can be detected, provided that OS X version of LAM at www.lam-mpi.org is installed. 

Change configure.in to configure.ac and add MPI_INCLUDE, LDFLAGS, CFLAGS, in case MPI_ROOT is supplied by users, suggested by "Dennis van Dok" <dennisvd at nikhef.nl>.

2006-10-11, version 0.5-3:

R raw data type has been added as type 4. This is used for transferring R objects, i.e., all R objects are serialized to raw type for transferring. This is in line with the change of serialize in R 2.4.0. Wrappers .mpi.serialize and .mpi.unserialize are used in case Rmpi is running under R 2.2.0 to R.2.3.1. 

Add a few examples to several mpi functions.

Add mpi.scatter.Robj, mpi.gather.Robj, and mpi.allgather.Robj.

2006-04-5, version 0.5-2:

Change "MPI_Comm_free(&master);" in mpi_is_master of Rmpi.c to "if (!check) MPI_Comm_free(&master);" Thanks to Martin Siegert <siegert at sfu.ca> for fixing.

Some patch to configure.in, suggested by Aaron A. King <aaron.king at umich.edu>, so that if MPI_ROOT is supplied as an environmental variable or is given by users, mpi.h will be searched within subfolders of MPI_ROOT. 

2005-12-15, version 0.5-1:

Rename some internal functions used by slaves. They follow by the convention .mpi.slave.xxx and hidden.

Automatically loading SPRNG in mpi.spawn.Rslaves is dropped. Instead one can use mpi.setup.sprng or mpi.setup.rngstream to setup parallel RNG on slaves. 

Some codes tidy on mpi.remote.exec and mpi.parSim. As a result, mpi.wrap.fun is dropped.

Add mpichhosts to find host names on windows platform.

Modify mpi.universe.size to return length(mpichhosts())-1 under windows platform.

Add mpi.apply: an array is scattered to slaves and then a fun is applied.Add mpi.applyLB: a load balancing version of mpi.apply.

Add mpi.parApply, mpi.parLapply, mpi.parRapply, mpi.parCapply, and mpi.parSapply: they are clones of those functions in package snow if job.num=slave.num. They are load balanced if job.num > slave.num.

Add mpi.parReplicate: wrapper for mpi.parSapply, mainly used in parallel simulation. It will eventually replace mpi.parSim due to its efficiency.

Add network map drive function in inst/Rslaves.bat. May see some error messages if multiple R slaves are spawned on the same remote node. 

Add nonblocking calls: mpi.isend, mpi.irecv, mpi.isend.Rojb, mpi.iprobe, mpi.test, mpi.testany, mpi.testsome, mpi.testall, mpi.wait, mpi.waitany, mpi.waitsome, mpi.waitall, mpi.cancel, and mpi.test.cancelled.

Add mpi.comm.maxsize, mpi.request.maxsize, mpi.status.maxsize, mpi.realloc.comm, mpi.realloc.request, and mpi.realloc.status.

2005-11-28, version 0.5-0:

Rmpi can be compiled and run under MPICH2 for Windows. src/Makevars.win is added so that Rmpi can be compiled without manual configuration if MPICH2 for Windows is installed at default place. Instructions to run Rmpi on windows platform are given in here.

inst/Rslaves.bat windows batch file is created for spawning R slaves under windows platform (localhost only, a restriction imposed by MPICH2 for Windows). 

Full sets of creating a Cartesian topology APIs are implemented. 

inst/Rslaves.sh is modified so that the same R program Master R is ruining will be used by slaves Rs. This allows co-existences of different versions of R.

inst/Rprofile used by mpiexec under unix and windows platforms is modified. The main change is that sys.parent() in eval is removed. This is due to a unknown reason that the infinite loop cannot be broken if the env R_RPOFILE is used to launch multiple Rs. Objects created on R slaves are no longer global unless <<- are used, though they are accessible during the duration of R slaves. If Rprofile is copied as .Rprofile at the working directory, sys.parent() can still be used (need to modify .Rprofile). 

To run multiple Rs without using .Rprofile, add the following line R_PROFILE=${R_HOME_DIR}/library/Rmpi/Rprofile; export R_PROFILE in the R shell script (after "export R_HOME_DIR") and rename it, say, Rmpi. Then one can use mpiexec -n number_of_nodes Rmpi --no-save -q to launch multiple Rs (assume Rmpi is in the PATH).

Parts of nonblocking point-to-point MPI APIs are implemented at C level only. They will be released in the next version of Rmpi

2005-05-09, version 0.4-9:

mpi.remote.fun is deprecated and mpi.remote.exec (mpi.remote.slave) has been simplified by using do.call instead.
replicate is used in mpi.parallel.sim (mpi.remote.slave) instead of sapply. To be consistent, mpi.parallel.sim uses cbind to combine all slave results together.

2004-04-21, version 0.4-8:

Delete .Random.seed in .First.lib if it is a user-defined RNG. Add needsprng = TRUE argument to mpi.spawn.Rslaves so that SPRNG can be loaded if there is the package rsprng. Implement mpi.parallel.sim for parallel Monte Carlo simulation. Some ideas from student projects of Mr. K. Yu, Ms. Tsai, and Mr. W. Wolterman have been used. Thanks to them for their wonderful works. 

2004-04-05, version 0.4-7: 

Update to require R-1.8.1 and remove the requirement for the serialise package (build into R-1.8.1).

Some minor bugs fix. 

Add #ifdef MPI2 to src/Rmpi.c and change MPI_Comm_set_errhandler back to MPI_Errhandler_set for backward compatibility with MPI-1.2, suggested by Ross Boylan

Add -DMPI2 in configure.in in case lam lib is detected. Otherwise MPI-2 related functions will be disabled.

Add detect MPI in /usr/lib/mpi or /usr/lib/mpich, hopefully to catch MPICH MPI. 

2003-09-24, version 0.4-6:

Add some lines to configure.in so that configure can detect lam on Debian Linux. Thanks to Kurt Hornik <Kurt.Hornik at wu-wien.ac.at> for Debian instructions.

2003-02-03, version 0.4-5: 

Internal use.

2002-06-01, version 0.4-4:

Add a function string for creating a (binary) character buffer. Remove all C++ style comments. Remove all deprecated functions. Remove mpi.barrier from slavedaemon.R (it causes slaves hanging if master crashed).

2002-05-06, version 0.4-3:

Take SPRNG out. Use rsprng package instead. Use mpi.init.sprng to start SPRNG.

Require serialize package

mpi.send.Robj, mpi.recv.Robj, mpi.bcast.Robj, and mpi.bcast.Robj2slave use serialize to transfer R objects.

mpi.bcast.send and mpi.bcast.recv are deprecated.

mpi.bcast.send.cmd and mpi.bcast.recv.cmd are replaced by mpi.bcast.cmd

mpi.bcast.send.Robj and mpi.bcast.recv.Robj are replaced by mpi.bcast.Robj

New instructions for installing Rmpi for other MPI implementations. 

Thanks to Kurt Hornik <Kurt.Hornik at wu-wien.ac.at> for Debian instructions.

2002-05-02, version 0.4.2: 

Some bugs fix. Add Rprofile in inst Add a function mpi.comm.dup

2002-04-29, version 0.4.1: 

Some bugs fix. Add a function mpi.quit

2002-04-26, version 0.4.0: 

Initial public release