Contents

Introduction

This page covers the gnarly system-specific overhead required to run R in parallel on XSEDE's Gordon and Trestles resources at SDSC, and more generally, on shared computing resources such as clusters and supercomputers. It does not cover the conceptual or practical details of specific parallel libraries available to R programmers; this information can be found in my page on Parallel Options for R. Rather, it covers what you need to know about installing libraries and running R on someone else's supercomputer rather than your personal laptop or desktop.

Loading the R module

As of December 2013, the following versions of R are available on SDSC's XSEDE resources, Gordon and Trestles:

You can load the appropriate modules using module load R/3.0.1. In addition to these versions of R which are officially provided by SDSC User Services, I provide a custom build of R 3.0.2 on Gordon that has been optimized for high-performance and parallel computations. To access this, you have to add the following line to the bottom of your ~/.bashrc file:

export MODULEPATH=/home/glock/apps/modulefiles:$MODULEPATH

Then you can load this version of R by using the following two commands:

$ module swap mvapich2_ib openmpi_ib
$ module load R/3.0.2

The tutorials that accompany this guide use a few special libraries that may be difficult for you to install yourself. As such, my R/3.0.2 module is designed to allow you to try out the examples provided in my Parallel Options for R and Parallel R using Hadoop pages without having to install anything.

One note of caution--because my R and our system-wide R are both the same version, the install.packages command for each one will both install in the same ~/R/x86_64-unknown-linux-gnu-library/3.0 directory which can cause problems. If you need to install your own libraries, you will have to choose to either use my R/3.0.2 or the system R/3.0.1 but not both.

Installing Libraries

Most Libraries

Users cannot install R libraries globally on our machines, but R makes it very easy for users to install libraries in their home directories. To do this, fire up R and when presented with the > prompt, use the install.packages() method to install things:

> install.packages('doSNOW')
Installing package(s) into '/opt/R/local/lib'
(as 'lib' is unspecified)
Warning in install.packages("doSNOW") :
 'lib = "/opt/R/local/lib"' is not writable
Would you like to use a personal library instead?  (y/n)

This error comes up because you can't install libraries system-wide as a non-root user. Say y and accept the default which should be something similar to ~/R/x86_64-unknown-linux-gnu-library/3.0. Pick a mirror and let her rip. If you want to install multiple packages at once, you can just do something like

> install.packages(c('multicore','doMC'))
Common Error

When installing libraries, you may eventually get an error like this:

> install.packages('lmtest');
Installing package(s) into '/home/glock/R/x86_64-unknown-linux-gnu-library/2.15'
(as 'lib' is unspecified)
trying URL 'http://cran.stat.ucla.edu/src/contrib/lmtest_0.9-30.tar.gz'
Content type 'application/x-tar' length 176106 bytes (171 Kb)
opened URL
==================================================
downloaded 171 Kb
 
* installing *source* package 'lmtest' ...
** package 'lmtest' successfully unpacked and MD5 sums checked
** libs
gfortran   -fpic  -g -O2  -c pan.f -o pan.o
gcc -std=gnu99 -shared -L/usr/local/lib64 -o lmtest.so pan.o -lgfortran -lm -lquadmath
/usr/bin/ld: cannot find -lquadmath
make: *** [lmtest.so] Error 2
ERROR: compilation failed for package 'lmtest'
* removing '/home/glock/R/x86_64-unknown-linux-gnu-library/2.15/lmtest'
 
The downloaded source packages are in
   '/tmp/RtmpDTJxhk/downloaded_packages'
Warning message:
In install.packages("lmtest") :
 installation of package 'lmtest' had non-zero exit status

The relevant part is the failure to compile the portion of the module written in Fortran due to /usr/bin/ld: cannot find -lquadmath. This missing quadmath library first appeared in GNU GCC 4.6, so if this error comes up, it means your default gcc is not the same one used to compile R. On SDSC's systems, this is remedied by doing module load gnubase.

Alternatively, you might see an error like this:

Error: in routine alloca() there is a
stack overflow: thread 0, max 137437318994KB, used 0KB, request 48B

Again, this is caused by gnu/4.8.2 not being loaded. The following commands will remedy the issue on Trestles:

$ module unload pgi
$ module load gnu/4.8.2 openmpi_ib R

Installing Rmpi

In order to use distributed memory parallelism (i.e., multi-node jobs) within R, you will need to use Rmpi in some form or another. Under the hood, the ClusterApply and related functions provided in the parallel library use Rmpi as the most efficient way to utilize the high-performance interconnect available on supercomputers.

Rmpi is already installed if you do module swap mvapich2_ib openmpi_ib before loading the R module we provide on Gordon or module swap pgi gnu/4.8.2 and module load openmpi_ib on Trestles. The following information details how you would have to install it yourself on your own laptop, lab workstation, etc.

It's a lot harder to install Rmpi using install.packages() because you have to feed the library installation process some system-specific library locations before it will work correctly. So, exit R and return to the Linux prompt. Download the Rmpi source distribution:

$ wget http://cran.r-project.org/src/contrib/Rmpi_0.6-3.tar.gz

Rmpi 0.6-3 was the most recent version at the time I wrote this, but you can get the most recent version's download URL from the Rmpi page at CRAN. Once you have downloaded the file, make sure you have both the gnu and openmpi_ib modules loaded, then issue the following gnarly command:

$ mkdir -p ~/R/x86_64-unknown-linux-gnu-library/3.0
$ R CMD INSTALL \
   --configure-vars="CPPFLAGS=-I$MPIHOME/include LDFLAGS='-L$MPIHOME/lib'" \
   --configure-args="--with-Rmpi-include=$MPIHOME/include \
                     --with-Rmpi-libpath=$MPIHOME/lib \
                     --with-Rmpi-type=OPENMPI" \
   -l ~/R/x86_64-unknown-linux-gnu-library/3.0 Rmpi_0.6-3.tar.gz

Be sure to double-check the text in green! These might change if your version of R or Rmpi are different. If all goes well, you should see a lot of garbage that ends with

gcc -std=gnu99 -shared -L/usr/local/lib64 ...
installing to /home/glock/R/x86_64-unknown-linux-gnu-library/3.0/Rmpi/libs
** R
** demo
** inst
** preparing package for lazy loading
** help
*** installing help indices
** building package indices
** testing if installed package can be loaded
--------------------------------------------------------------------------
The OpenFabrics (openib) BTL failed to initialize while trying to
allocate some locked memory.  This typically can indicate that the
memlock limits are set too low.  For most HPC installations, the
memlock limits should be set to "unlimited".  The failure occured
here:
 
 Local host:    trestles-login2.sdsc.edu
 OMPI source:   btl_openib_component.c:1216
 Function:      ompi_free_list_init_ex_new()
 Device:        mlx4_0
 Memlock limit: 65536
 
You may need to consult with your system administrator to get this
problem fixed.  This FAQ entry on the Open MPI web site may also be
helpful:
 
   http://www.open-mpi.org/faq/?category=openfabrics#ib-locked-pages
--------------------------------------------------------------------------
--------------------------------------------------------------------------
WARNING: There was an error initializing an OpenFabrics device.
 
 Local host:   trestles-login2.sdsc.edu
 Local device: mlx4_0
--------------------------------------------------------------------------
* DONE (Rmpi)

That error about the OpenFabrics device is nothing to worry about; it happens because the test is running on one of the cluster's login nodes (where you are doing all of this) and cannot access the MPI execution environment that real jobs on compute nodes use.

At this point you should have Rmpi installed, and this allows the snow package to use MPI for distributed computing. If you run into an error that looks like this:

...
** preparing package for lazy loading
** help
*** installing help indices
** building package indices
** testing if installed package can be loaded
Error: in routine alloca() there is a
stack overflow: thread 0, max 10228KB, used 0KB, request 48B
ERROR: loading failed
* removing '/home/glock/R/x86_64-unknown-linux-gnu-library/2.15/Rmpi'

You probably forgot to load the prerequisite modules correctly. Purge all of your currently loaded modules, then re-load the ones necessary to build R libraries:

$ module purge
$ module load gnu/4.6.1 openmpi R
$ R CMD INSTALL --configure-vars=...
Note about MVAPICH2

If you log into Gordon, start an interactive job (do not run R on the login nodes!), and try run a snow-based script which calls ClusterApply, you may find that it just segfaults:

$ mpirun_rsh -np 1 -hostfile $PBS_NODEFILE $(which R) CMD BATCH ./snowapp.R
[gcn-14-17.sdsc.edu:mpispawn_0][readline] Unexpected End-Of-File on file descriptor 5. MPI process died?
[gcn-14-17.sdsc.edu:mpispawn_0][mtpmi_processops] Error while reading PMI socket. MPI process died?
/opt/R/lib64/R/bin/BATCH: line 60: 130758 Segmentation fault      ${R_HOME}/bin/R -f ${in} ${opts} ${R_BATCH_OPTIONS} > ${out} 2>&1
[gcn-14-17.sdsc.edu:mpispawn_0][child_handler] MPI process (rank: 0, pid: 130753) exited with status 139

If you instead try to use mpiexec (which is mpiexec.hydra) you will instead get this error:

*** caught segfault ***
address 0x5ddcbc7, cause 'memory not mapped'
 
Traceback:
1: .Call("mpi_comm_spawn", as.character(slave), as.character(slavearg),     as.integer(nslaves), as.integer(info), as.integer(root),     as.integer(intercomm), as.integer(quiet), PACKAGE = "Rmpi")
2: mpi.comm.spawn(slave = mpitask, slavearg = args, nslaves = count,     intercomm = intercomm)
3: snow::makeMPIcluster(spec, ...)
4: makeCluster(10, type = "MPI")
aborting ...

Alternatively, your application may produce this error instead of segfaulting:

Error in mpi.universe.size() :
 This function is not supported under MPI 1.2
Calls: mpi.spawn.Rslaves -> mpi.comm.spawn -> mpi.universe.size
Execution halted

These errors all indicate a major bug in the Rmpi package which remains fixed. I take this to mean that Rmpi simply does not work with MVAPICH2. Use OpenMPI when using Rmpi or its derivatives. You can do this by loading the openmpi_ib module before loading the R module.

Submitting Parallel R Jobs

On personal workstations, R is often used by running the R shell in an interactive fashion and either typing in commands or doing something like source('script.R'). Supercomputers generally operate through batch schedulers though, so you will want to get your R script running non-interactively. There are a few ways of doing this:

  1. Add #!/usr/bin/env Rscript to the very top of your R script and make it executable (chmod +x script.R), then just run the script as you would a bash script or any program (./script.R)
  2. Call Rscript with the script's name as a command line parameter: Rscript script.R. I've seen this break an otherwise working R script though, and I never got to the bottom of it.
  3. Call R CMD BATCH script.R

I am going to use #3 in the following examples because it is the most proper way. The sample job submit scripts that follow are for Torque, which is the batch manager we run on SDSC Gordon and Trestles. These scripts can be trivially adapted to SGE/Grid Engine, Slurm, LSF, Load Leveler, or whatever other batch system your system may have.

Running Serial R Jobs

Running an R script in serial is quite straightforward. On XSEDE's Gordon resource at SDSC, your queue script should look something like this:

#!/bin/sh
#PBS -q normal
#PBS -l nodes=1:ppn=16:native
#PBS -l walltime=0:05:00
 
module load R
 
cd $PBS_O_WORKDIR
 
export OMP_NUM_THREADS=1
R CMD BATCH test.serial.R

The peculiar bit to note here is our use of export OMP_NUM_THREADS=1. If you don't specify this, R will use as many threads as it can grab if your script uses a library that supports multithreading. This isn't bad per se, but explicitly specifying OMP_NUM_THREADS is good practice--that way you always know exactly how many cores your script will use.

Running Shared-Memory R Jobs

Running shared-memory parallel R on a single node is also quite simple. Here is a sample queue script that uses all 16 cores on a dedicated (non-shared) node.

#!/bin/sh
#PBS -q normal
#PBS -l nodes=1:ppn=16:native
#PBS -l walltime=0:05:00
 
module load R
 
cd $PBS_O_WORKDIR
 
export OMP_NUM_THREADS=1
R CMD BATCH test.doMC.R

It is actually the same script as the serial version. Bear in mind that the OMP_NUM_THREADS only controls the number of cores used by libraries which support OpenMP. By comparison, the multicore (and parallel) libraries do not use OpenMP; they let you control the number of cores from within R.

Running Parallel Jobs with snow/doSNOW

Snow (and its derived libraries) does its own process managements, so you must run it as if it were a one-way MPI job. For example,

#!/bin/sh
#PBS -q normal
#PBS -l nodes=2:ppn=16:native
#PBS -l walltime=0:05:00
 
module swap mvapich2_ib openmpi_ib
module load R
 
cd $PBS_O_WORKDIR
 
export OMP_NUM_THREADS=1
mpirun -np 1 -hostfile $PBS_NODEFILE R CMD BATCH test.doSNOW.R

If you forget to request only one core, your job will fail and you will get a lot of horrific errors:

CMA: unable to open RDMA device
CMA: unable to open RDMA device
[[59223,26],16][btl_openib_component.c:1493:init_one_device] error obtaining device context for mlx4_0 errno says No such device
[[59223,27],23][btl_openib_component.c:1493:init_one_device] error obtaining device context for mlx4_0 errno says No such device
[[59223,31],5][btl_openib_component.c:1493:init_one_device] error obtaining device context for mlx4_0 errno says No such device

This is because the R script winds up running in duplicate, and each duplicate tries to spawn a full complement of its own MPI ranks. Thus, instead of getting 2×16 ranks, you get (2×16)×(2×16).

Trivial sample code

I've created some trivial Hello World samples that can be found on my GitHub account. They illustrate the very minimum needed to use the parallel backends for the foreach package and are a good way to verify that your parallel libraries and R environment are set up correctly. The idea here is that you can replace the hello.world() function with something useful and be off to a good start.

However, these samples do not illustrate how data, libraries, and subfunctions may have to be transferred to other nodes when using MPI. For more details on how to approach those more realistic problems, see the next part in this series: Parallel Options for R.