A function for parallel processing on Windows and servers (R parallelisation, part 2)

Jan 24, 2019·
Julián Cabezas Peña
Julián Cabezas Peña
· 3 min read
post parallel

This short tutorial is the second part of my introduction to parallelising R on Windows. I recommend reading the first part.

The problem: creating clusters is tedious

Creating clusters can be tedious: loading packages one by one and exporting datasets is slow, and it is easy to forget a crucial package. We want a more convenient approach. We may also be working on a server without Internet access, which complicates package installation.

The solution: a function to create clusters

I created a function, available in this GitHub file. To give credit where it is due, it is inspired by parallelsugar, with a few modifications.

The function creates a cluster with n nodes, specified by the nnodes argument.

create_cluster is designed to work on servers. When they have no Internet connection, packages can be installed in a shared folder and loaded from there. Use libroute to specify the library path.

create_cluster <- function(nnodes, libroute = NULL) {

nnodes: Number of nodes

libroute: Provided library route (different of the default library)

tryCatch({ cl <- parallel::makeCluster(nnodes) # Create an empty cluster with n nodes

loaded.package.names &lt;- c( # Find the loaded packages in the session
  sessionInfo()$basePkgs, # base packages
  names(sessionInfo()$otherPkgs)
) # aditional loaded packages
# Send the packages and the complete environment to the cluster
parallel::clusterExport(cl,
  &quot;loaded.package.names&quot;, # loaded packages
  envir = environment()
) # &quot;Environment&quot;

## Charge the packages in all nodes
if (is.null(libroute)) { # Use default library location
  parallel::parLapply(cl, 1:length(cl), function(xx) {
    lapply(loaded.package.names, function(yy) {
      library(yy, character.only = TRUE)
    })
  })
} else {
  # If we provide a library location use it (useful when working in remote machines)
  parallel::parLapply(cl, 1:length(cl), function(xx) {
    lapply(loaded.package.names, function(yy) {
      library(yy, character.only = TRUE, lib.loc = libroute)
    })
  })
}

}) return(cl) # The result is the cluster }

A small, recycled example

In the example from the previous tutorial, we used clusterCall to load randomForest. Here, if the package and any other dependencies are already loaded in our session, we can export them to the nodes in one step.

First, load the package and create the function:

library(randomForest)

rf_boot_fun <- function(iter, data, n_sample) { ind <- sample(1:nrow(data), size = n_sample) data_sample <- data[ind, ]

Fit the Random Forest model

model1 <- randomForest(Sepal.Length ~ Petal.Length, data = data_sample)

Extract the pseudo-R2

R2 <- model1$rsq[length(model1$rsq)]

Extract the MSE

MSE <- model1$mse[length(model1$mse)] results <- c("R2" = R2, "MSE" = MSE) return(results) }

Now I can apply the function with parLapply.

library(parallel)

Define a library path (optional)

ruta_libreria <- "C:/Users/Julian/Documents/R/win-library/3.5"

This can simply be nuevoclus<-create_cluster(4)

nuevoclus <- create_cluster(4, ruta_libreria)

10,000 iterations of our function using the "iris" dataset

results <- parLapply(nuevoclus, 1:10000, rf_boot_fun, data = iris, n_sample = 100)

stopCluster(nuevoclus) # Close the cluster; do not forget this step

The first two results

head(results, n = 2)

## [[1]]
##       R2      MSE
## 0.778244 0.174310
##
## [[2]]
##        R2       MSE
## 0.6912495 0.1865890

Done—parallelisation is much easier now!

I hope you find this useful!

I welcome your comments and questions!

Julián Cabezas Peña
Authors
AI and the environment

I am a data scientist and Natural Resources Engineer from Universidad de Chile, with a Master of Data Science from the University of Adelaide. I combine artificial intelligence, data science, statistical analysis and geospatial analysis to tackle environmental challenges.

Since February 2022, I have coordinated the Data Science team at Chile’s Superintendence of the Environment (SMA). I develop automation, data validation and analytical solutions for environmental monitoring and enforcement, using R, Python and SQL.

My experience also includes computer vision and deep learning for agricultural robotics, forest carbon modelling, remote sensing and survey data analysis. I have worked in Chile, Australia and Germany, combining applied research, algorithm development and interdisciplinary team coordination.