A function for parallel processing on Windows and servers (R parallelisation, part 2)

This short tutorial is the second part of my introduction to parallelising R on Windows. I recommend reading the first part.
The problem: creating clusters is tedious
Creating clusters can be tedious: loading packages one by one and exporting datasets is slow, and it is easy to forget a crucial package. We want a more convenient approach. We may also be working on a server without Internet access, which complicates package installation.
The solution: a function to create clusters
I created a function, available in this GitHub file. To give credit where it is due, it is inspired by parallelsugar, with a few modifications.
The function creates a cluster with n nodes, specified by the nnodes argument.
create_cluster is designed to work on servers. When they have no Internet connection, packages can be installed in a shared folder and loaded from there. Use libroute to specify the library path.
create_cluster <- function(nnodes, libroute = NULL) {
nnodes: Number of nodes
libroute: Provided library route (different of the default library)
tryCatch({
cl <- parallel::makeCluster(nnodes) # Create an empty cluster with n nodes
loaded.package.names <- c( # Find the loaded packages in the session
sessionInfo()$basePkgs, # base packages
names(sessionInfo()$otherPkgs)
) # aditional loaded packages
# Send the packages and the complete environment to the cluster
parallel::clusterExport(cl,
"loaded.package.names", # loaded packages
envir = environment()
) # "Environment"
## Charge the packages in all nodes
if (is.null(libroute)) { # Use default library location
parallel::parLapply(cl, 1:length(cl), function(xx) {
lapply(loaded.package.names, function(yy) {
library(yy, character.only = TRUE)
})
})
} else {
# If we provide a library location use it (useful when working in remote machines)
parallel::parLapply(cl, 1:length(cl), function(xx) {
lapply(loaded.package.names, function(yy) {
library(yy, character.only = TRUE, lib.loc = libroute)
})
})
}
})
return(cl) # The result is the cluster
}
A small, recycled example
In the example from the previous tutorial, we used clusterCall to load randomForest. Here, if the package and any other dependencies are already loaded in our session, we can export them to the nodes in one step.
First, load the package and create the function:
library(randomForest)
rf_boot_fun <- function(iter, data, n_sample) {
ind <- sample(1:nrow(data), size = n_sample)
data_sample <- data[ind, ]
Fit the Random Forest model
model1 <- randomForest(Sepal.Length ~ Petal.Length, data = data_sample)
Extract the pseudo-R2
R2 <- model1$rsq[length(model1$rsq)]
Extract the MSE
MSE <- model1$mse[length(model1$mse)]
results <- c("R2" = R2, "MSE" = MSE)
return(results)
}
Now I can apply the function with parLapply.
library(parallel)
Define a library path (optional)
ruta_libreria <- "C:/Users/Julian/Documents/R/win-library/3.5"
This can simply be nuevoclus<-create_cluster(4)
nuevoclus <- create_cluster(4, ruta_libreria)
10,000 iterations of our function using the "iris" dataset
results <- parLapply(nuevoclus, 1:10000, rf_boot_fun, data = iris, n_sample = 100)
stopCluster(nuevoclus) # Close the cluster; do not forget this step
The first two results
head(results, n = 2)
## [[1]]
## R2 MSE
## 0.778244 0.174310
##
## [[2]]
## R2 MSE
## 0.6912495 0.1865890Done—parallelisation is much easier now!

I hope you find this useful!
I welcome your comments and questions!

I am a data scientist and Natural Resources Engineer from Universidad de Chile, with a Master of Data Science from the University of Adelaide. I combine artificial intelligence, data science, statistical analysis and geospatial analysis to tackle environmental challenges.
Since February 2022, I have coordinated the Data Science team at Chile’s Superintendence of the Environment (SMA). I develop automation, data validation and analytical solutions for environmental monitoring and enforcement, using R, Python and SQL.
My experience also includes computer vision and deep learning for agricultural robotics, forest carbon modelling, remote sensing and survey data analysis. I have worked in Chile, Australia and Germany, combining applied research, algorithm development and interdisciplinary team coordination.