Parallelising R on Windows with the parallel package

Jan 7, 2019·
Julián Cabezas Peña
Julián Cabezas Peña
· 11 min read
post R

Why parallelise?

You have been asked to run code that takes 20 hours to finish. It applies the same function to different parameters or datasets many, many times.

That might not seem like a problem, but requests often arrive yesterday without 20 hours of notice. If you are wondering why your PC or workstation has 4, 8 or 16 cores, this tutorial is for you!

Parallelising R processes can be difficult. Add bad luck Windows to the equation, and it can feel almost impossible.

In this tutorial (the first on this website!), we will introduce parallel processing in R on Windows. We will start with the apply family, explore the parallel package and create a Windows cluster, before finishing with a common machine learning application.

Tutorial level

This tutorial assumes you are an intermediate or advanced R user.

Turn your loop into a function

We will use the iris dataset, familiar to many R users. It contains petal and sepal length and width measurements for three flower species.

data(iris)
knitr::kable(head(iris))
Sepal.LengthSepal.WidthPetal.LengthPetal.WidthSpecies
5.13.51.40.2setosa
4.93.01.40.2setosa
4.73.21.30.2setosa
4.63.11.50.2setosa
5.03.61.40.2setosa
5.43.91.70.4setosa

Suppose a colleague wants to fit a linear model relating petals to sepals. Because the sample is small (150 individuals), you are asked to run 20,000 bootstrap iterations to examine the model coefficients and R2.

We can do this with a loop that draws a sample of 100 individuals (two-thirds of the dataset), fits a linear model, and saves the R2 and coefficients in a data frame.

# Create an empty data frame to store the results
results <- data.frame()

system.time({ for (i in 1:20000) { # Draw a sample of 100 numbers between 1 and 150 ind <- sample(1:nrow(iris), size = 100) iris_sample <- iris[ind, ] # Fit a linear model using petal and sepal measurements model1 <- lm(iris_sample[, "Sepal.Length"] ~ iris_sample[, "Petal.Length"])

# Extract R2 and the coefficients
R2 &lt;- summary(model1)$r.squared
coef &lt;- coefficients(model1)
# Combine everything in the results data frame
results &lt;- rbind(results, c(R2, coef))

} })

##    user  system elapsed
##   20.90    0.03   20.95

The algorithm takes just over 20 seconds. That may be acceptable now, but what if tomorrow you need to do the same thing with half a million observations?

The problem is that the loop uses only one processor core, running one iteration after another. This is not very efficient.

The first step towards parallelisation is to turn the loop into a function. Here the only argument will be an iteration vector, “iter”, which lets us request more iterations.

lm_boot_fun <- function(iter) {
  ind <- sample(1:nrow(iris), size = 100)
  iris_sample <- iris[ind, ]
  model1 <- lm(iris_sample[, "Sepal.Length"] ~ iris_sample[, "Petal.Length"])
  R2 <- summary(model1)$r.squared
  coef <- coefficients(model1)
  # Create a vector of results
  results <- c("R2" = R2, coef)
  # Return the function output
  return(results)
}

Let us test the function. As the following code shows, it returns a result regardless of the argument we pass.

result <- lm_boot_fun(1)
result
##                            R2                   (Intercept)
##                     0.7554039                     4.3353075
## iris_sample[, "Petal.Length"]
##                     0.3999710
result <- lm_boot_fun(56876)
result
##                            R2                   (Intercept)
##                     0.7393922                     4.3426829
## iris_sample[, "Petal.Length"]
##                     0.3939433

Passing a vector still returns just one result, because “results” is overwritten on every iteration.

result <- lm_boot_fun(1:10)
result
##                            R2                   (Intercept)
##                     0.7732318                     4.2822007
## iris_sample[, "Petal.Length"]
##                     0.4256894

Using lapply

To keep all the results, we can use lapply.

lapply belongs to the apply family (apply, sapply, vapply, lapply and mapply), one of R’s most powerful tools—and one I wish I had learned earlier. This tutorial is not about the whole family, so we will focus on lapply.

lapply applies another function, such as the one we created, to a list or vector of arguments, and returns a list of the same length.

iter <- seq(1, 20000)
system.time({
  results <- lapply(iter, lm_boot_fun)
})
##    user  system elapsed
##   15.05    0.00   15.07

Processing time has already improved a little, but we want to make it much faster. One detail to handle is that lapply returns a list.

head(results, n = 3)
## [[1]]
##                            R2                   (Intercept)
##                     0.7795166                     4.2839537
## iris_sample[, "Petal.Length"]
##                     0.4123361
##
## [[2]]
##                            R2                   (Intercept)
##                     0.7683068                     4.2132627
## iris_sample[, "Petal.Length"]
##                     0.4311986
##
## [[3]]
##                            R2                   (Intercept)
##                     0.7909086                     4.2600338
## iris_sample[, "Petal.Length"]
##                     0.4163103

To convert this list into a data frame, use unlist to build a matrix and then convert it. Here we specify three columns.

results <- data.frame(matrix(unlist(results), ncol = 3, byrow = T))
head(results, n = 2)
##          X1       X2        X3
## 1 0.7795166 4.283954 0.4123361
## 2 0.7683068 4.213263 0.4311986

You can find more elegant solutions in this Stack Overflow discussion.

The parallel package

The parallel package is included with R. It provides parallel versions of functions in the apply family. Before using them, we need to find out how many cores are available.

library(parallel)
numCores <- detectCores()
numCores
## [1] 8

We have eight logical cores! As a rule of thumb for this example, we will use half of them (four). On a Unix-like system, such as Linux or macOS, we could use mclapply. However, mclapply relies on forking, which creates a copy of the process and is available only on Unix-like systems.

iter <- seq(1, 20000)
system.time({
  results <- mclapply(iter, lm_boot_fun, mc.cores = 7)
})
## Error in mclapply(iter, lm_boot_fun, mc.cores = 7): 'mc.cores' > 1 is not supported on Windows
## Timing stopped at: 0 0 0

On Windows, we will create a cluster manually using makeCluster. The resulting object becomes an argument to parLapply. In this case, we will use four cores.

cl <- makeCluster(4) # Create the cluster
system.time(
  results <- parLapply(cl, iter, lm_boot_fun)
)
##    user  system elapsed
##    0.03    0.00    4.76
stopCluster(cl) # Close the cluster; do not forget this step

Wow! We have substantially reduced the bootstrap processing time. We can enjoy a cup of mate—after shutting down the cluster with stopCluster.

Great!

We could stop here, but this is a very simple example. In practice, we often use datasets stored on our computers and five to ten packages.

Making the function more complex

Let us pretend iris is a dataset stored on our computer. We will also add two arguments to the function: the dataset and the sample size.

# Write the iris dataset to disk
write.csv(iris, file = "iris.csv")

Read the dataset created above

datos_pc <- read.csv("iris.csv")

lm_boot_fun2 <- function(iter, data, n_sample) {

data and n_sample are now function arguments

ind <- sample(1:nrow(data), size = n_sample) data_sample <- data[ind, ] model1 <- lm(data_sample[, "Sepal.Length"] ~ data_sample[, "Petal.Length"]) R2 <- summary(model1)$r.squared coef <- coefficients(model1) results <- c("R2" = R2, coef) return(results) }

Now we can repeat the operation using the additional arguments of parLapply.

cl <- makeCluster(4) # Create the cluster
system.time(
  results <- parLapply(cl, iter, lm_boot_fun2, data = datos_pc, n_sample = 100)
)
##    user  system elapsed
##    0.01    0.00    4.52
stopCluster(cl) # Close the cluster; do not forget this step

Suppose your manager walks in and says the client wants a Random Forest—perhaps because it sounds impressive. Let us adapt the function using the randomForest package and save the pseudo-R2 and MSE.

library(randomForest)

rf_boot_fun <- function(iter, data, n_sample) { ind <- sample(1:nrow(data), size = n_sample) data_sample <- data[ind, ]

Fit the Random Forest model

model1 <- randomForest(Sepal.Length ~ Petal.Length, data = data_sample)

Extract the pseudo-R2

R2 <- model1$rsq[length(model1$rsq)]

Extract the MSE

MSE <- model1$mse[length(model1$mse)] results <- c("R2" = R2, "MSE" = MSE) return(results) }

Now let us use parLapply.

library(randomForest)
cl <- makeCluster(4) # Create the cluster
system.time(
  results <- parLapply(cl, iter, rf_boot_fun, data = datos_pc, n_sample = 100)
)
## Error in checkForRemoteErrors(val): 4 nodes produced errors; first error: could not find function "randomForest"
## Timing stopped at: 0 0 0
stopCluster(cl) # Close the cluster; do not forget this step

What happened? I already called library(randomForest)!

The randomForest package is loaded in our current session. However, makeCluster opens n new R sessions, each starting from scratch. We must load the packages in every session. In some cases, we must also export datasets and functions.

To load randomForest on the cluster nodes, use clusterCall. It runs a function on every node; here, we use it to load the package on all of them.

cl <- makeCluster(4) # Create the cluster
clusterCall(cl, function() library(randomForest))
system.time(
  results <- parLapply(cl, iter, rf_boot_fun, data = datos_pc, n_sample = 100)
)
##    user  system elapsed
##    0.02    0.00  137.75
stopCluster(cl) # Close the cluster; do not forget this step

A more realistic example: machine learning

Now for a more realistic example: model tuning. This usually means choosing the best value of one or more parameters according to a performance statistic, such as R2 or RMSE. Tuning uses the training or calibration dataset, while the model is evaluated on validation data.

We will use iris as our training dataset and assume we have only sepal measurements, to make the task a little harder.

We will use a Support Vector Machine (SVM) to predict species from flower measurements. This model works well for classification, but can depend heavily on the cost and gamma parameters when using a radial kernel. We will try to optimise both.

The e1071 package includes a tune function that uses cross-validation. Here we will use 10-fold cross-validation with three repetitions. This example comes from a real project with a large data matrix, where each model run took about an hour and a half. Without parallelisation, tuning would have taken days.

Enough talking—let us look at the model!

library(e1071)
svm_model <- svm(Species ~ Sepal.Length + Sepal.Width, data = iris)
summary(svm_model)
##
## Call:
## svm(formula = Species ~ Sepal.Length + Sepal.Width, data = iris)
##
##
## Parameters:
##    SVM-Type:  C-classification
##  SVM-Kernel:  radial
##        cost:  1
##       gamma:  0.5
##
## Number of Support Vectors:  86
##
##  ( 10 40 36 )
##
##
## Number of Classes:  3
##
## Levels:
##  setosa versicolor virginica

The default cost is 1 and gamma is 0.5. Let us examine the performance.

library(caret)
confusionMatrix(iris$Species, predict(svm_model))
## Confusion Matrix and Statistics
##
##             Reference
## Prediction   setosa versicolor virginica
##   setosa         49          1         0
##   versicolor      0         36        14
##   virginica       0         12        38
##
## Overall Statistics
##
##                Accuracy : 0.82
##                  95% CI : (0.749, 0.8779)
##     No Information Rate : 0.3467
##     P-Value [Acc > NIR] : < 2.2e-16
##
##                   Kappa : 0.73
##  Mcnemar's Test P-Value : NA
##
## Statistics by Class:
##
##                      Class: setosa Class: versicolor Class: virginica
## Sensitivity                 1.0000            0.7347           0.7308
## Specificity                 0.9901            0.8614           0.8776
## Pos Pred Value              0.9800            0.7200           0.7600
## Neg Pred Value              1.0000            0.8700           0.8600
## Prevalence                  0.3267            0.3267           0.3467
## Detection Rate              0.3267            0.2400           0.2533
## Detection Prevalence        0.3333            0.3333           0.3333
## Balanced Accuracy           0.9950            0.7980           0.8042

Accuracy is 0.82. In many cases that is not enough, so let us try to improve it by tuning the parameters.

We create a function that returns model performance for a given cost and gamma. Notice that I call tune with only one parameter combination at a time. It could accept a list of parameters, but that would be harder to parallelise.

tune_svm_par <- function(costo, gamma) {
  control <- tune.control(
    nrepeat = 3, repeat.aggregate = mean,
    sampling = "cross", sampling.aggregate = mean,
    sampling.dispersion = sd,
    cross = 10
  )

svm_cv <- tune("svm", train.x = iris[, c("Sepal.Length", "Sepal.Width")], train.y = iris$Species, kernel = "radial", scale = TRUE, type = "C-classification", tunecontrol = control, ranges = list(cost = costo, gamma = gamma) )

return(svm_cv$performances) }

For parallelisation, we will use clusterMap, which accepts more than one varying argument. It is the equivalent of mapply in the apply family.

costos.test <- seq(0.2, 4, by = 0.1)
gammas.test <- seq(0.2, 4, by = 0.1)

grilla <- expand.grid("costos" = costos.test, "gammas" = gammas.test) nrow(grilla)

## [1] 1521

There are 1,521 combinations. This should take a while.

t1 <- Sys.time()

cl <- makeCluster(4) # Create the cluster clusterCall(cl, function() library(e1071))

list.errores <- clusterMap( cl = cl, fun = tune_svm_par, costo = grilla$costos, gamma = grilla$gammas, SIMPLIFY = FALSE, RECYCLE = FALSE )

library(dplyr) errores.bind <- bind_rows(list.errores)

stopCluster(cl)

t2 <- Sys.time()

print(t2 - t1)
## Time difference of 52.99884 secs

That hardly took any time! Now let us find the cost and gamma combination with the lowest error.

knitr::kable(subset(errores.bind, errores.bind$error == min(errores.bind$error)))
costgammaerrordispersion
8852.82.40.180.1044681

Let us test the SVM with those parameters.

library(e1071)
svm_model <- svm(Species ~ Sepal.Length + Sepal.Width, data = iris, cost = 2.6, gamma = 2)
library(caret)
confusionMatrix(iris$Species, predict(svm_model))
## Confusion Matrix and Statistics
##
##             Reference
## Prediction   setosa versicolor virginica
##   setosa         50          0         0
##   versicolor      0         36        14
##   virginica       0         10        40
##
## Overall Statistics
##
##                Accuracy : 0.84
##                  95% CI : (0.7714, 0.8947)
##     No Information Rate : 0.36
##     P-Value [Acc > NIR] : < 2.2e-16
##
##                   Kappa : 0.76
##  Mcnemar's Test P-Value : NA
##
## Statistics by Class:
##
##                      Class: setosa Class: versicolor Class: virginica
## Sensitivity                 1.0000            0.7826           0.7407
## Specificity                 1.0000            0.8654           0.8958
## Pos Pred Value              1.0000            0.7200           0.8000
## Neg Pred Value              1.0000            0.9000           0.8600
## Prevalence                  0.3333            0.3067           0.3600
## Detection Rate              0.3333            0.2400           0.2667
## Detection Prevalence        0.3333            0.3333           0.3333
## Balanced Accuracy           1.0000            0.8240           0.8183

Training accuracy increased from 0.82 to 0.84. It may not look like much, but a two-percentage-point improvement from a procedure that took less than a minute—thanks to parallelisation—is a good result.

Thank you for reading to the end of this tutorial! I welcome your comments and questions.

Julián Cabezas Peña
Authors
AI and the environment

I am a data scientist and Natural Resources Engineer from Universidad de Chile, with a Master of Data Science from the University of Adelaide. I combine artificial intelligence, data science, statistical analysis and geospatial analysis to tackle environmental challenges.

Since February 2022, I have coordinated the Data Science team at Chile’s Superintendence of the Environment (SMA). I develop automation, data validation and analytical solutions for environmental monitoring and enforcement, using R, Python and SQL.

My experience also includes computer vision and deep learning for agricultural robotics, forest carbon modelling, remote sensing and survey data analysis. I have worked in Chile, Australia and Germany, combining applied research, algorithm development and interdisciplinary team coordination.