<?xml version="1.0" encoding="utf-8" standalone="yes"?><rss version="2.0" xmlns:atom="http://www.w3.org/2005/Atom"><channel><title>Servers | Julián Cabezas Peña</title><link>https://www.juliancabezas.com/en/tags/servers/</link><atom:link href="https://www.juliancabezas.com/en/tags/servers/index.xml" rel="self" type="application/rss+xml"/><description>Servers</description><generator>HugoBlox Kit (https://hugoblox.com)</generator><language>en-US</language><lastBuildDate>Thu, 24 Jan 2019 00:00:00 +0000</lastBuildDate><image><url>https://www.juliancabezas.com/media/icon_hu_a22537ec31d66339.png</url><title>Servers</title><link>https://www.juliancabezas.com/en/tags/servers/</link></image><item><title>A function for parallel processing on Windows and servers (R parallelisation, part 2)</title><link>https://www.juliancabezas.com/en/post/parallel-cluster-function/</link><pubDate>Thu, 24 Jan 2019 00:00:00 +0000</pubDate><guid>https://www.juliancabezas.com/en/post/parallel-cluster-function/</guid><description>&lt;div id="TOC"&gt;
&lt;ul&gt;
&lt;li&gt;&lt;a href="#el-problema-crear-clusters-es-tedioso"&gt;The problem: creating clusters is tedious&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href="#la-solucion-una-funcion-para-clear-clusters"&gt;The solution: a function to create clusters&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href="#un-pequeno-y-reciclado-ejemplo"&gt;A small, recycled example&lt;/a&gt;&lt;/li&gt;
&lt;/ul&gt;
&lt;/div&gt;
&lt;p&gt;This short tutorial is the second part of my introduction to parallelising R on Windows. &lt;a href="https://www.juliancabezas.com/en/post/parallel-r-windows/"&gt;I recommend reading the first part&lt;/a&gt;.&lt;/p&gt;
&lt;div id="el-problema-crear-clusters-es-tedioso" class="section level2"&gt;
&lt;h2&gt;The problem: creating clusters is tedious&lt;/h2&gt;
&lt;p&gt;Creating clusters can be tedious: loading packages one by one and exporting datasets is slow, and it is easy to forget a crucial package. We want a more convenient approach. We may also be working on a server without Internet access, which complicates package installation.&lt;/p&gt;
&lt;/div&gt;
&lt;div id="la-solucion-una-funcion-para-clear-clusters" class="section level2"&gt;
&lt;h2&gt;The solution: a function to create clusters&lt;/h2&gt;
&lt;p&gt;I created a function, available in &lt;a href="https://github.com/juliancabezas/random_functions/blob/master/create_cluster"&gt;this GitHub file&lt;/a&gt;. To give credit where it is due, it is inspired by &lt;a href="https://github.com/nathanvan/parallelsugar"&gt;parallelsugar&lt;/a&gt;, with a few modifications.&lt;/p&gt;
&lt;p&gt;The function creates a cluster with &lt;em&gt;n&lt;/em&gt; nodes, specified by the &lt;em&gt;nnodes&lt;/em&gt; argument.&lt;/p&gt;
&lt;p&gt;create_cluster is designed to work on servers. When they have no Internet connection, packages can be installed in a shared folder and loaded from there. Use &lt;em&gt;libroute&lt;/em&gt; to specify the library path.&lt;/p&gt;
&lt;pre class="r"&gt;&lt;code&gt;create_cluster &amp;lt;- function(nnodes, libroute = NULL) {
&lt;h1 id="nnodes-number-of-nodes"&gt;nnodes: Number of nodes&lt;/h1&gt;
&lt;h1 id="libroute-provided-library-route-different-of-the-default-library"&gt;libroute: Provided library route (different of the default library)&lt;/h1&gt;
&lt;p&gt;tryCatch({
cl &amp;lt;- parallel::makeCluster(nnodes) # Create an empty cluster with n nodes&lt;/p&gt;
&lt;pre&gt;&lt;code&gt;loaded.package.names &amp;amp;lt;- c( # Find the loaded packages in the session
sessionInfo()$basePkgs, # base packages
names(sessionInfo()$otherPkgs)
) # aditional loaded packages
# Send the packages and the complete environment to the cluster
parallel::clusterExport(cl,
&amp;amp;quot;loaded.package.names&amp;amp;quot;, # loaded packages
envir = environment()
) # &amp;amp;quot;Environment&amp;amp;quot;
## Charge the packages in all nodes
if (is.null(libroute)) { # Use default library location
parallel::parLapply(cl, 1:length(cl), function(xx) {
lapply(loaded.package.names, function(yy) {
library(yy, character.only = TRUE)
})
})
} else {
# If we provide a library location use it (useful when working in remote machines)
parallel::parLapply(cl, 1:length(cl), function(xx) {
lapply(loaded.package.names, function(yy) {
library(yy, character.only = TRUE, lib.loc = libroute)
})
})
}
&lt;/code&gt;&lt;/pre&gt;
&lt;p&gt;})
return(cl) # The result is the cluster
}&lt;/code&gt;&lt;/pre&gt;&lt;/p&gt;
&lt;/div&gt;
&lt;div id="un-pequeno-y-reciclado-ejemplo" class="section level2"&gt;
&lt;h2&gt;A small, recycled example&lt;/h2&gt;
&lt;p&gt;In the example from the previous tutorial, we used &lt;em&gt;clusterCall&lt;/em&gt; to load &lt;em&gt;randomForest&lt;/em&gt;. Here, if the package and any other dependencies are already loaded in our session, we can export them to the nodes in one step.&lt;/p&gt;
&lt;p&gt;First, load the package and create the function:&lt;/p&gt;
&lt;pre class="r"&gt;&lt;code&gt;library(randomForest)
&lt;p&gt;rf_boot_fun &amp;lt;- function(iter, data, n_sample) {
ind &amp;lt;- sample(1:nrow(data), size = n_sample)
data_sample &amp;lt;- data[ind, ]&lt;/p&gt;
&lt;h1 id="fit-the-random-forest-model"&gt;Fit the Random Forest model&lt;/h1&gt;
&lt;p&gt;model1 &amp;lt;- randomForest(Sepal.Length ~ Petal.Length, data = data_sample)&lt;/p&gt;
&lt;h1 id="extract-the-pseudo-r2"&gt;Extract the pseudo-R2&lt;/h1&gt;
&lt;p&gt;R2 &amp;lt;- model1$rsq[length(model1$rsq)]&lt;/p&gt;
&lt;h1 id="extract-the-mse"&gt;Extract the MSE&lt;/h1&gt;
&lt;p&gt;MSE &amp;lt;- model1$mse[length(model1$mse)]
results &amp;lt;- c(&amp;quot;R2&amp;quot; = R2, &amp;quot;MSE&amp;quot; = MSE)
return(results)
}&lt;/code&gt;&lt;/pre&gt;&lt;/p&gt;
&lt;p&gt;Now I can apply the function with &lt;em&gt;parLapply&lt;/em&gt;.&lt;/p&gt;
&lt;pre class="r"&gt;&lt;code&gt;library(parallel)
&lt;h1 id="define-a-library-path-optional"&gt;Define a library path (optional)&lt;/h1&gt;
&lt;p&gt;ruta_libreria &amp;lt;- &amp;quot;C:/Users/Julian/Documents/R/win-library/3.5&amp;quot;&lt;/p&gt;
&lt;h1 id="this-can-simply-be-nuevoclus-create_cluster4"&gt;This can simply be nuevoclus&amp;lt;-create_cluster(4)&lt;/h1&gt;
&lt;p&gt;nuevoclus &amp;lt;- create_cluster(4, ruta_libreria)&lt;/p&gt;
&lt;h1 id="10000-iterations-of-our-function-using-the-iris-dataset"&gt;10,000 iterations of our function using the &amp;quot;iris&amp;quot; dataset&lt;/h1&gt;
&lt;p&gt;results &amp;lt;- parLapply(nuevoclus, 1:10000, rf_boot_fun, data = iris, n_sample = 100)&lt;/p&gt;
&lt;p&gt;stopCluster(nuevoclus) # Close the cluster; do not forget this step&lt;/p&gt;
&lt;h1 id="the-first-two-results"&gt;The first two results&lt;/h1&gt;
&lt;p&gt;head(results, n = 2)&lt;/code&gt;&lt;/pre&gt;&lt;/p&gt;
&lt;pre&gt;&lt;code&gt;## [[1]]
## R2 MSE
## 0.778244 0.174310
##
## [[2]]
## R2 MSE
## 0.6912495 0.1865890&lt;/code&gt;&lt;/pre&gt;
&lt;p&gt;Done—parallelisation is much easier now!&lt;/p&gt;
&lt;div class="figure"&gt;
&lt;img src="https://media.giphy.com/media/uTuLngvL9p0Xe/giphy.gif" /&gt;
&lt;/div&gt;
&lt;p&gt;&lt;strong&gt;I hope you find this useful!&lt;/strong&gt;&lt;/p&gt;
&lt;p&gt;&lt;strong&gt;I welcome your comments and questions!&lt;/strong&gt;&lt;/p&gt;
&lt;/div&gt;</description></item></channel></rss>