<?xml version="1.0" encoding="utf-8" standalone="yes"?><rss version="2.0" xmlns:atom="http://www.w3.org/2005/Atom"><channel><title>Tutorials | Julián Cabezas Peña</title><link>https://www.juliancabezas.com/en/tags/tutorials/</link><atom:link href="https://www.juliancabezas.com/en/tags/tutorials/index.xml" rel="self" type="application/rss+xml"/><description>Tutorials</description><generator>HugoBlox Kit (https://hugoblox.com)</generator><language>en-US</language><lastBuildDate>Mon, 01 Apr 2019 00:00:00 +0000</lastBuildDate><image><url>https://www.juliancabezas.com/media/icon_hu_a22537ec31d66339.png</url><title>Tutorials</title><link>https://www.juliancabezas.com/en/tags/tutorials/</link></image><item><title>Working with CSV or TXT datasets larger than RAM in R using SQLite</title><link>https://www.juliancabezas.com/en/post/large-datasets-r-sqlite/</link><pubDate>Mon, 01 Apr 2019 00:00:00 +0000</pubDate><guid>https://www.juliancabezas.com/en/post/large-datasets-r-sqlite/</guid><description>&lt;div id="TOC"&gt;
&lt;ul&gt;
&lt;li&gt;&lt;a href="#el-problema-del-maldito-ram-y-las-bases-de-datos-enormes"&gt;The RAM problem and enormous datasets&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href="#primero-lo-primero-cuanta-memoria-tengo-disponible"&gt;First things first: how much memory is available?&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href="#csv-o-txt-a-sqllite"&gt;CSV or TXT to SQLite&lt;/a&gt;&lt;ul&gt;
&lt;li&gt;&lt;a href="#los-benditos-paquetes"&gt;The packages you need&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href="#que-tan-grande-es-mi-base-de-datos-cuales-son-sus-caracteristicas"&gt;How large is the dataset, and what does it look like?&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href="#crear-la-ruta-a-la-base-de-datos"&gt;Create the database path&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href="#la-prometida-funcion"&gt;The promised function&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href="#mover-la-csv-o-txt-a-la-base-de-datos-sqllite"&gt;Move the CSV or TXT file into SQLite&lt;/a&gt;&lt;/li&gt;
&lt;/ul&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href="#analizar-y-recoger-datos-con-dplyr"&gt;Analyse and retrieve data with dplyr&lt;/a&gt;&lt;/li&gt;
&lt;/ul&gt;
&lt;/div&gt;
&lt;div id="el-problema-del-maldito-ram-y-las-bases-de-datos-enormes" class="section level3"&gt;
&lt;h3&gt;The RAM problem and enormous datasets&lt;/h3&gt;
&lt;p&gt;I do not know whether this happens to everyone, but whenever I start a new job, I seem to get &lt;strong&gt;the worst computer in the office&lt;/strong&gt;: usually no more than 4 GB of RAM, and even 32-bit Windows if I am unlucky, as happened once. Since I use R for everything, large datasets can be difficult to handle, especially when they exceed the memory available to R. This tutorial also helps with &lt;strong&gt;enormous&lt;/strong&gt; datasets.&lt;/p&gt;
&lt;p&gt;We will check how much RAM is available to R and create a SQLite database. We can then retrieve manageable subsets and analyse them in memory, using a function I adapted from &lt;a href="https://gist.github.com/vnijs/e53a68c957747e82c2e3"&gt;this gist by vnijs&lt;/a&gt;.&lt;/p&gt;
&lt;p&gt;We will use &lt;a href="https://www.sqlite.org/index.html"&gt;SQLite&lt;/a&gt; because it is easy to configure and run locally. PostgreSQL and MySQL are alternatives, but setting them up would be a little more involved.&lt;/p&gt;
&lt;p&gt;This tutorial assumes intermediate-to-advanced R skills and some familiarity with SQL.&lt;/p&gt;
&lt;/div&gt;
&lt;div id="primero-lo-primero-cuanta-memoria-tengo-disponible" class="section level3"&gt;
&lt;h3&gt;First things first: how much memory is available?&lt;/h3&gt;
&lt;p&gt;To check the memory available to R, we will use the memory limit command shown below.&lt;/p&gt;
&lt;pre class="r"&gt;&lt;code&gt;memory.limit()&lt;/code&gt;&lt;/pre&gt;
&lt;pre&gt;&lt;code&gt;## [1] 16269&lt;/code&gt;&lt;/pre&gt;
&lt;p&gt;I have approximately 16 GB of memory available, or about 16,000 MB—the RAM installed in this computer.&lt;/p&gt;
&lt;p&gt;&lt;strong&gt;Note:&lt;/strong&gt; On a 32-bit operating system, the memory limit cannot exceed roughly 2–3 GB, regardless of how much RAM is installed.&lt;/p&gt;
&lt;/div&gt;
&lt;div id="csv-o-txt-a-sqllite" class="section level3"&gt;
&lt;h3&gt;CSV or TXT to SQLite&lt;/h3&gt;
&lt;p&gt;Suppose we have a &lt;strong&gt;HUGE&lt;/strong&gt; CSV or TXT dataset. I will use Chile’s 2017 Population and Housing Census. It is not exceptionally large, but works well for this example. You can download it from &lt;a href="http://www.ine.cl/docs/default-source/censos/censo-2017/base-de-datos/csv/microdatos-persona/microdato_censo2017-personas.rar?sfvrsn=3"&gt;Chile’s National Statistics Institute&lt;/a&gt;.&lt;/p&gt;
&lt;p&gt;Extract the RAR archive to obtain the CSV file. It is approximately 2.5 GB—not enormous, but potentially too large for R on a low-spec computer. Opening it in Excel can also be difficult.&lt;/p&gt;
&lt;p&gt;I will extract the dataset into a folder called “Censo” on the C:/ drive.&lt;/p&gt;
&lt;div class="figure"&gt;
&lt;img src="https://www.juliancabezas.com/img/archivo_censo.JPG" /&gt;
&lt;/div&gt;
&lt;div id="los-benditos-paquetes" class="section level4"&gt;
&lt;h4&gt;The packages you need&lt;/h4&gt;
&lt;p&gt;Make sure these five packages are installed:&lt;/p&gt;
&lt;ul&gt;
&lt;li&gt;&lt;strong&gt;readr&lt;/strong&gt;: to read comma-, tab- or otherwise-delimited files.&lt;/li&gt;
&lt;li&gt;&lt;strong&gt;DBI and RSQLite&lt;/strong&gt;: to communicate with databases and use SQLite.&lt;/li&gt;
&lt;li&gt;&lt;strong&gt;dplyr and dbplyr&lt;/strong&gt;: to retrieve information from the database.&lt;/li&gt;
&lt;/ul&gt;
&lt;pre class="r"&gt;&lt;code&gt;library(readr)
library(DBI)
library(RSQLite)
library(dplyr)
library(dbplyr)&lt;/code&gt;&lt;/pre&gt;
&lt;/div&gt;
&lt;div id="que-tan-grande-es-mi-base-de-datos-cuales-son-sus-caracteristicas" class="section level4"&gt;
&lt;h4&gt;How large is the dataset, and what does it look like?&lt;/h4&gt;
&lt;p&gt;First, define the dataset path and assign it to delim_file:&lt;/p&gt;
&lt;pre class="r"&gt;&lt;code&gt;delim_file &amp;lt;- &amp;quot;C:/Censo/Microdato_Censo2017-Personas.csv&amp;quot;&lt;/code&gt;&lt;/pre&gt;
&lt;p&gt;To inspect the file, read its first two lines using &lt;strong&gt;readLines&lt;/strong&gt;.&lt;/p&gt;
&lt;pre class="r"&gt;&lt;code&gt;readLines(delim_file, n=2)&lt;/code&gt;&lt;/pre&gt;
&lt;pre&gt;&lt;code&gt;## [1] &amp;quot;REGION;PROVINCIA;COMUNA;DC;AREA;ZC_LOC;ID_ZONA_LOC;NVIV;NHOGAR;PERSONAN;P07;P08;P09;P10;P10COMUNA;P10PAIS;P11;P11COMUNA;P11PAIS;P12;P12COMUNA;P12PAIS;P12A_LLEGADA;P12A_TRAMO;P13;P14;P15;P15A;P16;P16A;P16A_OTRO;P17;P18;P19;P20;P21M;P21A;P10PAIS_GRUPO;P11PAIS_GRUPO;P12PAIS_GRUPO;ESCOLARIDAD;P16A_GRUPO;REGION_15R;PROVINCIA_15R;COMUNA_15R;P10COMUNA_15R;P11COMUNA_15R;P12COMUNA_15R&amp;quot;
## [2] &amp;quot;15;152;15202;1;2;6;13225;1;1;1;1;1;73;1;98;998;3;15101;998;1;98;998;9998;98;2;4;6;2;1;2;98;7;98;98;98;98;9998;998;998;998;4;2;15;152;15202;98;15101;98&amp;quot;&lt;/code&gt;&lt;/pre&gt;
&lt;p&gt;We can now see the column names and identify the delimiter: a semicolon (;).&lt;/p&gt;
&lt;p&gt;R cannot count rows by loading a file that does not fit in memory. Instead, we will use a PowerShell command (Get-Content) on Windows, or a system command (wc) on Linux.&lt;/p&gt;
&lt;p&gt;To count the rows in the file—this takes a little while:&lt;/p&gt;
&lt;p&gt;On Windows:&lt;/p&gt;
&lt;pre class="r"&gt;&lt;code&gt;total_records &amp;lt;- system2(&amp;quot;powershell&amp;quot;, args = c(&amp;quot;Get-content&amp;quot;, delim_file, &amp;quot;|&amp;quot;, &amp;quot;Measure-Object&amp;quot;, &amp;quot;–Line&amp;quot;),
stdout = TRUE)
total_records &amp;lt;- as.numeric(gsub(&amp;quot;[^0-9]&amp;quot;, &amp;quot;&amp;quot;, paste(total_records, collapse = &amp;quot;&amp;quot;))) - 1
total_records&lt;/code&gt;&lt;/pre&gt;
&lt;pre&gt;&lt;code&gt;## [1] 17574003&lt;/code&gt;&lt;/pre&gt;
&lt;p&gt;There are 17,574,003 records, excluding the header. That matches the number of people recorded by the 2017 census.&lt;/p&gt;
&lt;p&gt;Note: Windows 10 should work without problems. On Windows 7 or 8, check your PowerShell version using &lt;a href="https://docs.microsoft.com/en-us/skypeforbusiness/set-up-your-computer-for-windows-powershell/download-and-install-windows-powershell-5-1"&gt;these Microsoft instructions&lt;/a&gt;, and install PowerShell 5.1 by following &lt;a href="https://docs.microsoft.com/en-us/powershell/scripting/install/installing-windows-powershell?view=powershell-6"&gt;this Microsoft guide&lt;/a&gt;.&lt;/p&gt;
&lt;p&gt;On Linux, it would look like this:&lt;/p&gt;
&lt;pre class="r"&gt;&lt;code&gt; total_records &amp;lt;- system2(&amp;quot;wc&amp;quot;, args = c(&amp;quot;-l&amp;quot;, delim_file), stdout = TRUE) %&amp;gt;%
sub(normalizePath(delim_file), &amp;quot;&amp;quot;, .) %&amp;gt;%
as.integer() %&amp;gt;%
{
. - 1
}&lt;/code&gt;&lt;/pre&gt;
&lt;/div&gt;
&lt;div id="crear-la-ruta-a-la-base-de-datos" class="section level4"&gt;
&lt;h4&gt;Create the database path&lt;/h4&gt;
&lt;p&gt;Choose a path for the future database file, with the extension .sqlite3.&lt;/p&gt;
&lt;p&gt;Define the database path and the table name. We will need the table name to retrieve data later. Another advantage of SQLite databases is that they can contain several tables.&lt;/p&gt;
&lt;pre class="r"&gt;&lt;code&gt;table_name_censo &amp;lt;- &amp;quot;censo2017&amp;quot;
sqlite_file_censo &amp;lt;- &amp;quot;C:/Censo/censo2017.sqlite3&amp;quot;&lt;/code&gt;&lt;/pre&gt;
&lt;/div&gt;
&lt;div id="la-prometida-funcion" class="section level4"&gt;
&lt;h4&gt;The promised function&lt;/h4&gt;
&lt;p&gt;With the total row count, file paths and delimiter (comma, semicolon, space, and so on) identified, we can use the following function:&lt;/p&gt;
&lt;pre class="r"&gt;&lt;code&gt;read_delim2sqlite &amp;lt;- function(delim_file, delim, sqlite_file, table_name, batch_size = 10000, OS = &amp;quot;Windows&amp;quot;) {
&lt;h2 id="establish-a-connection-to-the-database"&gt;establish a connection to the database&lt;/h2&gt;
&lt;p&gt;condb &amp;lt;- dbConnect(SQLite(), sqlite_file)&lt;/p&gt;
&lt;h2 id="get-the-total-number-of-records-in-the-file"&gt;get the total number of records in the file&lt;/h2&gt;
&lt;h1 id="in-unix"&gt;in Unix&lt;/h1&gt;
&lt;p&gt;if (OS == &amp;quot;Unix&amp;quot;) {
total_records &amp;lt;- system2(&amp;quot;wc&amp;quot;, args = c(&amp;quot;-l&amp;quot;, delim_file), stdout = TRUE) %&amp;gt;%
sub(normalizePath(delim_file), &amp;quot;&amp;quot;, .) %&amp;gt;%
as.integer() %&amp;gt;%
{
. - 1
}
} else {
# In windows
total_records &amp;lt;- system2(&amp;quot;powershell&amp;quot;, args = c(&amp;quot;Get-content&amp;quot;, delim_file, &amp;quot;|&amp;quot;, &amp;quot;Measure-Object&amp;quot;, &amp;quot;–Line&amp;quot;), stdout = TRUE)
total_records &amp;lt;- as.numeric(gsub(&amp;quot;[^0-9]&amp;quot;, &amp;quot;&amp;quot;, paste(total_records, collapse = &amp;quot;&amp;quot;))) - 1
}&lt;/p&gt;
&lt;p&gt;message(&amp;quot;Total records: &amp;quot;, total_records)&lt;/p&gt;
&lt;h2 id="find-the-number-of-passes-needed-based-on-size-of-each-batch"&gt;find the number of passes needed based on size of each batch&lt;/h2&gt;
&lt;p&gt;passes &amp;lt;- total_records %/% batch_size
remaining &amp;lt;- total_records %% batch_size&lt;/p&gt;
&lt;p&gt;message(&amp;quot;Total Passes to complete: &amp;quot;, passes)&lt;/p&gt;
&lt;h2 id="first-pass-determines-header-and-column-types"&gt;first pass determines header and column types&lt;/h2&gt;
&lt;p&gt;dat &amp;lt;- read_delim(delim_file, delim, n_max = batch_size, progress = FALSE) %&amp;gt;% as.data.frame()
if (nrow(problems(dat)) &amp;gt; 0) print(problems(dat))
col_names &amp;lt;- colnames(dat)
col_types &amp;lt;- c(character = &amp;quot;c&amp;quot;, numeric = &amp;quot;d&amp;quot;, integer = &amp;quot;i&amp;quot;, logical = &amp;quot;l&amp;quot;, Date = &amp;quot;c&amp;quot;) %&amp;gt;%
.[sapply(dat, class)] %&amp;gt;%
paste0(collapse = &amp;quot;&amp;quot;)&lt;/p&gt;
&lt;h2 id="write-to-database-table"&gt;write to database table&lt;/h2&gt;
&lt;p&gt;dbWriteTable(condb, table_name, dat, overwrite = TRUE)&lt;/p&gt;
&lt;h2 id="multiple-passes"&gt;multiple passes&lt;/h2&gt;
&lt;p&gt;for (p in 2:passes) {
message(&amp;quot;Pass number: &amp;quot;, p, &amp;quot;, Progress:&amp;quot;, round(p / passes, 2) * 100, &amp;quot;%&amp;quot;)
read_delim(delim_file, delim,
col_names = col_names, col_types = col_types,
skip = (p - 1) * batch_size + 1, n_max = batch_size, progress = FALSE
) %&amp;gt;%
as.data.frame() %&amp;gt;%
dbWriteTable(condb, table_name, ., append = TRUE)
}&lt;/p&gt;
&lt;p&gt;if (remaining) {
read_delim(delim_file, delim,
col_names = col_names, col_types = col_types,
skip = p * batch_size + 1, n_max = remaining, progress = FALSE
) %&amp;gt;%
as.data.frame() %&amp;gt;%
dbWriteTable(condb, table_name, ., append = TRUE)
}&lt;/p&gt;
&lt;h2 id="close-the-database-connection"&gt;close the database connection&lt;/h2&gt;
&lt;p&gt;dbDisconnect(condb)
}&lt;/code&gt;&lt;/pre&gt;&lt;/p&gt;
&lt;p&gt;The function may look intimidating, but it is fairly straightforward. It reads the file in chunks of batch_size rows using read_delim from readr, inserts each chunk into the SQLite database, and repeats the process through the final line.&lt;/p&gt;
&lt;p&gt;The function arguments are:&lt;/p&gt;
&lt;p&gt;&lt;strong&gt;delim_file&lt;/strong&gt;: path to the delimited file, in this case the census dataset. &lt;strong&gt;delim&lt;/strong&gt;: delimiter character, a semicolon here. &lt;strong&gt;sqlite_file&lt;/strong&gt;: path to the SQLite database, ending in “.sqlite3”. &lt;strong&gt;table_name&lt;/strong&gt;: name of the table within the SQLite database. &lt;strong&gt;batch_size&lt;/strong&gt;: number of records to read in each iteration. &lt;strong&gt;OS&lt;/strong&gt;: the operating system, either “Windows” or “Unix”; the Unix option was tested on Linux, not macOS.&lt;/p&gt;
&lt;/div&gt;
&lt;div id="mover-la-csv-o-txt-a-la-base-de-datos-sqllite" class="section level4"&gt;
&lt;h4&gt;Move the CSV or TXT file into SQLite&lt;/h4&gt;
&lt;p&gt;We are ready to use the function. To avoid running out of RAM, I will transfer the dataset in batches of one million rows. It should finish in 18 iterations.&lt;/p&gt;
&lt;p&gt;&lt;strong&gt;This will take a few minutes—time to make some mate!&lt;/strong&gt;&lt;/p&gt;
&lt;pre class="r"&gt;&lt;code&gt;read_delim2sqlite(delim_file, delim=&amp;quot;;&amp;quot;, sqlite_file = sqlite_file_censo,
table_name = table_name_censo, batch_size=1000000)&lt;/code&gt;&lt;/pre&gt;
&lt;pre&gt;&lt;code&gt;## Total records: 17574003&lt;/code&gt;&lt;/pre&gt;
&lt;pre&gt;&lt;code&gt;## Total Passes to complete: 17&lt;/code&gt;&lt;/pre&gt;
&lt;pre&gt;&lt;code&gt;## Parsed with column specification:
## cols(
## .default = col_integer(),
## P18 = col_character()
## )&lt;/code&gt;&lt;/pre&gt;
&lt;pre&gt;&lt;code&gt;## See spec(...) for full column specifications.&lt;/code&gt;&lt;/pre&gt;
&lt;pre&gt;&lt;code&gt;## Pass number: 2, Progress:12%&lt;/code&gt;&lt;/pre&gt;
&lt;pre&gt;&lt;code&gt;## Pass number: 3, Progress:18%&lt;/code&gt;&lt;/pre&gt;
&lt;pre&gt;&lt;code&gt;## Pass number: 4, Progress:24%&lt;/code&gt;&lt;/pre&gt;
&lt;pre&gt;&lt;code&gt;## Pass number: 5, Progress:29%&lt;/code&gt;&lt;/pre&gt;
&lt;pre&gt;&lt;code&gt;## Pass number: 6, Progress:35%&lt;/code&gt;&lt;/pre&gt;
&lt;pre&gt;&lt;code&gt;## Pass number: 7, Progress:41%&lt;/code&gt;&lt;/pre&gt;
&lt;pre&gt;&lt;code&gt;## Pass number: 8, Progress:47%&lt;/code&gt;&lt;/pre&gt;
&lt;pre&gt;&lt;code&gt;## Pass number: 9, Progress:53%&lt;/code&gt;&lt;/pre&gt;
&lt;pre&gt;&lt;code&gt;## Pass number: 10, Progress:59%&lt;/code&gt;&lt;/pre&gt;
&lt;pre&gt;&lt;code&gt;## Pass number: 11, Progress:65%&lt;/code&gt;&lt;/pre&gt;
&lt;pre&gt;&lt;code&gt;## Pass number: 12, Progress:71%&lt;/code&gt;&lt;/pre&gt;
&lt;pre&gt;&lt;code&gt;## Pass number: 13, Progress:76%&lt;/code&gt;&lt;/pre&gt;
&lt;pre&gt;&lt;code&gt;## Pass number: 14, Progress:82%&lt;/code&gt;&lt;/pre&gt;
&lt;pre&gt;&lt;code&gt;## Pass number: 15, Progress:88%&lt;/code&gt;&lt;/pre&gt;
&lt;pre&gt;&lt;code&gt;## Pass number: 16, Progress:94%&lt;/code&gt;&lt;/pre&gt;
&lt;pre&gt;&lt;code&gt;## Pass number: 17, Progress:100%&lt;/code&gt;&lt;/pre&gt;
&lt;p&gt;We only need to run this lengthy conversion once. It creates a .sqlite3 file that is slightly smaller than the CSV.&lt;/p&gt;
&lt;div class="figure"&gt;
&lt;img src="https://www.juliancabezas.com/img/archivo_censo_sqlite.JPG" alt="SQLite file created" /&gt;
&lt;p class="caption"&gt;SQLite file created&lt;/p&gt;
&lt;/div&gt;
&lt;/div&gt;
&lt;/div&gt;
&lt;div id="analizar-y-recoger-datos-con-dplyr" class="section level3"&gt;
&lt;h3&gt;Analyse and retrieve data with dplyr&lt;/h3&gt;
&lt;p&gt;To start retrieving data from the table, first connect to the database.&lt;/p&gt;
&lt;pre class="r"&gt;&lt;code&gt;con &amp;lt;- dbConnect(SQLite(), sqlite_file_censo)
con&lt;/code&gt;&lt;/pre&gt;
&lt;pre&gt;&lt;code&gt;## &amp;lt;SQLiteConnection&amp;gt;
## Path: C:\Censo\censo2017.sqlite3
## Extensions: TRUE&lt;/code&gt;&lt;/pre&gt;
&lt;p&gt;We can inspect the connection details and the table name assigned to the dataset.&lt;/p&gt;
&lt;pre class="r"&gt;&lt;code&gt;dbListTables(con)&lt;/code&gt;&lt;/pre&gt;
&lt;pre&gt;&lt;code&gt;## [1] &amp;quot;censo2017&amp;quot;&lt;/code&gt;&lt;/pre&gt;
&lt;p&gt;Create an object representing the data. It does not load the dataset into RAM: it is only a connection, which is why the total row count is not shown.&lt;/p&gt;
&lt;pre class="r"&gt;&lt;code&gt;datos_db &amp;lt;- tbl(con, &amp;quot;censo2017&amp;quot;)
datos_db&lt;/code&gt;&lt;/pre&gt;
&lt;pre&gt;&lt;code&gt;## # Source: table&amp;lt;censo2017&amp;gt; [?? x 48]
## # Database: sqlite 3.22.0 [C:\Censo\censo2017.sqlite3]
## REGION PROVINCIA COMUNA DC AREA ZC_LOC ID_ZONA_LOC NVIV NHOGAR
## &amp;lt;int&amp;gt; &amp;lt;int&amp;gt; &amp;lt;int&amp;gt; &amp;lt;int&amp;gt; &amp;lt;int&amp;gt; &amp;lt;int&amp;gt; &amp;lt;int&amp;gt; &amp;lt;int&amp;gt; &amp;lt;int&amp;gt;
## 1 15 152 15202 1 2 6 13225 1 1
## 2 15 152 15202 1 2 6 13225 3 1
## 3 15 152 15202 1 2 6 13225 3 1
## 4 15 152 15202 1 2 6 13225 3 1
## 5 15 152 15202 1 2 6 13225 3 1
## 6 15 152 15202 1 2 6 13225 9 1
## 7 15 152 15202 1 2 6 13225 9 1
## 8 15 152 15202 1 2 6 13225 9 1
## 9 15 152 15202 1 2 6 13225 9 1
## 10 15 152 15202 1 2 6 13225 10 1
## # ... with more rows, and 39 more variables: PERSONAN &amp;lt;int&amp;gt;, P07 &amp;lt;int&amp;gt;,
## # P08 &amp;lt;int&amp;gt;, P09 &amp;lt;int&amp;gt;, P10 &amp;lt;int&amp;gt;, P10COMUNA &amp;lt;int&amp;gt;, P10PAIS &amp;lt;int&amp;gt;,
## # P11 &amp;lt;int&amp;gt;, P11COMUNA &amp;lt;int&amp;gt;, P11PAIS &amp;lt;int&amp;gt;, P12 &amp;lt;int&amp;gt;, P12COMUNA &amp;lt;int&amp;gt;,
## # P12PAIS &amp;lt;int&amp;gt;, P12A_LLEGADA &amp;lt;int&amp;gt;, P12A_TRAMO &amp;lt;int&amp;gt;, P13 &amp;lt;int&amp;gt;,
## # P14 &amp;lt;int&amp;gt;, P15 &amp;lt;int&amp;gt;, P15A &amp;lt;int&amp;gt;, P16 &amp;lt;int&amp;gt;, P16A &amp;lt;int&amp;gt;,
## # P16A_OTRO &amp;lt;int&amp;gt;, P17 &amp;lt;int&amp;gt;, P18 &amp;lt;chr&amp;gt;, P19 &amp;lt;int&amp;gt;, P20 &amp;lt;int&amp;gt;,
## # P21M &amp;lt;int&amp;gt;, P21A &amp;lt;int&amp;gt;, P10PAIS_GRUPO &amp;lt;int&amp;gt;, P11PAIS_GRUPO &amp;lt;int&amp;gt;,
## # P12PAIS_GRUPO &amp;lt;int&amp;gt;, ESCOLARIDAD &amp;lt;int&amp;gt;, P16A_GRUPO &amp;lt;int&amp;gt;,
## # REGION_15R &amp;lt;int&amp;gt;, PROVINCIA_15R &amp;lt;int&amp;gt;, COMUNA_15R &amp;lt;int&amp;gt;,
## # P10COMUNA_15R &amp;lt;int&amp;gt;, P11COMUNA_15R &amp;lt;int&amp;gt;, P12COMUNA_15R &amp;lt;int&amp;gt;&lt;/code&gt;&lt;/pre&gt;
&lt;p&gt;Now we can use dplyr verbs. For example, we might retrieve a vector of ages to make a histogram. In the census dataset, age is stored in P09.&lt;/p&gt;
&lt;pre class="r"&gt;&lt;code&gt;edades &amp;lt;- datos_db %&amp;gt;%
select(P09)
&lt;p&gt;edades&lt;/code&gt;&lt;/pre&gt;&lt;/p&gt;
&lt;pre&gt;&lt;code&gt;## # Source: lazy query [?? x 1]
## # Database: sqlite 3.22.0 [C:\Censo\censo2017.sqlite3]
## P09
## &amp;lt;int&amp;gt;
## 1 73
## 2 78
## 3 78
## 4 52
## 5 44
## 6 39
## 7 35
## 8 13
## 9 12
## 10 65
## # ... with more rows&lt;/code&gt;&lt;/pre&gt;
&lt;p&gt;Everything looks fine, but the object is still a connection. R is using lazy evaluation: it performs only part of the operation to avoid unnecessary work.&lt;/p&gt;
&lt;p&gt;We can also confirm this in the environment:&lt;/p&gt;
&lt;div class="figure"&gt;
&lt;img src="https://www.juliancabezas.com/img/archivo_censo_con.JPG" /&gt;
&lt;/div&gt;
&lt;p&gt;As you can see, even the row count is unavailable:&lt;/p&gt;
&lt;pre class="r"&gt;&lt;code&gt;nrow(edades)&lt;/code&gt;&lt;/pre&gt;
&lt;pre&gt;&lt;code&gt;## [1] NA&lt;/code&gt;&lt;/pre&gt;
&lt;p&gt;To bring this column into RAM, use &lt;strong&gt;collect&lt;/strong&gt;. I recommend &lt;strong&gt;always&lt;/strong&gt; using it when you need a complete in-memory object rather than a database connection.&lt;/p&gt;
&lt;pre class="r"&gt;&lt;code&gt;edades&amp;lt;-collect(edades)
&lt;p&gt;nrow(edades)&lt;/code&gt;&lt;/pre&gt;&lt;/p&gt;
&lt;pre&gt;&lt;code&gt;## [1] 17574003&lt;/code&gt;&lt;/pre&gt;
&lt;pre class="r"&gt;&lt;code&gt;edades&lt;/code&gt;&lt;/pre&gt;
&lt;pre&gt;&lt;code&gt;## # A tibble: 17,574,003 x 1
## P09
## &amp;lt;int&amp;gt;
## 1 73
## 2 78
## 3 78
## 4 52
## 5 44
## 6 39
## 7 35
## 8 13
## 9 12
## 10 65
## # ... with 17,573,993 more rows&lt;/code&gt;&lt;/pre&gt;
&lt;p&gt;Now it works!&lt;/p&gt;
&lt;p&gt;Another example: P09 represents age and P08 represents sex. Suppose we want the average age of men (sex = 1) by region. Do not forget &lt;strong&gt;collect&lt;/strong&gt;!&lt;/p&gt;
&lt;pre class="r"&gt;&lt;code&gt;hombres_edad &amp;lt;- datos_db %&amp;gt;%
filter(P08==1) %&amp;gt;%
group_by(REGION) %&amp;gt;%
summarise(Edad_media = mean(P09))
&lt;p&gt;hombres_edad &amp;lt;- collect(hombres_edad)&lt;/code&gt;&lt;/pre&gt;&lt;/p&gt;
&lt;pre&gt;&lt;code&gt;## Warning: Missing values are always removed in SQL.
## Use `AVG(x, na.rm = TRUE)` to silence this warning&lt;/code&gt;&lt;/pre&gt;
&lt;pre class="r"&gt;&lt;code&gt;hombres_edad&lt;/code&gt;&lt;/pre&gt;
&lt;pre&gt;&lt;code&gt;## # A tibble: 16 x 2
## REGION Edad_media
## &amp;lt;int&amp;gt; &amp;lt;dbl&amp;gt;
## 1 1 32.1
## 2 2 33.0
## 3 3 33.7
## 4 4 34.6
## 5 5 35.8
## 6 6 35.7
## 7 7 35.9
## 8 8 35.0
## 9 9 35.3
## 10 10 35.1
## 11 11 34.2
## 12 12 35.9
## 13 13 34.2
## 14 14 35.7
## 15 15 33.5
## 16 16 36.8&lt;/code&gt;&lt;/pre&gt;
&lt;p&gt;We can now work with large structured datasets, retrieve subsets and perform analyses.&lt;/p&gt;
&lt;p&gt;I hope you enjoyed this tutorial. Feel free to get in touch!&lt;/p&gt;
&lt;div class="figure"&gt;
&lt;img src="https://media.giphy.com/media/3hvmlYNsOTFWE/giphy.gif" /&gt;
&lt;/div&gt;
&lt;/div&gt;</description></item><item><title>A function for parallel processing on Windows and servers (R parallelisation, part 2)</title><link>https://www.juliancabezas.com/en/post/parallel-cluster-function/</link><pubDate>Thu, 24 Jan 2019 00:00:00 +0000</pubDate><guid>https://www.juliancabezas.com/en/post/parallel-cluster-function/</guid><description>&lt;div id="TOC"&gt;
&lt;ul&gt;
&lt;li&gt;&lt;a href="#el-problema-crear-clusters-es-tedioso"&gt;The problem: creating clusters is tedious&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href="#la-solucion-una-funcion-para-clear-clusters"&gt;The solution: a function to create clusters&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href="#un-pequeno-y-reciclado-ejemplo"&gt;A small, recycled example&lt;/a&gt;&lt;/li&gt;
&lt;/ul&gt;
&lt;/div&gt;
&lt;p&gt;This short tutorial is the second part of my introduction to parallelising R on Windows. &lt;a href="https://www.juliancabezas.com/en/post/parallel-r-windows/"&gt;I recommend reading the first part&lt;/a&gt;.&lt;/p&gt;
&lt;div id="el-problema-crear-clusters-es-tedioso" class="section level2"&gt;
&lt;h2&gt;The problem: creating clusters is tedious&lt;/h2&gt;
&lt;p&gt;Creating clusters can be tedious: loading packages one by one and exporting datasets is slow, and it is easy to forget a crucial package. We want a more convenient approach. We may also be working on a server without Internet access, which complicates package installation.&lt;/p&gt;
&lt;/div&gt;
&lt;div id="la-solucion-una-funcion-para-clear-clusters" class="section level2"&gt;
&lt;h2&gt;The solution: a function to create clusters&lt;/h2&gt;
&lt;p&gt;I created a function, available in &lt;a href="https://github.com/juliancabezas/random_functions/blob/master/create_cluster"&gt;this GitHub file&lt;/a&gt;. To give credit where it is due, it is inspired by &lt;a href="https://github.com/nathanvan/parallelsugar"&gt;parallelsugar&lt;/a&gt;, with a few modifications.&lt;/p&gt;
&lt;p&gt;The function creates a cluster with &lt;em&gt;n&lt;/em&gt; nodes, specified by the &lt;em&gt;nnodes&lt;/em&gt; argument.&lt;/p&gt;
&lt;p&gt;create_cluster is designed to work on servers. When they have no Internet connection, packages can be installed in a shared folder and loaded from there. Use &lt;em&gt;libroute&lt;/em&gt; to specify the library path.&lt;/p&gt;
&lt;pre class="r"&gt;&lt;code&gt;create_cluster &amp;lt;- function(nnodes, libroute = NULL) {
&lt;h1 id="nnodes-number-of-nodes"&gt;nnodes: Number of nodes&lt;/h1&gt;
&lt;h1 id="libroute-provided-library-route-different-of-the-default-library"&gt;libroute: Provided library route (different of the default library)&lt;/h1&gt;
&lt;p&gt;tryCatch({
cl &amp;lt;- parallel::makeCluster(nnodes) # Create an empty cluster with n nodes&lt;/p&gt;
&lt;pre&gt;&lt;code&gt;loaded.package.names &amp;amp;lt;- c( # Find the loaded packages in the session
sessionInfo()$basePkgs, # base packages
names(sessionInfo()$otherPkgs)
) # aditional loaded packages
# Send the packages and the complete environment to the cluster
parallel::clusterExport(cl,
&amp;amp;quot;loaded.package.names&amp;amp;quot;, # loaded packages
envir = environment()
) # &amp;amp;quot;Environment&amp;amp;quot;
## Charge the packages in all nodes
if (is.null(libroute)) { # Use default library location
parallel::parLapply(cl, 1:length(cl), function(xx) {
lapply(loaded.package.names, function(yy) {
library(yy, character.only = TRUE)
})
})
} else {
# If we provide a library location use it (useful when working in remote machines)
parallel::parLapply(cl, 1:length(cl), function(xx) {
lapply(loaded.package.names, function(yy) {
library(yy, character.only = TRUE, lib.loc = libroute)
})
})
}
&lt;/code&gt;&lt;/pre&gt;
&lt;p&gt;})
return(cl) # The result is the cluster
}&lt;/code&gt;&lt;/pre&gt;&lt;/p&gt;
&lt;/div&gt;
&lt;div id="un-pequeno-y-reciclado-ejemplo" class="section level2"&gt;
&lt;h2&gt;A small, recycled example&lt;/h2&gt;
&lt;p&gt;In the example from the previous tutorial, we used &lt;em&gt;clusterCall&lt;/em&gt; to load &lt;em&gt;randomForest&lt;/em&gt;. Here, if the package and any other dependencies are already loaded in our session, we can export them to the nodes in one step.&lt;/p&gt;
&lt;p&gt;First, load the package and create the function:&lt;/p&gt;
&lt;pre class="r"&gt;&lt;code&gt;library(randomForest)
&lt;p&gt;rf_boot_fun &amp;lt;- function(iter, data, n_sample) {
ind &amp;lt;- sample(1:nrow(data), size = n_sample)
data_sample &amp;lt;- data[ind, ]&lt;/p&gt;
&lt;h1 id="fit-the-random-forest-model"&gt;Fit the Random Forest model&lt;/h1&gt;
&lt;p&gt;model1 &amp;lt;- randomForest(Sepal.Length ~ Petal.Length, data = data_sample)&lt;/p&gt;
&lt;h1 id="extract-the-pseudo-r2"&gt;Extract the pseudo-R2&lt;/h1&gt;
&lt;p&gt;R2 &amp;lt;- model1$rsq[length(model1$rsq)]&lt;/p&gt;
&lt;h1 id="extract-the-mse"&gt;Extract the MSE&lt;/h1&gt;
&lt;p&gt;MSE &amp;lt;- model1$mse[length(model1$mse)]
results &amp;lt;- c(&amp;quot;R2&amp;quot; = R2, &amp;quot;MSE&amp;quot; = MSE)
return(results)
}&lt;/code&gt;&lt;/pre&gt;&lt;/p&gt;
&lt;p&gt;Now I can apply the function with &lt;em&gt;parLapply&lt;/em&gt;.&lt;/p&gt;
&lt;pre class="r"&gt;&lt;code&gt;library(parallel)
&lt;h1 id="define-a-library-path-optional"&gt;Define a library path (optional)&lt;/h1&gt;
&lt;p&gt;ruta_libreria &amp;lt;- &amp;quot;C:/Users/Julian/Documents/R/win-library/3.5&amp;quot;&lt;/p&gt;
&lt;h1 id="this-can-simply-be-nuevoclus-create_cluster4"&gt;This can simply be nuevoclus&amp;lt;-create_cluster(4)&lt;/h1&gt;
&lt;p&gt;nuevoclus &amp;lt;- create_cluster(4, ruta_libreria)&lt;/p&gt;
&lt;h1 id="10000-iterations-of-our-function-using-the-iris-dataset"&gt;10,000 iterations of our function using the &amp;quot;iris&amp;quot; dataset&lt;/h1&gt;
&lt;p&gt;results &amp;lt;- parLapply(nuevoclus, 1:10000, rf_boot_fun, data = iris, n_sample = 100)&lt;/p&gt;
&lt;p&gt;stopCluster(nuevoclus) # Close the cluster; do not forget this step&lt;/p&gt;
&lt;h1 id="the-first-two-results"&gt;The first two results&lt;/h1&gt;
&lt;p&gt;head(results, n = 2)&lt;/code&gt;&lt;/pre&gt;&lt;/p&gt;
&lt;pre&gt;&lt;code&gt;## [[1]]
## R2 MSE
## 0.778244 0.174310
##
## [[2]]
## R2 MSE
## 0.6912495 0.1865890&lt;/code&gt;&lt;/pre&gt;
&lt;p&gt;Done—parallelisation is much easier now!&lt;/p&gt;
&lt;div class="figure"&gt;
&lt;img src="https://media.giphy.com/media/uTuLngvL9p0Xe/giphy.gif" /&gt;
&lt;/div&gt;
&lt;p&gt;&lt;strong&gt;I hope you find this useful!&lt;/strong&gt;&lt;/p&gt;
&lt;p&gt;&lt;strong&gt;I welcome your comments and questions!&lt;/strong&gt;&lt;/p&gt;
&lt;/div&gt;</description></item><item><title>Parallelising R on Windows with the parallel package</title><link>https://www.juliancabezas.com/en/post/parallel-r-windows/</link><pubDate>Mon, 07 Jan 2019 00:00:00 +0000</pubDate><guid>https://www.juliancabezas.com/en/post/parallel-r-windows/</guid><description>&lt;div id="TOC"&gt;
&lt;ul&gt;
&lt;li&gt;&lt;a href="#por-que-paralelizar"&gt;Why parallelise?&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href="#convertir-tu-loop-en-una-funcion"&gt;Turn your loop into a function&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href="#usando-la-funcion-lapply"&gt;Using lapply&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href="#el-paquete-parallel"&gt;The parallel package&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href="#un-ejemplo-mas-realista-usando-machine-learning"&gt;A more realistic example: machine learning&lt;/a&gt;&lt;/li&gt;
&lt;/ul&gt;
&lt;/div&gt;
&lt;div id="por-que-paralelizar" class="section level2"&gt;
&lt;h2&gt;Why parallelise?&lt;/h2&gt;
&lt;p&gt;You have been asked to run code that takes 20 hours to finish. It applies the same function to different parameters or datasets many, &lt;strong&gt;many&lt;/strong&gt; times.&lt;/p&gt;
&lt;p&gt;That might not seem like a problem, but requests often arrive &lt;del&gt;yesterday&lt;/del&gt; without 20 hours of notice. If you are wondering why your PC or workstation has 4, 8 or 16 cores, this tutorial is for you!&lt;/p&gt;
&lt;p&gt;Parallelising R processes can be difficult. Add &lt;del&gt;bad luck&lt;/del&gt; Windows to the equation, and it can feel almost impossible.&lt;/p&gt;
&lt;p&gt;In this tutorial (&lt;strong&gt;the first on this website!&lt;/strong&gt;), we will introduce parallel processing in R on Windows. We will start with the &lt;strong&gt;apply&lt;/strong&gt; family, explore the parallel package and create a Windows cluster, before finishing with a common &lt;em&gt;machine learning&lt;/em&gt; application.&lt;/p&gt;
&lt;div id="nivel-de-este-tutorial" class="section level4"&gt;
&lt;h4&gt;Tutorial level&lt;/h4&gt;
&lt;p&gt;This tutorial assumes you are an intermediate or advanced R user.&lt;/p&gt;
&lt;/div&gt;
&lt;/div&gt;
&lt;div id="convertir-tu-loop-en-una-funcion" class="section level2"&gt;
&lt;h2&gt;Turn your loop into a function&lt;/h2&gt;
&lt;p&gt;We will use the &lt;strong&gt;iris&lt;/strong&gt; dataset, familiar to many R users. It contains petal and sepal length and width measurements for three flower species.&lt;/p&gt;
&lt;pre class="r"&gt;&lt;code&gt;data(iris)
knitr::kable(head(iris))&lt;/code&gt;&lt;/pre&gt;
&lt;table&gt;
&lt;thead&gt;
&lt;tr class="header"&gt;
&lt;th align="right"&gt;Sepal.Length&lt;/th&gt;
&lt;th align="right"&gt;Sepal.Width&lt;/th&gt;
&lt;th align="right"&gt;Petal.Length&lt;/th&gt;
&lt;th align="right"&gt;Petal.Width&lt;/th&gt;
&lt;th align="left"&gt;Species&lt;/th&gt;
&lt;/tr&gt;
&lt;/thead&gt;
&lt;tbody&gt;
&lt;tr class="odd"&gt;
&lt;td align="right"&gt;5.1&lt;/td&gt;
&lt;td align="right"&gt;3.5&lt;/td&gt;
&lt;td align="right"&gt;1.4&lt;/td&gt;
&lt;td align="right"&gt;0.2&lt;/td&gt;
&lt;td align="left"&gt;setosa&lt;/td&gt;
&lt;/tr&gt;
&lt;tr class="even"&gt;
&lt;td align="right"&gt;4.9&lt;/td&gt;
&lt;td align="right"&gt;3.0&lt;/td&gt;
&lt;td align="right"&gt;1.4&lt;/td&gt;
&lt;td align="right"&gt;0.2&lt;/td&gt;
&lt;td align="left"&gt;setosa&lt;/td&gt;
&lt;/tr&gt;
&lt;tr class="odd"&gt;
&lt;td align="right"&gt;4.7&lt;/td&gt;
&lt;td align="right"&gt;3.2&lt;/td&gt;
&lt;td align="right"&gt;1.3&lt;/td&gt;
&lt;td align="right"&gt;0.2&lt;/td&gt;
&lt;td align="left"&gt;setosa&lt;/td&gt;
&lt;/tr&gt;
&lt;tr class="even"&gt;
&lt;td align="right"&gt;4.6&lt;/td&gt;
&lt;td align="right"&gt;3.1&lt;/td&gt;
&lt;td align="right"&gt;1.5&lt;/td&gt;
&lt;td align="right"&gt;0.2&lt;/td&gt;
&lt;td align="left"&gt;setosa&lt;/td&gt;
&lt;/tr&gt;
&lt;tr class="odd"&gt;
&lt;td align="right"&gt;5.0&lt;/td&gt;
&lt;td align="right"&gt;3.6&lt;/td&gt;
&lt;td align="right"&gt;1.4&lt;/td&gt;
&lt;td align="right"&gt;0.2&lt;/td&gt;
&lt;td align="left"&gt;setosa&lt;/td&gt;
&lt;/tr&gt;
&lt;tr class="even"&gt;
&lt;td align="right"&gt;5.4&lt;/td&gt;
&lt;td align="right"&gt;3.9&lt;/td&gt;
&lt;td align="right"&gt;1.7&lt;/td&gt;
&lt;td align="right"&gt;0.4&lt;/td&gt;
&lt;td align="left"&gt;setosa&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;
&lt;p&gt;Suppose a colleague wants to fit a linear model relating petals to sepals. Because the sample is small (150 individuals), you are asked to run 20,000 bootstrap iterations to examine the model coefficients and R&lt;sup&gt;2&lt;/sup&gt;.&lt;/p&gt;
&lt;p&gt;We can do this with a loop that draws a sample of 100 individuals (two-thirds of the dataset), fits a linear model, and saves the R&lt;sup&gt;2&lt;/sup&gt; and coefficients in a data frame.&lt;/p&gt;
&lt;pre class="r"&gt;&lt;code&gt;# Create an empty data frame to store the results
results &amp;lt;- data.frame()
&lt;p&gt;system.time({
for (i in 1:20000) {
# Draw a sample of 100 numbers between 1 and 150
ind &amp;lt;- sample(1:nrow(iris), size = 100)
iris_sample &amp;lt;- iris[ind, ]
# Fit a linear model using petal and sepal measurements
model1 &amp;lt;- lm(iris_sample[, &amp;quot;Sepal.Length&amp;quot;] ~ iris_sample[, &amp;quot;Petal.Length&amp;quot;])&lt;/p&gt;
&lt;pre&gt;&lt;code&gt;# Extract R2 and the coefficients
R2 &amp;amp;lt;- summary(model1)$r.squared
coef &amp;amp;lt;- coefficients(model1)
# Combine everything in the results data frame
results &amp;amp;lt;- rbind(results, c(R2, coef))
&lt;/code&gt;&lt;/pre&gt;
&lt;p&gt;}
})&lt;/code&gt;&lt;/pre&gt;&lt;/p&gt;
&lt;pre&gt;&lt;code&gt;## user system elapsed
## 20.90 0.03 20.95&lt;/code&gt;&lt;/pre&gt;
&lt;p&gt;The algorithm takes just over 20 seconds. That may be acceptable now, but what if tomorrow you need to do the same thing with half a million observations?&lt;/p&gt;
&lt;p&gt;The problem is that the loop uses only one processor core, running one iteration after another. This is not very efficient.&lt;/p&gt;
&lt;p&gt;The first step towards parallelisation is to turn the loop into a function. Here the only argument will be an iteration vector, “iter”, which lets us request more iterations.&lt;/p&gt;
&lt;pre class="r"&gt;&lt;code&gt;lm_boot_fun &amp;lt;- function(iter) {
ind &amp;lt;- sample(1:nrow(iris), size = 100)
iris_sample &amp;lt;- iris[ind, ]
model1 &amp;lt;- lm(iris_sample[, &amp;quot;Sepal.Length&amp;quot;] ~ iris_sample[, &amp;quot;Petal.Length&amp;quot;])
R2 &amp;lt;- summary(model1)$r.squared
coef &amp;lt;- coefficients(model1)
# Create a vector of results
results &amp;lt;- c(&amp;quot;R2&amp;quot; = R2, coef)
# Return the function output
return(results)
}&lt;/code&gt;&lt;/pre&gt;
&lt;p&gt;Let us test the function. As the following code shows, it returns a result regardless of the argument we pass.&lt;/p&gt;
&lt;pre class="r"&gt;&lt;code&gt;result &amp;lt;- lm_boot_fun(1)
result&lt;/code&gt;&lt;/pre&gt;
&lt;pre&gt;&lt;code&gt;## R2 (Intercept)
## 0.7554039 4.3353075
## iris_sample[, &amp;quot;Petal.Length&amp;quot;]
## 0.3999710&lt;/code&gt;&lt;/pre&gt;
&lt;pre class="r"&gt;&lt;code&gt;result &amp;lt;- lm_boot_fun(56876)
result&lt;/code&gt;&lt;/pre&gt;
&lt;pre&gt;&lt;code&gt;## R2 (Intercept)
## 0.7393922 4.3426829
## iris_sample[, &amp;quot;Petal.Length&amp;quot;]
## 0.3939433&lt;/code&gt;&lt;/pre&gt;
&lt;p&gt;Passing a vector still returns just one result, because “results” is overwritten on every iteration.&lt;/p&gt;
&lt;pre class="r"&gt;&lt;code&gt;result &amp;lt;- lm_boot_fun(1:10)
result&lt;/code&gt;&lt;/pre&gt;
&lt;pre&gt;&lt;code&gt;## R2 (Intercept)
## 0.7732318 4.2822007
## iris_sample[, &amp;quot;Petal.Length&amp;quot;]
## 0.4256894&lt;/code&gt;&lt;/pre&gt;
&lt;/div&gt;
&lt;div id="usando-la-funcion-lapply" class="section level2"&gt;
&lt;h2&gt;Using lapply&lt;/h2&gt;
&lt;p&gt;To keep all the results, we can use lapply.&lt;/p&gt;
&lt;p&gt;lapply belongs to the apply family (apply, sapply, vapply, lapply and mapply), one of R’s most powerful tools—and one I wish I had learned earlier. This tutorial is not about the whole family, so we will focus on lapply.&lt;/p&gt;
&lt;p&gt;lapply applies another function, such as the one we created, to a list or vector of arguments, and returns a list of the same length.&lt;/p&gt;
&lt;pre class="r"&gt;&lt;code&gt;iter &amp;lt;- seq(1, 20000)
system.time({
results &amp;lt;- lapply(iter, lm_boot_fun)
})&lt;/code&gt;&lt;/pre&gt;
&lt;pre&gt;&lt;code&gt;## user system elapsed
## 15.05 0.00 15.07&lt;/code&gt;&lt;/pre&gt;
&lt;p&gt;Processing time has already improved a little, but we want to make it much faster. One detail to handle is that lapply returns a list.&lt;/p&gt;
&lt;pre class="r"&gt;&lt;code&gt;head(results, n = 3)&lt;/code&gt;&lt;/pre&gt;
&lt;pre&gt;&lt;code&gt;## [[1]]
## R2 (Intercept)
## 0.7795166 4.2839537
## iris_sample[, &amp;quot;Petal.Length&amp;quot;]
## 0.4123361
##
## [[2]]
## R2 (Intercept)
## 0.7683068 4.2132627
## iris_sample[, &amp;quot;Petal.Length&amp;quot;]
## 0.4311986
##
## [[3]]
## R2 (Intercept)
## 0.7909086 4.2600338
## iris_sample[, &amp;quot;Petal.Length&amp;quot;]
## 0.4163103&lt;/code&gt;&lt;/pre&gt;
&lt;p&gt;To convert this list into a data frame, use unlist to build a matrix and then convert it. Here we specify three columns.&lt;/p&gt;
&lt;pre class="r"&gt;&lt;code&gt;results &amp;lt;- data.frame(matrix(unlist(results), ncol = 3, byrow = T))
head(results, n = 2)&lt;/code&gt;&lt;/pre&gt;
&lt;pre&gt;&lt;code&gt;## X1 X2 X3
## 1 0.7795166 4.283954 0.4123361
## 2 0.7683068 4.213263 0.4311986&lt;/code&gt;&lt;/pre&gt;
&lt;p&gt;You can find more elegant solutions in &lt;a href="https://stackoverflow.com/questions/4227223/r-list-to-data-frame"&gt;this Stack Overflow discussion&lt;/a&gt;.&lt;/p&gt;
&lt;/div&gt;
&lt;div id="el-paquete-parallel" class="section level2"&gt;
&lt;h2&gt;The parallel package&lt;/h2&gt;
&lt;p&gt;The parallel package is included with R. It provides parallel versions of functions in the apply family. Before using them, we need to find out how many cores are available.&lt;/p&gt;
&lt;pre class="r"&gt;&lt;code&gt;library(parallel)
numCores &amp;lt;- detectCores()
numCores&lt;/code&gt;&lt;/pre&gt;
&lt;pre&gt;&lt;code&gt;## [1] 8&lt;/code&gt;&lt;/pre&gt;
&lt;p&gt;We have eight logical cores! As a rule of thumb for this example, we will use half of them (four). On a Unix-like system, such as Linux or macOS, we could use mclapply. However, mclapply relies on &lt;em&gt;forking&lt;/em&gt;, which creates a copy of the process and is available only on Unix-like systems.&lt;/p&gt;
&lt;pre class="r"&gt;&lt;code&gt;iter &amp;lt;- seq(1, 20000)
system.time({
results &amp;lt;- mclapply(iter, lm_boot_fun, mc.cores = 7)
})&lt;/code&gt;&lt;/pre&gt;
&lt;pre&gt;&lt;code&gt;## Error in mclapply(iter, lm_boot_fun, mc.cores = 7): &amp;#39;mc.cores&amp;#39; &amp;gt; 1 is not supported on Windows&lt;/code&gt;&lt;/pre&gt;
&lt;pre&gt;&lt;code&gt;## Timing stopped at: 0 0 0&lt;/code&gt;&lt;/pre&gt;
&lt;p&gt;On Windows, we will create a cluster manually using makeCluster. The resulting object becomes an argument to parLapply. In this case, we will use four cores.&lt;/p&gt;
&lt;pre class="r"&gt;&lt;code&gt;cl &amp;lt;- makeCluster(4) # Create the cluster
system.time(
results &amp;lt;- parLapply(cl, iter, lm_boot_fun)
)&lt;/code&gt;&lt;/pre&gt;
&lt;pre&gt;&lt;code&gt;## user system elapsed
## 0.03 0.00 4.76&lt;/code&gt;&lt;/pre&gt;
&lt;pre class="r"&gt;&lt;code&gt;stopCluster(cl) # Close the cluster; do not forget this step&lt;/code&gt;&lt;/pre&gt;
&lt;p&gt;&lt;strong&gt;Wow!&lt;/strong&gt; We have substantially reduced the bootstrap processing time. We can enjoy a cup of mate—after shutting down the cluster with stopCluster.&lt;/p&gt;
&lt;p&gt;&lt;strong&gt;Great!&lt;/strong&gt; &lt;img src="https://media.giphy.com/media/4xpB3eE00FfBm/giphy.gif" /&gt;&lt;/p&gt;
&lt;p&gt;We could stop here, but this is a very simple example. In practice, we often use datasets stored on our computers and five to ten packages.&lt;/p&gt;
&lt;div id="complejizando-la-funcion" class="section level3"&gt;
&lt;h3&gt;Making the function more complex&lt;/h3&gt;
&lt;p&gt;Let us pretend iris is a dataset stored on our computer. We will also add two arguments to the function: the dataset and the sample size.&lt;/p&gt;
&lt;pre class="r"&gt;&lt;code&gt;# Write the iris dataset to disk
write.csv(iris, file = &amp;quot;iris.csv&amp;quot;)
&lt;h1 id="read-the-dataset-created-above"&gt;Read the dataset created above&lt;/h1&gt;
&lt;p&gt;datos_pc &amp;lt;- read.csv(&amp;quot;iris.csv&amp;quot;)&lt;/p&gt;
&lt;p&gt;lm_boot_fun2 &amp;lt;- function(iter, data, n_sample) {&lt;/p&gt;
&lt;h1 id="data-and-n_sample-are-now-function-arguments"&gt;data and n_sample are now function arguments&lt;/h1&gt;
&lt;p&gt;ind &amp;lt;- sample(1:nrow(data), size = n_sample)
data_sample &amp;lt;- data[ind, ]
model1 &amp;lt;- lm(data_sample[, &amp;quot;Sepal.Length&amp;quot;] ~ data_sample[, &amp;quot;Petal.Length&amp;quot;])
R2 &amp;lt;- summary(model1)$r.squared
coef &amp;lt;- coefficients(model1)
results &amp;lt;- c(&amp;quot;R2&amp;quot; = R2, coef)
return(results)
}&lt;/code&gt;&lt;/pre&gt;&lt;/p&gt;
&lt;p&gt;Now we can repeat the operation using the additional arguments of parLapply.&lt;/p&gt;
&lt;pre class="r"&gt;&lt;code&gt;cl &amp;lt;- makeCluster(4) # Create the cluster
system.time(
results &amp;lt;- parLapply(cl, iter, lm_boot_fun2, data = datos_pc, n_sample = 100)
)&lt;/code&gt;&lt;/pre&gt;
&lt;pre&gt;&lt;code&gt;## user system elapsed
## 0.01 0.00 4.52&lt;/code&gt;&lt;/pre&gt;
&lt;pre class="r"&gt;&lt;code&gt;stopCluster(cl) # Close the cluster; do not forget this step&lt;/code&gt;&lt;/pre&gt;
&lt;p&gt;Suppose your manager walks in and says the client wants a Random Forest—perhaps because it sounds impressive. Let us adapt the function using the randomForest package and save the pseudo-R&lt;sup&gt;2&lt;/sup&gt; and MSE.&lt;/p&gt;
&lt;pre class="r"&gt;&lt;code&gt;library(randomForest)
&lt;p&gt;rf_boot_fun &amp;lt;- function(iter, data, n_sample) {
ind &amp;lt;- sample(1:nrow(data), size = n_sample)
data_sample &amp;lt;- data[ind, ]&lt;/p&gt;
&lt;h1 id="fit-the-random-forest-model"&gt;Fit the Random Forest model&lt;/h1&gt;
&lt;p&gt;model1 &amp;lt;- randomForest(Sepal.Length ~ Petal.Length, data = data_sample)&lt;/p&gt;
&lt;h1 id="extract-the-pseudo-r2"&gt;Extract the pseudo-R2&lt;/h1&gt;
&lt;p&gt;R2 &amp;lt;- model1$rsq[length(model1$rsq)]&lt;/p&gt;
&lt;h1 id="extract-the-mse"&gt;Extract the MSE&lt;/h1&gt;
&lt;p&gt;MSE &amp;lt;- model1$mse[length(model1$mse)]
results &amp;lt;- c(&amp;quot;R2&amp;quot; = R2, &amp;quot;MSE&amp;quot; = MSE)
return(results)
}&lt;/code&gt;&lt;/pre&gt;&lt;/p&gt;
&lt;p&gt;Now let us use &lt;em&gt;parLapply&lt;/em&gt;.&lt;/p&gt;
&lt;pre class="r"&gt;&lt;code&gt;library(randomForest)
cl &amp;lt;- makeCluster(4) # Create the cluster
system.time(
results &amp;lt;- parLapply(cl, iter, rf_boot_fun, data = datos_pc, n_sample = 100)
)&lt;/code&gt;&lt;/pre&gt;
&lt;pre&gt;&lt;code&gt;## Error in checkForRemoteErrors(val): 4 nodes produced errors; first error: could not find function &amp;quot;randomForest&amp;quot;&lt;/code&gt;&lt;/pre&gt;
&lt;pre&gt;&lt;code&gt;## Timing stopped at: 0 0 0&lt;/code&gt;&lt;/pre&gt;
&lt;pre class="r"&gt;&lt;code&gt;stopCluster(cl) # Close the cluster; do not forget this step&lt;/code&gt;&lt;/pre&gt;
&lt;p&gt;&lt;strong&gt;What happened?&lt;/strong&gt; I already called &lt;em&gt;library(randomForest)&lt;/em&gt;!&lt;/p&gt;
&lt;div class="figure"&gt;
&lt;img src="https://media.giphy.com/media/DVkZv0SltX1h6/giphy.gif" /&gt;
&lt;/div&gt;
&lt;p&gt;The &lt;em&gt;randomForest&lt;/em&gt; package is loaded in our current session. However, &lt;em&gt;makeCluster&lt;/em&gt; opens &lt;em&gt;n&lt;/em&gt; new R sessions, each starting from scratch. We must load the packages in every session. In some cases, we must also export datasets and functions.&lt;/p&gt;
&lt;p&gt;To load &lt;em&gt;randomForest&lt;/em&gt; on the cluster nodes, use &lt;em&gt;clusterCall&lt;/em&gt;. It runs a function on every node; here, we use it to load the package on all of them.&lt;/p&gt;
&lt;pre class="r"&gt;&lt;code&gt;cl &amp;lt;- makeCluster(4) # Create the cluster
clusterCall(cl, function() library(randomForest))&lt;/code&gt;&lt;/pre&gt;
&lt;pre class="r"&gt;&lt;code&gt;system.time(
results &amp;lt;- parLapply(cl, iter, rf_boot_fun, data = datos_pc, n_sample = 100)
)&lt;/code&gt;&lt;/pre&gt;
&lt;pre&gt;&lt;code&gt;## user system elapsed
## 0.02 0.00 137.75&lt;/code&gt;&lt;/pre&gt;
&lt;pre class="r"&gt;&lt;code&gt;stopCluster(cl) # Close the cluster; do not forget this step&lt;/code&gt;&lt;/pre&gt;
&lt;/div&gt;
&lt;/div&gt;
&lt;div id="un-ejemplo-mas-realista-usando-machine-learning" class="section level2"&gt;
&lt;h2&gt;A more realistic example: machine learning&lt;/h2&gt;
&lt;p&gt;Now for a more realistic example: model &lt;em&gt;tuning&lt;/em&gt;. This usually means choosing the best value of one or more parameters according to a performance statistic, such as &lt;em&gt;R&lt;sup&gt;2&lt;/sup&gt;&lt;/em&gt; or &lt;em&gt;RMSE&lt;/em&gt;. Tuning uses the training or calibration dataset, while the model is evaluated on validation data.&lt;/p&gt;
&lt;p&gt;We will use iris as our training dataset and assume we have only sepal measurements, to make the task a little harder.&lt;/p&gt;
&lt;p&gt;We will use a &lt;em&gt;Support Vector Machine&lt;/em&gt; (SVM) to predict species from flower measurements. This model works well for classification, but can depend heavily on the cost and gamma parameters when using a radial kernel. We will try to optimise both.&lt;/p&gt;
&lt;p&gt;The &lt;strong&gt;e1071&lt;/strong&gt; package includes a &lt;em&gt;tune&lt;/em&gt; function that uses &lt;em&gt;cross-validation&lt;/em&gt;. Here we will use &lt;em&gt;10-fold&lt;/em&gt; cross-validation with three repetitions. This example comes from a real project with a large data matrix, where each model run took about an hour and a half. Without parallelisation, tuning would have taken days.&lt;/p&gt;
&lt;p&gt;Enough talking—let us look at the model!&lt;/p&gt;
&lt;pre class="r"&gt;&lt;code&gt;library(e1071)
svm_model &amp;lt;- svm(Species ~ Sepal.Length + Sepal.Width, data = iris)
summary(svm_model)&lt;/code&gt;&lt;/pre&gt;
&lt;pre&gt;&lt;code&gt;##
## Call:
## svm(formula = Species ~ Sepal.Length + Sepal.Width, data = iris)
##
##
## Parameters:
## SVM-Type: C-classification
## SVM-Kernel: radial
## cost: 1
## gamma: 0.5
##
## Number of Support Vectors: 86
##
## ( 10 40 36 )
##
##
## Number of Classes: 3
##
## Levels:
## setosa versicolor virginica&lt;/code&gt;&lt;/pre&gt;
&lt;p&gt;The default cost is 1 and gamma is 0.5. Let us examine the &lt;em&gt;performance&lt;/em&gt;.&lt;/p&gt;
&lt;pre class="r"&gt;&lt;code&gt;library(caret)
confusionMatrix(iris$Species, predict(svm_model))&lt;/code&gt;&lt;/pre&gt;
&lt;pre&gt;&lt;code&gt;## Confusion Matrix and Statistics
##
## Reference
## Prediction setosa versicolor virginica
## setosa 49 1 0
## versicolor 0 36 14
## virginica 0 12 38
##
## Overall Statistics
##
## Accuracy : 0.82
## 95% CI : (0.749, 0.8779)
## No Information Rate : 0.3467
## P-Value [Acc &amp;gt; NIR] : &amp;lt; 2.2e-16
##
## Kappa : 0.73
## Mcnemar&amp;#39;s Test P-Value : NA
##
## Statistics by Class:
##
## Class: setosa Class: versicolor Class: virginica
## Sensitivity 1.0000 0.7347 0.7308
## Specificity 0.9901 0.8614 0.8776
## Pos Pred Value 0.9800 0.7200 0.7600
## Neg Pred Value 1.0000 0.8700 0.8600
## Prevalence 0.3267 0.3267 0.3467
## Detection Rate 0.3267 0.2400 0.2533
## Detection Prevalence 0.3333 0.3333 0.3333
## Balanced Accuracy 0.9950 0.7980 0.8042&lt;/code&gt;&lt;/pre&gt;
&lt;p&gt;Accuracy is 0.82. In many cases that is not enough, so let us try to improve it by tuning the parameters.&lt;/p&gt;
&lt;p&gt;We create a function that returns model performance for a given cost and gamma. Notice that I call &lt;em&gt;tune&lt;/em&gt; with only one parameter combination at a time. It could accept a list of parameters, but that would be harder to parallelise.&lt;/p&gt;
&lt;pre class="r"&gt;&lt;code&gt;tune_svm_par &amp;lt;- function(costo, gamma) {
control &amp;lt;- tune.control(
nrepeat = 3, repeat.aggregate = mean,
sampling = &amp;quot;cross&amp;quot;, sampling.aggregate = mean,
sampling.dispersion = sd,
cross = 10
)
&lt;p&gt;svm_cv &amp;lt;- tune(&amp;quot;svm&amp;quot;,
train.x = iris[, c(&amp;quot;Sepal.Length&amp;quot;, &amp;quot;Sepal.Width&amp;quot;)],
train.y = iris$Species,
kernel = &amp;quot;radial&amp;quot;,
scale = TRUE,
type = &amp;quot;C-classification&amp;quot;,
tunecontrol = control,
ranges = list(cost = costo, gamma = gamma)
)&lt;/p&gt;
&lt;p&gt;return(svm_cv$performances)
}&lt;/code&gt;&lt;/pre&gt;&lt;/p&gt;
&lt;p&gt;For parallelisation, we will use &lt;em&gt;clusterMap&lt;/em&gt;, which accepts more than one varying argument. It is the equivalent of &lt;em&gt;mapply&lt;/em&gt; in the apply family.&lt;/p&gt;
&lt;pre class="r"&gt;&lt;code&gt;costos.test &amp;lt;- seq(0.2, 4, by = 0.1)
gammas.test &amp;lt;- seq(0.2, 4, by = 0.1)
&lt;p&gt;grilla &amp;lt;- expand.grid(&amp;quot;costos&amp;quot; = costos.test, &amp;quot;gammas&amp;quot; = gammas.test)
nrow(grilla)&lt;/code&gt;&lt;/pre&gt;&lt;/p&gt;
&lt;pre&gt;&lt;code&gt;## [1] 1521&lt;/code&gt;&lt;/pre&gt;
&lt;p&gt;There are 1,521 combinations. This &lt;em&gt;should&lt;/em&gt; take a while.&lt;/p&gt;
&lt;pre class="r"&gt;&lt;code&gt;t1 &amp;lt;- Sys.time()
&lt;p&gt;cl &amp;lt;- makeCluster(4) # Create the cluster
clusterCall(cl, function() library(e1071))&lt;/p&gt;
&lt;p&gt;list.errores &amp;lt;- clusterMap(
cl = cl, fun = tune_svm_par,
costo = grilla$costos,
gamma = grilla$gammas,
SIMPLIFY = FALSE, RECYCLE = FALSE
)&lt;/p&gt;
&lt;p&gt;library(dplyr)
errores.bind &amp;lt;- bind_rows(list.errores)&lt;/p&gt;
&lt;p&gt;stopCluster(cl)&lt;/p&gt;
&lt;p&gt;t2 &amp;lt;- Sys.time()&lt;/code&gt;&lt;/pre&gt;&lt;/p&gt;
&lt;pre class="r"&gt;&lt;code&gt;print(t2 - t1)&lt;/code&gt;&lt;/pre&gt;
&lt;pre&gt;&lt;code&gt;## Time difference of 52.99884 secs&lt;/code&gt;&lt;/pre&gt;
&lt;p&gt;That hardly took any time! Now let us find the cost and gamma combination with the lowest error.&lt;/p&gt;
&lt;pre class="r"&gt;&lt;code&gt;knitr::kable(subset(errores.bind, errores.bind$error == min(errores.bind$error)))&lt;/code&gt;&lt;/pre&gt;
&lt;table&gt;
&lt;thead&gt;
&lt;tr class="header"&gt;
&lt;th&gt;&lt;/th&gt;
&lt;th align="right"&gt;cost&lt;/th&gt;
&lt;th align="right"&gt;gamma&lt;/th&gt;
&lt;th align="right"&gt;error&lt;/th&gt;
&lt;th align="right"&gt;dispersion&lt;/th&gt;
&lt;/tr&gt;
&lt;/thead&gt;
&lt;tbody&gt;
&lt;tr class="odd"&gt;
&lt;td&gt;885&lt;/td&gt;
&lt;td align="right"&gt;2.8&lt;/td&gt;
&lt;td align="right"&gt;2.4&lt;/td&gt;
&lt;td align="right"&gt;0.18&lt;/td&gt;
&lt;td align="right"&gt;0.1044681&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;
&lt;p&gt;Let us test the SVM with those parameters.&lt;/p&gt;
&lt;pre class="r"&gt;&lt;code&gt;library(e1071)
svm_model &amp;lt;- svm(Species ~ Sepal.Length + Sepal.Width, data = iris, cost = 2.6, gamma = 2)
library(caret)
confusionMatrix(iris$Species, predict(svm_model))&lt;/code&gt;&lt;/pre&gt;
&lt;pre&gt;&lt;code&gt;## Confusion Matrix and Statistics
##
## Reference
## Prediction setosa versicolor virginica
## setosa 50 0 0
## versicolor 0 36 14
## virginica 0 10 40
##
## Overall Statistics
##
## Accuracy : 0.84
## 95% CI : (0.7714, 0.8947)
## No Information Rate : 0.36
## P-Value [Acc &amp;gt; NIR] : &amp;lt; 2.2e-16
##
## Kappa : 0.76
## Mcnemar&amp;#39;s Test P-Value : NA
##
## Statistics by Class:
##
## Class: setosa Class: versicolor Class: virginica
## Sensitivity 1.0000 0.7826 0.7407
## Specificity 1.0000 0.8654 0.8958
## Pos Pred Value 1.0000 0.7200 0.8000
## Neg Pred Value 1.0000 0.9000 0.8600
## Prevalence 0.3333 0.3067 0.3600
## Detection Rate 0.3333 0.2400 0.2667
## Detection Prevalence 0.3333 0.3333 0.3333
## Balanced Accuracy 1.0000 0.8240 0.8183&lt;/code&gt;&lt;/pre&gt;
&lt;p&gt;Training accuracy increased from 0.82 to 0.84. It may not look like much, but a two-percentage-point improvement from a procedure that took less than a minute—thanks to parallelisation—is a good result.&lt;/p&gt;
&lt;p&gt;&lt;strong&gt;Thank you for reading to the end of this tutorial!&lt;/strong&gt; &lt;strong&gt;I welcome your comments and questions.&lt;/strong&gt;&lt;/p&gt;
&lt;/div&gt;</description></item></channel></rss>