<?xml version="1.0" encoding="utf-8" standalone="yes"?><rss version="2.0" xmlns:atom="http://www.w3.org/2005/Atom"><channel><title>Csv | Julián Cabezas Peña</title><link>https://www.juliancabezas.com/en/tags/csv/</link><atom:link href="https://www.juliancabezas.com/en/tags/csv/index.xml" rel="self" type="application/rss+xml"/><description>Csv</description><generator>HugoBlox Kit (https://hugoblox.com)</generator><language>en-US</language><lastBuildDate>Mon, 01 Apr 2019 00:00:00 +0000</lastBuildDate><image><url>https://www.juliancabezas.com/media/icon_hu_a22537ec31d66339.png</url><title>Csv</title><link>https://www.juliancabezas.com/en/tags/csv/</link></image><item><title>Working with CSV or TXT datasets larger than RAM in R using SQLite</title><link>https://www.juliancabezas.com/en/post/large-datasets-r-sqlite/</link><pubDate>Mon, 01 Apr 2019 00:00:00 +0000</pubDate><guid>https://www.juliancabezas.com/en/post/large-datasets-r-sqlite/</guid><description>&lt;div id="TOC"&gt;
&lt;ul&gt;
&lt;li&gt;&lt;a href="#el-problema-del-maldito-ram-y-las-bases-de-datos-enormes"&gt;The RAM problem and enormous datasets&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href="#primero-lo-primero-cuanta-memoria-tengo-disponible"&gt;First things first: how much memory is available?&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href="#csv-o-txt-a-sqllite"&gt;CSV or TXT to SQLite&lt;/a&gt;&lt;ul&gt;
&lt;li&gt;&lt;a href="#los-benditos-paquetes"&gt;The packages you need&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href="#que-tan-grande-es-mi-base-de-datos-cuales-son-sus-caracteristicas"&gt;How large is the dataset, and what does it look like?&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href="#crear-la-ruta-a-la-base-de-datos"&gt;Create the database path&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href="#la-prometida-funcion"&gt;The promised function&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href="#mover-la-csv-o-txt-a-la-base-de-datos-sqllite"&gt;Move the CSV or TXT file into SQLite&lt;/a&gt;&lt;/li&gt;
&lt;/ul&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href="#analizar-y-recoger-datos-con-dplyr"&gt;Analyse and retrieve data with dplyr&lt;/a&gt;&lt;/li&gt;
&lt;/ul&gt;
&lt;/div&gt;
&lt;div id="el-problema-del-maldito-ram-y-las-bases-de-datos-enormes" class="section level3"&gt;
&lt;h3&gt;The RAM problem and enormous datasets&lt;/h3&gt;
&lt;p&gt;I do not know whether this happens to everyone, but whenever I start a new job, I seem to get &lt;strong&gt;the worst computer in the office&lt;/strong&gt;: usually no more than 4 GB of RAM, and even 32-bit Windows if I am unlucky, as happened once. Since I use R for everything, large datasets can be difficult to handle, especially when they exceed the memory available to R. This tutorial also helps with &lt;strong&gt;enormous&lt;/strong&gt; datasets.&lt;/p&gt;
&lt;p&gt;We will check how much RAM is available to R and create a SQLite database. We can then retrieve manageable subsets and analyse them in memory, using a function I adapted from &lt;a href="https://gist.github.com/vnijs/e53a68c957747e82c2e3"&gt;this gist by vnijs&lt;/a&gt;.&lt;/p&gt;
&lt;p&gt;We will use &lt;a href="https://www.sqlite.org/index.html"&gt;SQLite&lt;/a&gt; because it is easy to configure and run locally. PostgreSQL and MySQL are alternatives, but setting them up would be a little more involved.&lt;/p&gt;
&lt;p&gt;This tutorial assumes intermediate-to-advanced R skills and some familiarity with SQL.&lt;/p&gt;
&lt;/div&gt;
&lt;div id="primero-lo-primero-cuanta-memoria-tengo-disponible" class="section level3"&gt;
&lt;h3&gt;First things first: how much memory is available?&lt;/h3&gt;
&lt;p&gt;To check the memory available to R, we will use the memory limit command shown below.&lt;/p&gt;
&lt;pre class="r"&gt;&lt;code&gt;memory.limit()&lt;/code&gt;&lt;/pre&gt;
&lt;pre&gt;&lt;code&gt;## [1] 16269&lt;/code&gt;&lt;/pre&gt;
&lt;p&gt;I have approximately 16 GB of memory available, or about 16,000 MB—the RAM installed in this computer.&lt;/p&gt;
&lt;p&gt;&lt;strong&gt;Note:&lt;/strong&gt; On a 32-bit operating system, the memory limit cannot exceed roughly 2–3 GB, regardless of how much RAM is installed.&lt;/p&gt;
&lt;/div&gt;
&lt;div id="csv-o-txt-a-sqllite" class="section level3"&gt;
&lt;h3&gt;CSV or TXT to SQLite&lt;/h3&gt;
&lt;p&gt;Suppose we have a &lt;strong&gt;HUGE&lt;/strong&gt; CSV or TXT dataset. I will use Chile’s 2017 Population and Housing Census. It is not exceptionally large, but works well for this example. You can download it from &lt;a href="http://www.ine.cl/docs/default-source/censos/censo-2017/base-de-datos/csv/microdatos-persona/microdato_censo2017-personas.rar?sfvrsn=3"&gt;Chile’s National Statistics Institute&lt;/a&gt;.&lt;/p&gt;
&lt;p&gt;Extract the RAR archive to obtain the CSV file. It is approximately 2.5 GB—not enormous, but potentially too large for R on a low-spec computer. Opening it in Excel can also be difficult.&lt;/p&gt;
&lt;p&gt;I will extract the dataset into a folder called “Censo” on the C:/ drive.&lt;/p&gt;
&lt;div class="figure"&gt;
&lt;img src="https://www.juliancabezas.com/img/archivo_censo.JPG" /&gt;
&lt;/div&gt;
&lt;div id="los-benditos-paquetes" class="section level4"&gt;
&lt;h4&gt;The packages you need&lt;/h4&gt;
&lt;p&gt;Make sure these five packages are installed:&lt;/p&gt;
&lt;ul&gt;
&lt;li&gt;&lt;strong&gt;readr&lt;/strong&gt;: to read comma-, tab- or otherwise-delimited files.&lt;/li&gt;
&lt;li&gt;&lt;strong&gt;DBI and RSQLite&lt;/strong&gt;: to communicate with databases and use SQLite.&lt;/li&gt;
&lt;li&gt;&lt;strong&gt;dplyr and dbplyr&lt;/strong&gt;: to retrieve information from the database.&lt;/li&gt;
&lt;/ul&gt;
&lt;pre class="r"&gt;&lt;code&gt;library(readr)
library(DBI)
library(RSQLite)
library(dplyr)
library(dbplyr)&lt;/code&gt;&lt;/pre&gt;
&lt;/div&gt;
&lt;div id="que-tan-grande-es-mi-base-de-datos-cuales-son-sus-caracteristicas" class="section level4"&gt;
&lt;h4&gt;How large is the dataset, and what does it look like?&lt;/h4&gt;
&lt;p&gt;First, define the dataset path and assign it to delim_file:&lt;/p&gt;
&lt;pre class="r"&gt;&lt;code&gt;delim_file &amp;lt;- &amp;quot;C:/Censo/Microdato_Censo2017-Personas.csv&amp;quot;&lt;/code&gt;&lt;/pre&gt;
&lt;p&gt;To inspect the file, read its first two lines using &lt;strong&gt;readLines&lt;/strong&gt;.&lt;/p&gt;
&lt;pre class="r"&gt;&lt;code&gt;readLines(delim_file, n=2)&lt;/code&gt;&lt;/pre&gt;
&lt;pre&gt;&lt;code&gt;## [1] &amp;quot;REGION;PROVINCIA;COMUNA;DC;AREA;ZC_LOC;ID_ZONA_LOC;NVIV;NHOGAR;PERSONAN;P07;P08;P09;P10;P10COMUNA;P10PAIS;P11;P11COMUNA;P11PAIS;P12;P12COMUNA;P12PAIS;P12A_LLEGADA;P12A_TRAMO;P13;P14;P15;P15A;P16;P16A;P16A_OTRO;P17;P18;P19;P20;P21M;P21A;P10PAIS_GRUPO;P11PAIS_GRUPO;P12PAIS_GRUPO;ESCOLARIDAD;P16A_GRUPO;REGION_15R;PROVINCIA_15R;COMUNA_15R;P10COMUNA_15R;P11COMUNA_15R;P12COMUNA_15R&amp;quot;
## [2] &amp;quot;15;152;15202;1;2;6;13225;1;1;1;1;1;73;1;98;998;3;15101;998;1;98;998;9998;98;2;4;6;2;1;2;98;7;98;98;98;98;9998;998;998;998;4;2;15;152;15202;98;15101;98&amp;quot;&lt;/code&gt;&lt;/pre&gt;
&lt;p&gt;We can now see the column names and identify the delimiter: a semicolon (;).&lt;/p&gt;
&lt;p&gt;R cannot count rows by loading a file that does not fit in memory. Instead, we will use a PowerShell command (Get-Content) on Windows, or a system command (wc) on Linux.&lt;/p&gt;
&lt;p&gt;To count the rows in the file—this takes a little while:&lt;/p&gt;
&lt;p&gt;On Windows:&lt;/p&gt;
&lt;pre class="r"&gt;&lt;code&gt;total_records &amp;lt;- system2(&amp;quot;powershell&amp;quot;, args = c(&amp;quot;Get-content&amp;quot;, delim_file, &amp;quot;|&amp;quot;, &amp;quot;Measure-Object&amp;quot;, &amp;quot;–Line&amp;quot;),
stdout = TRUE)
total_records &amp;lt;- as.numeric(gsub(&amp;quot;[^0-9]&amp;quot;, &amp;quot;&amp;quot;, paste(total_records, collapse = &amp;quot;&amp;quot;))) - 1
total_records&lt;/code&gt;&lt;/pre&gt;
&lt;pre&gt;&lt;code&gt;## [1] 17574003&lt;/code&gt;&lt;/pre&gt;
&lt;p&gt;There are 17,574,003 records, excluding the header. That matches the number of people recorded by the 2017 census.&lt;/p&gt;
&lt;p&gt;Note: Windows 10 should work without problems. On Windows 7 or 8, check your PowerShell version using &lt;a href="https://docs.microsoft.com/en-us/skypeforbusiness/set-up-your-computer-for-windows-powershell/download-and-install-windows-powershell-5-1"&gt;these Microsoft instructions&lt;/a&gt;, and install PowerShell 5.1 by following &lt;a href="https://docs.microsoft.com/en-us/powershell/scripting/install/installing-windows-powershell?view=powershell-6"&gt;this Microsoft guide&lt;/a&gt;.&lt;/p&gt;
&lt;p&gt;On Linux, it would look like this:&lt;/p&gt;
&lt;pre class="r"&gt;&lt;code&gt; total_records &amp;lt;- system2(&amp;quot;wc&amp;quot;, args = c(&amp;quot;-l&amp;quot;, delim_file), stdout = TRUE) %&amp;gt;%
sub(normalizePath(delim_file), &amp;quot;&amp;quot;, .) %&amp;gt;%
as.integer() %&amp;gt;%
{
. - 1
}&lt;/code&gt;&lt;/pre&gt;
&lt;/div&gt;
&lt;div id="crear-la-ruta-a-la-base-de-datos" class="section level4"&gt;
&lt;h4&gt;Create the database path&lt;/h4&gt;
&lt;p&gt;Choose a path for the future database file, with the extension .sqlite3.&lt;/p&gt;
&lt;p&gt;Define the database path and the table name. We will need the table name to retrieve data later. Another advantage of SQLite databases is that they can contain several tables.&lt;/p&gt;
&lt;pre class="r"&gt;&lt;code&gt;table_name_censo &amp;lt;- &amp;quot;censo2017&amp;quot;
sqlite_file_censo &amp;lt;- &amp;quot;C:/Censo/censo2017.sqlite3&amp;quot;&lt;/code&gt;&lt;/pre&gt;
&lt;/div&gt;
&lt;div id="la-prometida-funcion" class="section level4"&gt;
&lt;h4&gt;The promised function&lt;/h4&gt;
&lt;p&gt;With the total row count, file paths and delimiter (comma, semicolon, space, and so on) identified, we can use the following function:&lt;/p&gt;
&lt;pre class="r"&gt;&lt;code&gt;read_delim2sqlite &amp;lt;- function(delim_file, delim, sqlite_file, table_name, batch_size = 10000, OS = &amp;quot;Windows&amp;quot;) {
&lt;h2 id="establish-a-connection-to-the-database"&gt;establish a connection to the database&lt;/h2&gt;
&lt;p&gt;condb &amp;lt;- dbConnect(SQLite(), sqlite_file)&lt;/p&gt;
&lt;h2 id="get-the-total-number-of-records-in-the-file"&gt;get the total number of records in the file&lt;/h2&gt;
&lt;h1 id="in-unix"&gt;in Unix&lt;/h1&gt;
&lt;p&gt;if (OS == &amp;quot;Unix&amp;quot;) {
total_records &amp;lt;- system2(&amp;quot;wc&amp;quot;, args = c(&amp;quot;-l&amp;quot;, delim_file), stdout = TRUE) %&amp;gt;%
sub(normalizePath(delim_file), &amp;quot;&amp;quot;, .) %&amp;gt;%
as.integer() %&amp;gt;%
{
. - 1
}
} else {
# In windows
total_records &amp;lt;- system2(&amp;quot;powershell&amp;quot;, args = c(&amp;quot;Get-content&amp;quot;, delim_file, &amp;quot;|&amp;quot;, &amp;quot;Measure-Object&amp;quot;, &amp;quot;–Line&amp;quot;), stdout = TRUE)
total_records &amp;lt;- as.numeric(gsub(&amp;quot;[^0-9]&amp;quot;, &amp;quot;&amp;quot;, paste(total_records, collapse = &amp;quot;&amp;quot;))) - 1
}&lt;/p&gt;
&lt;p&gt;message(&amp;quot;Total records: &amp;quot;, total_records)&lt;/p&gt;
&lt;h2 id="find-the-number-of-passes-needed-based-on-size-of-each-batch"&gt;find the number of passes needed based on size of each batch&lt;/h2&gt;
&lt;p&gt;passes &amp;lt;- total_records %/% batch_size
remaining &amp;lt;- total_records %% batch_size&lt;/p&gt;
&lt;p&gt;message(&amp;quot;Total Passes to complete: &amp;quot;, passes)&lt;/p&gt;
&lt;h2 id="first-pass-determines-header-and-column-types"&gt;first pass determines header and column types&lt;/h2&gt;
&lt;p&gt;dat &amp;lt;- read_delim(delim_file, delim, n_max = batch_size, progress = FALSE) %&amp;gt;% as.data.frame()
if (nrow(problems(dat)) &amp;gt; 0) print(problems(dat))
col_names &amp;lt;- colnames(dat)
col_types &amp;lt;- c(character = &amp;quot;c&amp;quot;, numeric = &amp;quot;d&amp;quot;, integer = &amp;quot;i&amp;quot;, logical = &amp;quot;l&amp;quot;, Date = &amp;quot;c&amp;quot;) %&amp;gt;%
.[sapply(dat, class)] %&amp;gt;%
paste0(collapse = &amp;quot;&amp;quot;)&lt;/p&gt;
&lt;h2 id="write-to-database-table"&gt;write to database table&lt;/h2&gt;
&lt;p&gt;dbWriteTable(condb, table_name, dat, overwrite = TRUE)&lt;/p&gt;
&lt;h2 id="multiple-passes"&gt;multiple passes&lt;/h2&gt;
&lt;p&gt;for (p in 2:passes) {
message(&amp;quot;Pass number: &amp;quot;, p, &amp;quot;, Progress:&amp;quot;, round(p / passes, 2) * 100, &amp;quot;%&amp;quot;)
read_delim(delim_file, delim,
col_names = col_names, col_types = col_types,
skip = (p - 1) * batch_size + 1, n_max = batch_size, progress = FALSE
) %&amp;gt;%
as.data.frame() %&amp;gt;%
dbWriteTable(condb, table_name, ., append = TRUE)
}&lt;/p&gt;
&lt;p&gt;if (remaining) {
read_delim(delim_file, delim,
col_names = col_names, col_types = col_types,
skip = p * batch_size + 1, n_max = remaining, progress = FALSE
) %&amp;gt;%
as.data.frame() %&amp;gt;%
dbWriteTable(condb, table_name, ., append = TRUE)
}&lt;/p&gt;
&lt;h2 id="close-the-database-connection"&gt;close the database connection&lt;/h2&gt;
&lt;p&gt;dbDisconnect(condb)
}&lt;/code&gt;&lt;/pre&gt;&lt;/p&gt;
&lt;p&gt;The function may look intimidating, but it is fairly straightforward. It reads the file in chunks of batch_size rows using read_delim from readr, inserts each chunk into the SQLite database, and repeats the process through the final line.&lt;/p&gt;
&lt;p&gt;The function arguments are:&lt;/p&gt;
&lt;p&gt;&lt;strong&gt;delim_file&lt;/strong&gt;: path to the delimited file, in this case the census dataset. &lt;strong&gt;delim&lt;/strong&gt;: delimiter character, a semicolon here. &lt;strong&gt;sqlite_file&lt;/strong&gt;: path to the SQLite database, ending in “.sqlite3”. &lt;strong&gt;table_name&lt;/strong&gt;: name of the table within the SQLite database. &lt;strong&gt;batch_size&lt;/strong&gt;: number of records to read in each iteration. &lt;strong&gt;OS&lt;/strong&gt;: the operating system, either “Windows” or “Unix”; the Unix option was tested on Linux, not macOS.&lt;/p&gt;
&lt;/div&gt;
&lt;div id="mover-la-csv-o-txt-a-la-base-de-datos-sqllite" class="section level4"&gt;
&lt;h4&gt;Move the CSV or TXT file into SQLite&lt;/h4&gt;
&lt;p&gt;We are ready to use the function. To avoid running out of RAM, I will transfer the dataset in batches of one million rows. It should finish in 18 iterations.&lt;/p&gt;
&lt;p&gt;&lt;strong&gt;This will take a few minutes—time to make some mate!&lt;/strong&gt;&lt;/p&gt;
&lt;pre class="r"&gt;&lt;code&gt;read_delim2sqlite(delim_file, delim=&amp;quot;;&amp;quot;, sqlite_file = sqlite_file_censo,
table_name = table_name_censo, batch_size=1000000)&lt;/code&gt;&lt;/pre&gt;
&lt;pre&gt;&lt;code&gt;## Total records: 17574003&lt;/code&gt;&lt;/pre&gt;
&lt;pre&gt;&lt;code&gt;## Total Passes to complete: 17&lt;/code&gt;&lt;/pre&gt;
&lt;pre&gt;&lt;code&gt;## Parsed with column specification:
## cols(
## .default = col_integer(),
## P18 = col_character()
## )&lt;/code&gt;&lt;/pre&gt;
&lt;pre&gt;&lt;code&gt;## See spec(...) for full column specifications.&lt;/code&gt;&lt;/pre&gt;
&lt;pre&gt;&lt;code&gt;## Pass number: 2, Progress:12%&lt;/code&gt;&lt;/pre&gt;
&lt;pre&gt;&lt;code&gt;## Pass number: 3, Progress:18%&lt;/code&gt;&lt;/pre&gt;
&lt;pre&gt;&lt;code&gt;## Pass number: 4, Progress:24%&lt;/code&gt;&lt;/pre&gt;
&lt;pre&gt;&lt;code&gt;## Pass number: 5, Progress:29%&lt;/code&gt;&lt;/pre&gt;
&lt;pre&gt;&lt;code&gt;## Pass number: 6, Progress:35%&lt;/code&gt;&lt;/pre&gt;
&lt;pre&gt;&lt;code&gt;## Pass number: 7, Progress:41%&lt;/code&gt;&lt;/pre&gt;
&lt;pre&gt;&lt;code&gt;## Pass number: 8, Progress:47%&lt;/code&gt;&lt;/pre&gt;
&lt;pre&gt;&lt;code&gt;## Pass number: 9, Progress:53%&lt;/code&gt;&lt;/pre&gt;
&lt;pre&gt;&lt;code&gt;## Pass number: 10, Progress:59%&lt;/code&gt;&lt;/pre&gt;
&lt;pre&gt;&lt;code&gt;## Pass number: 11, Progress:65%&lt;/code&gt;&lt;/pre&gt;
&lt;pre&gt;&lt;code&gt;## Pass number: 12, Progress:71%&lt;/code&gt;&lt;/pre&gt;
&lt;pre&gt;&lt;code&gt;## Pass number: 13, Progress:76%&lt;/code&gt;&lt;/pre&gt;
&lt;pre&gt;&lt;code&gt;## Pass number: 14, Progress:82%&lt;/code&gt;&lt;/pre&gt;
&lt;pre&gt;&lt;code&gt;## Pass number: 15, Progress:88%&lt;/code&gt;&lt;/pre&gt;
&lt;pre&gt;&lt;code&gt;## Pass number: 16, Progress:94%&lt;/code&gt;&lt;/pre&gt;
&lt;pre&gt;&lt;code&gt;## Pass number: 17, Progress:100%&lt;/code&gt;&lt;/pre&gt;
&lt;p&gt;We only need to run this lengthy conversion once. It creates a .sqlite3 file that is slightly smaller than the CSV.&lt;/p&gt;
&lt;div class="figure"&gt;
&lt;img src="https://www.juliancabezas.com/img/archivo_censo_sqlite.JPG" alt="SQLite file created" /&gt;
&lt;p class="caption"&gt;SQLite file created&lt;/p&gt;
&lt;/div&gt;
&lt;/div&gt;
&lt;/div&gt;
&lt;div id="analizar-y-recoger-datos-con-dplyr" class="section level3"&gt;
&lt;h3&gt;Analyse and retrieve data with dplyr&lt;/h3&gt;
&lt;p&gt;To start retrieving data from the table, first connect to the database.&lt;/p&gt;
&lt;pre class="r"&gt;&lt;code&gt;con &amp;lt;- dbConnect(SQLite(), sqlite_file_censo)
con&lt;/code&gt;&lt;/pre&gt;
&lt;pre&gt;&lt;code&gt;## &amp;lt;SQLiteConnection&amp;gt;
## Path: C:\Censo\censo2017.sqlite3
## Extensions: TRUE&lt;/code&gt;&lt;/pre&gt;
&lt;p&gt;We can inspect the connection details and the table name assigned to the dataset.&lt;/p&gt;
&lt;pre class="r"&gt;&lt;code&gt;dbListTables(con)&lt;/code&gt;&lt;/pre&gt;
&lt;pre&gt;&lt;code&gt;## [1] &amp;quot;censo2017&amp;quot;&lt;/code&gt;&lt;/pre&gt;
&lt;p&gt;Create an object representing the data. It does not load the dataset into RAM: it is only a connection, which is why the total row count is not shown.&lt;/p&gt;
&lt;pre class="r"&gt;&lt;code&gt;datos_db &amp;lt;- tbl(con, &amp;quot;censo2017&amp;quot;)
datos_db&lt;/code&gt;&lt;/pre&gt;
&lt;pre&gt;&lt;code&gt;## # Source: table&amp;lt;censo2017&amp;gt; [?? x 48]
## # Database: sqlite 3.22.0 [C:\Censo\censo2017.sqlite3]
## REGION PROVINCIA COMUNA DC AREA ZC_LOC ID_ZONA_LOC NVIV NHOGAR
## &amp;lt;int&amp;gt; &amp;lt;int&amp;gt; &amp;lt;int&amp;gt; &amp;lt;int&amp;gt; &amp;lt;int&amp;gt; &amp;lt;int&amp;gt; &amp;lt;int&amp;gt; &amp;lt;int&amp;gt; &amp;lt;int&amp;gt;
## 1 15 152 15202 1 2 6 13225 1 1
## 2 15 152 15202 1 2 6 13225 3 1
## 3 15 152 15202 1 2 6 13225 3 1
## 4 15 152 15202 1 2 6 13225 3 1
## 5 15 152 15202 1 2 6 13225 3 1
## 6 15 152 15202 1 2 6 13225 9 1
## 7 15 152 15202 1 2 6 13225 9 1
## 8 15 152 15202 1 2 6 13225 9 1
## 9 15 152 15202 1 2 6 13225 9 1
## 10 15 152 15202 1 2 6 13225 10 1
## # ... with more rows, and 39 more variables: PERSONAN &amp;lt;int&amp;gt;, P07 &amp;lt;int&amp;gt;,
## # P08 &amp;lt;int&amp;gt;, P09 &amp;lt;int&amp;gt;, P10 &amp;lt;int&amp;gt;, P10COMUNA &amp;lt;int&amp;gt;, P10PAIS &amp;lt;int&amp;gt;,
## # P11 &amp;lt;int&amp;gt;, P11COMUNA &amp;lt;int&amp;gt;, P11PAIS &amp;lt;int&amp;gt;, P12 &amp;lt;int&amp;gt;, P12COMUNA &amp;lt;int&amp;gt;,
## # P12PAIS &amp;lt;int&amp;gt;, P12A_LLEGADA &amp;lt;int&amp;gt;, P12A_TRAMO &amp;lt;int&amp;gt;, P13 &amp;lt;int&amp;gt;,
## # P14 &amp;lt;int&amp;gt;, P15 &amp;lt;int&amp;gt;, P15A &amp;lt;int&amp;gt;, P16 &amp;lt;int&amp;gt;, P16A &amp;lt;int&amp;gt;,
## # P16A_OTRO &amp;lt;int&amp;gt;, P17 &amp;lt;int&amp;gt;, P18 &amp;lt;chr&amp;gt;, P19 &amp;lt;int&amp;gt;, P20 &amp;lt;int&amp;gt;,
## # P21M &amp;lt;int&amp;gt;, P21A &amp;lt;int&amp;gt;, P10PAIS_GRUPO &amp;lt;int&amp;gt;, P11PAIS_GRUPO &amp;lt;int&amp;gt;,
## # P12PAIS_GRUPO &amp;lt;int&amp;gt;, ESCOLARIDAD &amp;lt;int&amp;gt;, P16A_GRUPO &amp;lt;int&amp;gt;,
## # REGION_15R &amp;lt;int&amp;gt;, PROVINCIA_15R &amp;lt;int&amp;gt;, COMUNA_15R &amp;lt;int&amp;gt;,
## # P10COMUNA_15R &amp;lt;int&amp;gt;, P11COMUNA_15R &amp;lt;int&amp;gt;, P12COMUNA_15R &amp;lt;int&amp;gt;&lt;/code&gt;&lt;/pre&gt;
&lt;p&gt;Now we can use dplyr verbs. For example, we might retrieve a vector of ages to make a histogram. In the census dataset, age is stored in P09.&lt;/p&gt;
&lt;pre class="r"&gt;&lt;code&gt;edades &amp;lt;- datos_db %&amp;gt;%
select(P09)
&lt;p&gt;edades&lt;/code&gt;&lt;/pre&gt;&lt;/p&gt;
&lt;pre&gt;&lt;code&gt;## # Source: lazy query [?? x 1]
## # Database: sqlite 3.22.0 [C:\Censo\censo2017.sqlite3]
## P09
## &amp;lt;int&amp;gt;
## 1 73
## 2 78
## 3 78
## 4 52
## 5 44
## 6 39
## 7 35
## 8 13
## 9 12
## 10 65
## # ... with more rows&lt;/code&gt;&lt;/pre&gt;
&lt;p&gt;Everything looks fine, but the object is still a connection. R is using lazy evaluation: it performs only part of the operation to avoid unnecessary work.&lt;/p&gt;
&lt;p&gt;We can also confirm this in the environment:&lt;/p&gt;
&lt;div class="figure"&gt;
&lt;img src="https://www.juliancabezas.com/img/archivo_censo_con.JPG" /&gt;
&lt;/div&gt;
&lt;p&gt;As you can see, even the row count is unavailable:&lt;/p&gt;
&lt;pre class="r"&gt;&lt;code&gt;nrow(edades)&lt;/code&gt;&lt;/pre&gt;
&lt;pre&gt;&lt;code&gt;## [1] NA&lt;/code&gt;&lt;/pre&gt;
&lt;p&gt;To bring this column into RAM, use &lt;strong&gt;collect&lt;/strong&gt;. I recommend &lt;strong&gt;always&lt;/strong&gt; using it when you need a complete in-memory object rather than a database connection.&lt;/p&gt;
&lt;pre class="r"&gt;&lt;code&gt;edades&amp;lt;-collect(edades)
&lt;p&gt;nrow(edades)&lt;/code&gt;&lt;/pre&gt;&lt;/p&gt;
&lt;pre&gt;&lt;code&gt;## [1] 17574003&lt;/code&gt;&lt;/pre&gt;
&lt;pre class="r"&gt;&lt;code&gt;edades&lt;/code&gt;&lt;/pre&gt;
&lt;pre&gt;&lt;code&gt;## # A tibble: 17,574,003 x 1
## P09
## &amp;lt;int&amp;gt;
## 1 73
## 2 78
## 3 78
## 4 52
## 5 44
## 6 39
## 7 35
## 8 13
## 9 12
## 10 65
## # ... with 17,573,993 more rows&lt;/code&gt;&lt;/pre&gt;
&lt;p&gt;Now it works!&lt;/p&gt;
&lt;p&gt;Another example: P09 represents age and P08 represents sex. Suppose we want the average age of men (sex = 1) by region. Do not forget &lt;strong&gt;collect&lt;/strong&gt;!&lt;/p&gt;
&lt;pre class="r"&gt;&lt;code&gt;hombres_edad &amp;lt;- datos_db %&amp;gt;%
filter(P08==1) %&amp;gt;%
group_by(REGION) %&amp;gt;%
summarise(Edad_media = mean(P09))
&lt;p&gt;hombres_edad &amp;lt;- collect(hombres_edad)&lt;/code&gt;&lt;/pre&gt;&lt;/p&gt;
&lt;pre&gt;&lt;code&gt;## Warning: Missing values are always removed in SQL.
## Use `AVG(x, na.rm = TRUE)` to silence this warning&lt;/code&gt;&lt;/pre&gt;
&lt;pre class="r"&gt;&lt;code&gt;hombres_edad&lt;/code&gt;&lt;/pre&gt;
&lt;pre&gt;&lt;code&gt;## # A tibble: 16 x 2
## REGION Edad_media
## &amp;lt;int&amp;gt; &amp;lt;dbl&amp;gt;
## 1 1 32.1
## 2 2 33.0
## 3 3 33.7
## 4 4 34.6
## 5 5 35.8
## 6 6 35.7
## 7 7 35.9
## 8 8 35.0
## 9 9 35.3
## 10 10 35.1
## 11 11 34.2
## 12 12 35.9
## 13 13 34.2
## 14 14 35.7
## 15 15 33.5
## 16 16 36.8&lt;/code&gt;&lt;/pre&gt;
&lt;p&gt;We can now work with large structured datasets, retrieve subsets and perform analyses.&lt;/p&gt;
&lt;p&gt;I hope you enjoyed this tutorial. Feel free to get in touch!&lt;/p&gt;
&lt;div class="figure"&gt;
&lt;img src="https://media.giphy.com/media/3hvmlYNsOTFWE/giphy.gif" /&gt;
&lt;/div&gt;
&lt;/div&gt;</description></item></channel></rss>