<?xml version="1.0" encoding="utf-8" standalone="yes"?><rss version="2.0" xmlns:atom="http://www.w3.org/2005/Atom"><channel><title>Julián Cabezas Peña</title><link>https://www.juliancabezas.com/en/</link><atom:link href="https://www.juliancabezas.com/en/index.xml" rel="self" type="application/rss+xml"/><description>Julián Cabezas Peña</description><generator>HugoBlox Kit (https://hugoblox.com)</generator><language>en-US</language><lastBuildDate>Wed, 11 Sep 2019 00:00:00 +0000</lastBuildDate><image><url>https://www.juliancabezas.com/media/icon_hu_a22537ec31d66339.png</url><title>Julián Cabezas Peña</title><link>https://www.juliancabezas.com/en/</link></image><item><title>GCBM Implementation in Los Rios Region (Chile) - Pilot project (version 0.2)</title><link>https://www.juliancabezas.com/en/project/gcbm/pilot_losrios_v02/</link><pubDate>Wed, 11 Sep 2019 00:00:00 +0000</pubDate><guid>https://www.juliancabezas.com/en/project/gcbm/pilot_losrios_v02/</guid><description>&lt;p&gt;&lt;strong&gt;Disclaimer&lt;/strong&gt;: This is not an offitial release and is only meant for development and feed-back purposes&lt;/p&gt;
&lt;h3 id="included-disturbances"&gt;Included disturbances&lt;/h3&gt;
&lt;p&gt;In this run, we are only including the Deforestation and Substitution activities, we are using two default disturbances for their representation&lt;/p&gt;
&lt;ul&gt;
&lt;li&gt;Deforestation: Deforestation-agriculture - crop-salvage and burn&lt;/li&gt;
&lt;li&gt;Substitution: Deforestation-agriculture - pasture -salvage and burn&lt;/li&gt;
&lt;/ul&gt;
&lt;h3 id="some-assumptions-and-procedures"&gt;Some assumptions and procedures:&lt;/h3&gt;
&lt;ul&gt;
&lt;li&gt;Only native forest was included, forest plantations are considered to have &amp;ldquo;zero&amp;rdquo; carbon, this is for REDD+ accounting purposes&lt;/li&gt;
&lt;li&gt;The classifieres used for the mapping of the forests were &amp;ldquo;Forest type&amp;rdquo; and &amp;ldquo;Structure&amp;rdquo;&lt;/li&gt;
&lt;li&gt;The arborescent shrubland is considered a forest type&lt;/li&gt;
&lt;li&gt;The resolution used in the tiler was 0.001 degrees&lt;/li&gt;
&lt;li&gt;The disturbances take place in the same moment as the land use map is released (in this case the years 2006, 2013 and 2017), so we can see some spikes&lt;/li&gt;
&lt;li&gt;We are using linear growth curves&lt;/li&gt;
&lt;li&gt;The initial age was taken from a 2015 volume raster, so in order to get an age in 1997, the volume was assigned to an age in the growth curves and then the difference between 2015 and 1997 was substacted, in the case of resulting negative numbers, an initial age of 1 was assigned&lt;/li&gt;
&lt;/ul&gt;
&lt;h3 id="power-bi-results"&gt;Power BI results&lt;/h3&gt;
&lt;p&gt;In the following frame we can find the results for this run of the model.&lt;/p&gt;
&lt;p&gt;The pages of the Power BI visualization show:&lt;/p&gt;
&lt;ul&gt;
&lt;li&gt;Stocks in pages 1 and 2&lt;/li&gt;
&lt;li&gt;Fluxes in pages 3, 4 and 5&lt;/li&gt;
&lt;li&gt;Disturbances in page 6&lt;/li&gt;
&lt;li&gt;Ages in page 7&lt;/li&gt;
&lt;/ul&gt;
&lt;p&gt;&lt;strong&gt;Power BI&lt;/strong&gt;
&lt;iframe width="1100" height="750" src="https://app.powerbi.com/view?r=eyJrIjoiNDNjYjFkZjgtMTQwNi00ZmJlLWEwNDEtNzUzZGRjZjc3OGRjIiwidCI6ImU2ZjQ4MjNiLTljZmUtNGNiYi04ZDcwLWQyNWU5YTMxZDg1OSIsImMiOjR9" frameborder="0" allowFullScreen="true"&gt;&lt;/iframe&gt;
&lt;/p&gt;
&lt;h2 id="next-steps"&gt;Next steps&lt;/h2&gt;
&lt;ul&gt;
&lt;li&gt;Include afforestation&lt;/li&gt;
&lt;li&gt;Sparse the disturbances randomly in the different &amp;ldquo;phases&amp;rdquo; e.g. distribute the deforestations detected in 2006 along the years 1997 to 2006&lt;/li&gt;
&lt;/ul&gt;</description></item><item><title>GCBM Implementation in Los Rios Region (Chile) - Pilot project (version 0.6.4)</title><link>https://www.juliancabezas.com/en/project/gcbm/pilot_losrios_v0_6_4/</link><pubDate>Wed, 11 Sep 2019 00:00:00 +0000</pubDate><guid>https://www.juliancabezas.com/en/project/gcbm/pilot_losrios_v0_6_4/</guid><description>&lt;p&gt;&lt;strong&gt;Disclaimer&lt;/strong&gt;: This is not an official release and is only meant for development and feed-back purposes&lt;/p&gt;
&lt;h3 id="characteristics-of-this-version"&gt;Characteristics of this version&lt;/h3&gt;
&lt;p&gt;The idea of this version of the model is to reproduce as exactly as possible the assumptions made in the subnational Forest Reference Emission Levels (FREL/FRL) submitted by Chile to the UNFCCC&lt;/p&gt;
&lt;p&gt;The information to run this model comes from the layers of the cadastre of native forest (land use map) from Los Rios region, that has registries from the years 1997, 2006, 2013 and 2017. The biomass parameters come from the National Forest Inventory.&lt;/p&gt;
&lt;h3 id="ages-and-growth-curves"&gt;Ages and growth curves&lt;/h3&gt;
&lt;p&gt;To reproduce the mean regional volume of existence used in the FREL/FRL for Los Rios Region (375.29 m3/ha). In consequence, a single curve was used for all forest types, starting with zero and reaching the mean regional volume at 100 years, using an increment of 3.7529 m3/ha/year.&lt;/p&gt;
&lt;p&gt;
&lt;figure &gt;
&lt;div class="flex justify-center "&gt;
&lt;div class="w-full" &gt;
&lt;img alt="Growth curves"
srcset="https://www.juliancabezas.com/media/img/volume_v0_5_1_hu_419f2710aed012b0.webp 320w, https://www.juliancabezas.com/media/img/volume_v0_5_1_hu_b4cfe561bb9a931e.webp 480w, https://www.juliancabezas.com/media/img/volume_v0_5_1_hu_8418af5b66f0307a.webp 587w"
sizes="(max-width: 480px) 100vw, (max-width: 768px) 90vw, (max-width: 1024px) 80vw, 760px"
src="https://www.juliancabezas.com/media/img/volume_v0_5_1_hu_419f2710aed012b0.webp"
width="587"
height="524"
loading="lazy" data-zoomable /&gt;&lt;/div&gt;
&lt;/div&gt;&lt;/figure&gt;
&lt;/p&gt;
&lt;p&gt;The initial age of the forest registered in the 1997 land use map is considered 100, to consider a single value of volume (or carbon) at the moment of a deforestation event.&lt;/p&gt;
&lt;p&gt;For the arborescent shrubland, a similar approach was used, considering a maximum volume of 49.25 m3/ha (this is not the real value, it was calculated to use the same root parameters as in the normal forest, it will be changed in the future)&lt;/p&gt;
&lt;h3 id="disturbances"&gt;Disturbances&lt;/h3&gt;
&lt;p&gt;To get the disturbance data, land use changes were detected between consecutive dates of the cadastre (official land use map of Chile), following the rules described in the FREL/FRL. The disturbance polygons were randomly distributed between the years in between the two cadastre maps (those included). For example, if a deforestation was detected in 2006, the year of the disturbance was determined choosing a random number between 1997 and 2006.&lt;/p&gt;
&lt;p&gt;In this run, we are only including the Afforestation, Deforestation and Substitution activities, for the latter two we are using a single custom disturbance matrix for their representation, The idea is to transfer all the DOM and Biomass pools to CO2, just as in the FREL/FRL:&lt;/p&gt;
&lt;table&gt;
&lt;thead&gt;
&lt;tr&gt;
&lt;th&gt;Source&lt;/th&gt;
&lt;th&gt;Sink&lt;/th&gt;
&lt;th&gt;Proportion&lt;/th&gt;
&lt;/tr&gt;
&lt;/thead&gt;
&lt;tbody&gt;
&lt;tr&gt;
&lt;td&gt;SoftwoodMerch&lt;/td&gt;
&lt;td&gt;CO2&lt;/td&gt;
&lt;td&gt;1&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;SoftwoodFoliage&lt;/td&gt;
&lt;td&gt;CO2&lt;/td&gt;
&lt;td&gt;1&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;SoftwoodOther&lt;/td&gt;
&lt;td&gt;CO2&lt;/td&gt;
&lt;td&gt;1&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;SoftwoodCoarseRoots&lt;/td&gt;
&lt;td&gt;CO2&lt;/td&gt;
&lt;td&gt;1&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;SoftwoodFineRoots&lt;/td&gt;
&lt;td&gt;CO2&lt;/td&gt;
&lt;td&gt;1&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;HardwoodMerch&lt;/td&gt;
&lt;td&gt;CO2&lt;/td&gt;
&lt;td&gt;1&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;HardwoodFoliage&lt;/td&gt;
&lt;td&gt;CO2&lt;/td&gt;
&lt;td&gt;1&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;HardwoodOther&lt;/td&gt;
&lt;td&gt;CO2&lt;/td&gt;
&lt;td&gt;1&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;HardwoodCoarseRoots&lt;/td&gt;
&lt;td&gt;CO2&lt;/td&gt;
&lt;td&gt;1&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;HardwoodFineRoots&lt;/td&gt;
&lt;td&gt;CO2&lt;/td&gt;
&lt;td&gt;1&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;AboveGroundVeryFastSoil&lt;/td&gt;
&lt;td&gt;AboveGroundVeryFastSoil&lt;/td&gt;
&lt;td&gt;1&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;BelowGroundVeryFastSoil&lt;/td&gt;
&lt;td&gt;BelowGroundVeryFastSoil&lt;/td&gt;
&lt;td&gt;1&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;AboveGroundFastSoil&lt;/td&gt;
&lt;td&gt;AboveGroundFastSoil&lt;/td&gt;
&lt;td&gt;1&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;BelowGroundFastSoil&lt;/td&gt;
&lt;td&gt;BelowGroundFastSoil&lt;/td&gt;
&lt;td&gt;1&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;MediumSoil&lt;/td&gt;
&lt;td&gt;MediumSoil&lt;/td&gt;
&lt;td&gt;1&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;AboveGroundSlowSoil&lt;/td&gt;
&lt;td&gt;AboveGroundSlowSoil&lt;/td&gt;
&lt;td&gt;1&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;BelowGroundSlowSoil&lt;/td&gt;
&lt;td&gt;BelowGroundSlowSoil&lt;/td&gt;
&lt;td&gt;1&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;SoftwoodStemSnag&lt;/td&gt;
&lt;td&gt;CO2&lt;/td&gt;
&lt;td&gt;1&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;SoftwoodBranchSnag&lt;/td&gt;
&lt;td&gt;CO2&lt;/td&gt;
&lt;td&gt;1&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;HardwoodStemSnag&lt;/td&gt;
&lt;td&gt;CO2&lt;/td&gt;
&lt;td&gt;1&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;HardwoodBranchSnag&lt;/td&gt;
&lt;td&gt;CO2&lt;/td&gt;
&lt;td&gt;1&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;BlackCarbon&lt;/td&gt;
&lt;td&gt;BlackCarbon&lt;/td&gt;
&lt;td&gt;1&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Peat&lt;/td&gt;
&lt;td&gt;Peat&lt;/td&gt;
&lt;td&gt;1&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;
&lt;p&gt;The afforestation activity corresponds to new forests that are detected in the cadaster maps, these new forests grow with and Annual Periodic Increment that is determined by the Forest type and Structure of the new forest&lt;/p&gt;
&lt;h3 id="vol-to-bio-factors"&gt;Vol to bio factors&lt;/h3&gt;
&lt;p&gt;To reflect the characteristics of the FREL/FRL emission factors, the vol to bio factors were modified, creating a single &amp;ldquo;Generic Chile&amp;rdquo; specie. The conversion from merchantable volume to biomass consist in a multiplication by 0.875. Considering a expansion factor of 1.75 and a density of the wood of 0.5. Given those considerations, the parameters were adjusted following the guidelines described in Boudewyn et al (2007):&lt;/p&gt;
&lt;table&gt;
&lt;thead&gt;
&lt;tr&gt;
&lt;th&gt;Volume to biomass factor&lt;/th&gt;
&lt;th&gt;Value&lt;/th&gt;
&lt;/tr&gt;
&lt;/thead&gt;
&lt;tbody&gt;
&lt;tr&gt;
&lt;td&gt;a&lt;/td&gt;
&lt;td&gt;0.875&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;b&lt;/td&gt;
&lt;td&gt;1&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;a_nonmerch&lt;/td&gt;
&lt;td&gt;0&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;b_nonmerch&lt;/td&gt;
&lt;td&gt;1&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;k_nonmerch&lt;/td&gt;
&lt;td&gt;1&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;cap_nonmerch&lt;/td&gt;
&lt;td&gt;1&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;a_sap&lt;/td&gt;
&lt;td&gt;0&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;b_sap&lt;/td&gt;
&lt;td&gt;1&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;k_sap&lt;/td&gt;
&lt;td&gt;1&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;cap_sap&lt;/td&gt;
&lt;td&gt;1&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;a1&lt;/td&gt;
&lt;td&gt;-9999999&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;a2&lt;/td&gt;
&lt;td&gt;-9999999&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;a3&lt;/td&gt;
&lt;td&gt;-9999999&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;b1&lt;/td&gt;
&lt;td&gt;-9999999&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;b2&lt;/td&gt;
&lt;td&gt;-9999999&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;b3&lt;/td&gt;
&lt;td&gt;-9999999&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;c1&lt;/td&gt;
&lt;td&gt;-9999999&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;c2&lt;/td&gt;
&lt;td&gt;-9999999&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;c3&lt;/td&gt;
&lt;td&gt;-9999999&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;min_volume&lt;/td&gt;
&lt;td&gt;0.1&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;max_volume&lt;/td&gt;
&lt;td&gt;1000&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;low_stemwood_pop&lt;/td&gt;
&lt;td&gt;0.50249&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;high_stemwood_pop&lt;/td&gt;
&lt;td&gt;0.70391&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;low_stembark_prop&lt;/td&gt;
&lt;td&gt;0.09492&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;high_stembark_prop&lt;/td&gt;
&lt;td&gt;0.11402&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;low_branches_pop&lt;/td&gt;
&lt;td&gt;0.26098&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;high_branches_pop&lt;/td&gt;
&lt;td&gt;0.14622&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;low_foliage_prop&lt;/td&gt;
&lt;td&gt;0.1416&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;high_foliage_prop&lt;/td&gt;
&lt;td&gt;0.03585&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;
&lt;ul&gt;
&lt;li&gt;The nonmerchantable factor was set to 1, so in the end the nonmerchantable trees were not considered.&lt;/li&gt;
&lt;li&gt;The sapling factor was also set to 1, so sapling-sized trees were also not considered.&lt;/li&gt;
&lt;li&gt;Very low values were chosen for the parameters to determine branches, bark and foliage, giving them a biomass almost equal to zero (the expansion factor of the Chilean forest includes them)&lt;/li&gt;
&lt;li&gt;The parameters to determine the last proportions of total tree biomass found in stem wood, stem bark, branch and foliage for live trees of all sizes were based on the Red Alder species.&lt;/li&gt;
&lt;/ul&gt;
&lt;h3 id="root-parameters"&gt;Root parameters&lt;/h3&gt;
&lt;p&gt;The root-stem ratio is 0.2869, so the root parameters were modified as followed:&lt;/p&gt;
&lt;table&gt;
&lt;thead&gt;
&lt;tr&gt;
&lt;th&gt;Root_parameter&lt;/th&gt;
&lt;th&gt;Value&lt;/th&gt;
&lt;/tr&gt;
&lt;/thead&gt;
&lt;tbody&gt;
&lt;tr&gt;
&lt;td&gt;hw_a&lt;/td&gt;
&lt;td&gt;0.2869&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;sw_a&lt;/td&gt;
&lt;td&gt;0.2869&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;hw_b&lt;/td&gt;
&lt;td&gt;1&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;frp_a&lt;/td&gt;
&lt;td&gt;0.072&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;frp_b&lt;/td&gt;
&lt;td&gt;0.354&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;frp_c&lt;/td&gt;
&lt;td&gt;-0.060211946&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;
&lt;h3 id="some-assumptions-and-procedures"&gt;Some assumptions and procedures:&lt;/h3&gt;
&lt;ul&gt;
&lt;li&gt;Only native forest was included, forest plantations are considered to be nonforest (zero carbon), this is for REDD+ accounting purposes&lt;/li&gt;
&lt;li&gt;The classifiers used for the mapping of the forests were &amp;ldquo;Forest type&amp;rdquo;, &amp;ldquo;Structure&amp;rdquo; and &amp;ldquo;Origin&amp;rdquo;&lt;/li&gt;
&lt;li&gt;The arborescent shrubland is considered a forest type&lt;/li&gt;
&lt;li&gt;The resolution used in the tiler was 0.001 degrees&lt;/li&gt;
&lt;li&gt;The disturbances take place in the same moment as the land use map is released (in this case the years 2006, 2013 and 2017), so we can see some spikes&lt;/li&gt;
&lt;/ul&gt;
&lt;h3 id="results"&gt;Results&lt;/h3&gt;
&lt;p&gt;The results of the version showed very similar biomass emissions in comparison with the FREL/FRL, when looking at the emissions for the years in the period 2001-2013:&lt;/p&gt;
&lt;ul&gt;
&lt;li&gt;CO2 emissions coming from biomass for Deforestation: 502065 tCO2/year (+3.92% in comparison with the FREL)&lt;/li&gt;
&lt;li&gt;CO2 emissions coming from biomass for Substitution: 573868 tCO2/year (+0.50% in comparison with the FREL)&lt;/li&gt;
&lt;/ul&gt;
&lt;p&gt;Although the total emissions are underestimated, this is because the DOM values are much lower (-80% less) than the values used in the FREL/FRL&lt;/p&gt;
&lt;p&gt;&lt;strong&gt;Power BI Dashboard&lt;/strong&gt;
&lt;iframe width="1100" height="750" src="https://app.powerbi.com/view?r=eyJrIjoiNjkwOTQ0M2EtMzc3MS00NDk4LTg3YmEtY2ZiNDMyNjRmNzNhIiwidCI6ImU2ZjQ4MjNiLTljZmUtNGNiYi04ZDcwLWQyNWU5YTMxZDg1OSIsImMiOjR9" frameborder="0" allowFullScreen="true"&gt;&lt;/iframe&gt;
&lt;/p&gt;
&lt;h3 id="next-steps"&gt;Next steps&lt;/h3&gt;
&lt;ul&gt;
&lt;li&gt;Include a second set of root parameters for the arborescent shrubland&lt;/li&gt;
&lt;li&gt;Include degradation and enhancement of forest carbon stocks disturbances&lt;/li&gt;
&lt;li&gt;Include custom DOM and SOC values&lt;/li&gt;
&lt;/ul&gt;</description></item><item><title>Working with CSV or TXT datasets larger than RAM in R using SQLite</title><link>https://www.juliancabezas.com/en/post/large-datasets-r-sqlite/</link><pubDate>Mon, 01 Apr 2019 00:00:00 +0000</pubDate><guid>https://www.juliancabezas.com/en/post/large-datasets-r-sqlite/</guid><description>&lt;div id="TOC"&gt;
&lt;ul&gt;
&lt;li&gt;&lt;a href="#el-problema-del-maldito-ram-y-las-bases-de-datos-enormes"&gt;The RAM problem and enormous datasets&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href="#primero-lo-primero-cuanta-memoria-tengo-disponible"&gt;First things first: how much memory is available?&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href="#csv-o-txt-a-sqllite"&gt;CSV or TXT to SQLite&lt;/a&gt;&lt;ul&gt;
&lt;li&gt;&lt;a href="#los-benditos-paquetes"&gt;The packages you need&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href="#que-tan-grande-es-mi-base-de-datos-cuales-son-sus-caracteristicas"&gt;How large is the dataset, and what does it look like?&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href="#crear-la-ruta-a-la-base-de-datos"&gt;Create the database path&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href="#la-prometida-funcion"&gt;The promised function&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href="#mover-la-csv-o-txt-a-la-base-de-datos-sqllite"&gt;Move the CSV or TXT file into SQLite&lt;/a&gt;&lt;/li&gt;
&lt;/ul&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href="#analizar-y-recoger-datos-con-dplyr"&gt;Analyse and retrieve data with dplyr&lt;/a&gt;&lt;/li&gt;
&lt;/ul&gt;
&lt;/div&gt;
&lt;div id="el-problema-del-maldito-ram-y-las-bases-de-datos-enormes" class="section level3"&gt;
&lt;h3&gt;The RAM problem and enormous datasets&lt;/h3&gt;
&lt;p&gt;I do not know whether this happens to everyone, but whenever I start a new job, I seem to get &lt;strong&gt;the worst computer in the office&lt;/strong&gt;: usually no more than 4 GB of RAM, and even 32-bit Windows if I am unlucky, as happened once. Since I use R for everything, large datasets can be difficult to handle, especially when they exceed the memory available to R. This tutorial also helps with &lt;strong&gt;enormous&lt;/strong&gt; datasets.&lt;/p&gt;
&lt;p&gt;We will check how much RAM is available to R and create a SQLite database. We can then retrieve manageable subsets and analyse them in memory, using a function I adapted from &lt;a href="https://gist.github.com/vnijs/e53a68c957747e82c2e3"&gt;this gist by vnijs&lt;/a&gt;.&lt;/p&gt;
&lt;p&gt;We will use &lt;a href="https://www.sqlite.org/index.html"&gt;SQLite&lt;/a&gt; because it is easy to configure and run locally. PostgreSQL and MySQL are alternatives, but setting them up would be a little more involved.&lt;/p&gt;
&lt;p&gt;This tutorial assumes intermediate-to-advanced R skills and some familiarity with SQL.&lt;/p&gt;
&lt;/div&gt;
&lt;div id="primero-lo-primero-cuanta-memoria-tengo-disponible" class="section level3"&gt;
&lt;h3&gt;First things first: how much memory is available?&lt;/h3&gt;
&lt;p&gt;To check the memory available to R, we will use the memory limit command shown below.&lt;/p&gt;
&lt;pre class="r"&gt;&lt;code&gt;memory.limit()&lt;/code&gt;&lt;/pre&gt;
&lt;pre&gt;&lt;code&gt;## [1] 16269&lt;/code&gt;&lt;/pre&gt;
&lt;p&gt;I have approximately 16 GB of memory available, or about 16,000 MB—the RAM installed in this computer.&lt;/p&gt;
&lt;p&gt;&lt;strong&gt;Note:&lt;/strong&gt; On a 32-bit operating system, the memory limit cannot exceed roughly 2–3 GB, regardless of how much RAM is installed.&lt;/p&gt;
&lt;/div&gt;
&lt;div id="csv-o-txt-a-sqllite" class="section level3"&gt;
&lt;h3&gt;CSV or TXT to SQLite&lt;/h3&gt;
&lt;p&gt;Suppose we have a &lt;strong&gt;HUGE&lt;/strong&gt; CSV or TXT dataset. I will use Chile’s 2017 Population and Housing Census. It is not exceptionally large, but works well for this example. You can download it from &lt;a href="http://www.ine.cl/docs/default-source/censos/censo-2017/base-de-datos/csv/microdatos-persona/microdato_censo2017-personas.rar?sfvrsn=3"&gt;Chile’s National Statistics Institute&lt;/a&gt;.&lt;/p&gt;
&lt;p&gt;Extract the RAR archive to obtain the CSV file. It is approximately 2.5 GB—not enormous, but potentially too large for R on a low-spec computer. Opening it in Excel can also be difficult.&lt;/p&gt;
&lt;p&gt;I will extract the dataset into a folder called “Censo” on the C:/ drive.&lt;/p&gt;
&lt;div class="figure"&gt;
&lt;img src="https://www.juliancabezas.com/img/archivo_censo.JPG" /&gt;
&lt;/div&gt;
&lt;div id="los-benditos-paquetes" class="section level4"&gt;
&lt;h4&gt;The packages you need&lt;/h4&gt;
&lt;p&gt;Make sure these five packages are installed:&lt;/p&gt;
&lt;ul&gt;
&lt;li&gt;&lt;strong&gt;readr&lt;/strong&gt;: to read comma-, tab- or otherwise-delimited files.&lt;/li&gt;
&lt;li&gt;&lt;strong&gt;DBI and RSQLite&lt;/strong&gt;: to communicate with databases and use SQLite.&lt;/li&gt;
&lt;li&gt;&lt;strong&gt;dplyr and dbplyr&lt;/strong&gt;: to retrieve information from the database.&lt;/li&gt;
&lt;/ul&gt;
&lt;pre class="r"&gt;&lt;code&gt;library(readr)
library(DBI)
library(RSQLite)
library(dplyr)
library(dbplyr)&lt;/code&gt;&lt;/pre&gt;
&lt;/div&gt;
&lt;div id="que-tan-grande-es-mi-base-de-datos-cuales-son-sus-caracteristicas" class="section level4"&gt;
&lt;h4&gt;How large is the dataset, and what does it look like?&lt;/h4&gt;
&lt;p&gt;First, define the dataset path and assign it to delim_file:&lt;/p&gt;
&lt;pre class="r"&gt;&lt;code&gt;delim_file &amp;lt;- &amp;quot;C:/Censo/Microdato_Censo2017-Personas.csv&amp;quot;&lt;/code&gt;&lt;/pre&gt;
&lt;p&gt;To inspect the file, read its first two lines using &lt;strong&gt;readLines&lt;/strong&gt;.&lt;/p&gt;
&lt;pre class="r"&gt;&lt;code&gt;readLines(delim_file, n=2)&lt;/code&gt;&lt;/pre&gt;
&lt;pre&gt;&lt;code&gt;## [1] &amp;quot;REGION;PROVINCIA;COMUNA;DC;AREA;ZC_LOC;ID_ZONA_LOC;NVIV;NHOGAR;PERSONAN;P07;P08;P09;P10;P10COMUNA;P10PAIS;P11;P11COMUNA;P11PAIS;P12;P12COMUNA;P12PAIS;P12A_LLEGADA;P12A_TRAMO;P13;P14;P15;P15A;P16;P16A;P16A_OTRO;P17;P18;P19;P20;P21M;P21A;P10PAIS_GRUPO;P11PAIS_GRUPO;P12PAIS_GRUPO;ESCOLARIDAD;P16A_GRUPO;REGION_15R;PROVINCIA_15R;COMUNA_15R;P10COMUNA_15R;P11COMUNA_15R;P12COMUNA_15R&amp;quot;
## [2] &amp;quot;15;152;15202;1;2;6;13225;1;1;1;1;1;73;1;98;998;3;15101;998;1;98;998;9998;98;2;4;6;2;1;2;98;7;98;98;98;98;9998;998;998;998;4;2;15;152;15202;98;15101;98&amp;quot;&lt;/code&gt;&lt;/pre&gt;
&lt;p&gt;We can now see the column names and identify the delimiter: a semicolon (;).&lt;/p&gt;
&lt;p&gt;R cannot count rows by loading a file that does not fit in memory. Instead, we will use a PowerShell command (Get-Content) on Windows, or a system command (wc) on Linux.&lt;/p&gt;
&lt;p&gt;To count the rows in the file—this takes a little while:&lt;/p&gt;
&lt;p&gt;On Windows:&lt;/p&gt;
&lt;pre class="r"&gt;&lt;code&gt;total_records &amp;lt;- system2(&amp;quot;powershell&amp;quot;, args = c(&amp;quot;Get-content&amp;quot;, delim_file, &amp;quot;|&amp;quot;, &amp;quot;Measure-Object&amp;quot;, &amp;quot;–Line&amp;quot;),
stdout = TRUE)
total_records &amp;lt;- as.numeric(gsub(&amp;quot;[^0-9]&amp;quot;, &amp;quot;&amp;quot;, paste(total_records, collapse = &amp;quot;&amp;quot;))) - 1
total_records&lt;/code&gt;&lt;/pre&gt;
&lt;pre&gt;&lt;code&gt;## [1] 17574003&lt;/code&gt;&lt;/pre&gt;
&lt;p&gt;There are 17,574,003 records, excluding the header. That matches the number of people recorded by the 2017 census.&lt;/p&gt;
&lt;p&gt;Note: Windows 10 should work without problems. On Windows 7 or 8, check your PowerShell version using &lt;a href="https://docs.microsoft.com/en-us/skypeforbusiness/set-up-your-computer-for-windows-powershell/download-and-install-windows-powershell-5-1"&gt;these Microsoft instructions&lt;/a&gt;, and install PowerShell 5.1 by following &lt;a href="https://docs.microsoft.com/en-us/powershell/scripting/install/installing-windows-powershell?view=powershell-6"&gt;this Microsoft guide&lt;/a&gt;.&lt;/p&gt;
&lt;p&gt;On Linux, it would look like this:&lt;/p&gt;
&lt;pre class="r"&gt;&lt;code&gt; total_records &amp;lt;- system2(&amp;quot;wc&amp;quot;, args = c(&amp;quot;-l&amp;quot;, delim_file), stdout = TRUE) %&amp;gt;%
sub(normalizePath(delim_file), &amp;quot;&amp;quot;, .) %&amp;gt;%
as.integer() %&amp;gt;%
{
. - 1
}&lt;/code&gt;&lt;/pre&gt;
&lt;/div&gt;
&lt;div id="crear-la-ruta-a-la-base-de-datos" class="section level4"&gt;
&lt;h4&gt;Create the database path&lt;/h4&gt;
&lt;p&gt;Choose a path for the future database file, with the extension .sqlite3.&lt;/p&gt;
&lt;p&gt;Define the database path and the table name. We will need the table name to retrieve data later. Another advantage of SQLite databases is that they can contain several tables.&lt;/p&gt;
&lt;pre class="r"&gt;&lt;code&gt;table_name_censo &amp;lt;- &amp;quot;censo2017&amp;quot;
sqlite_file_censo &amp;lt;- &amp;quot;C:/Censo/censo2017.sqlite3&amp;quot;&lt;/code&gt;&lt;/pre&gt;
&lt;/div&gt;
&lt;div id="la-prometida-funcion" class="section level4"&gt;
&lt;h4&gt;The promised function&lt;/h4&gt;
&lt;p&gt;With the total row count, file paths and delimiter (comma, semicolon, space, and so on) identified, we can use the following function:&lt;/p&gt;
&lt;pre class="r"&gt;&lt;code&gt;read_delim2sqlite &amp;lt;- function(delim_file, delim, sqlite_file, table_name, batch_size = 10000, OS = &amp;quot;Windows&amp;quot;) {
&lt;h2 id="establish-a-connection-to-the-database"&gt;establish a connection to the database&lt;/h2&gt;
&lt;p&gt;condb &amp;lt;- dbConnect(SQLite(), sqlite_file)&lt;/p&gt;
&lt;h2 id="get-the-total-number-of-records-in-the-file"&gt;get the total number of records in the file&lt;/h2&gt;
&lt;h1 id="in-unix"&gt;in Unix&lt;/h1&gt;
&lt;p&gt;if (OS == &amp;quot;Unix&amp;quot;) {
total_records &amp;lt;- system2(&amp;quot;wc&amp;quot;, args = c(&amp;quot;-l&amp;quot;, delim_file), stdout = TRUE) %&amp;gt;%
sub(normalizePath(delim_file), &amp;quot;&amp;quot;, .) %&amp;gt;%
as.integer() %&amp;gt;%
{
. - 1
}
} else {
# In windows
total_records &amp;lt;- system2(&amp;quot;powershell&amp;quot;, args = c(&amp;quot;Get-content&amp;quot;, delim_file, &amp;quot;|&amp;quot;, &amp;quot;Measure-Object&amp;quot;, &amp;quot;–Line&amp;quot;), stdout = TRUE)
total_records &amp;lt;- as.numeric(gsub(&amp;quot;[^0-9]&amp;quot;, &amp;quot;&amp;quot;, paste(total_records, collapse = &amp;quot;&amp;quot;))) - 1
}&lt;/p&gt;
&lt;p&gt;message(&amp;quot;Total records: &amp;quot;, total_records)&lt;/p&gt;
&lt;h2 id="find-the-number-of-passes-needed-based-on-size-of-each-batch"&gt;find the number of passes needed based on size of each batch&lt;/h2&gt;
&lt;p&gt;passes &amp;lt;- total_records %/% batch_size
remaining &amp;lt;- total_records %% batch_size&lt;/p&gt;
&lt;p&gt;message(&amp;quot;Total Passes to complete: &amp;quot;, passes)&lt;/p&gt;
&lt;h2 id="first-pass-determines-header-and-column-types"&gt;first pass determines header and column types&lt;/h2&gt;
&lt;p&gt;dat &amp;lt;- read_delim(delim_file, delim, n_max = batch_size, progress = FALSE) %&amp;gt;% as.data.frame()
if (nrow(problems(dat)) &amp;gt; 0) print(problems(dat))
col_names &amp;lt;- colnames(dat)
col_types &amp;lt;- c(character = &amp;quot;c&amp;quot;, numeric = &amp;quot;d&amp;quot;, integer = &amp;quot;i&amp;quot;, logical = &amp;quot;l&amp;quot;, Date = &amp;quot;c&amp;quot;) %&amp;gt;%
.[sapply(dat, class)] %&amp;gt;%
paste0(collapse = &amp;quot;&amp;quot;)&lt;/p&gt;
&lt;h2 id="write-to-database-table"&gt;write to database table&lt;/h2&gt;
&lt;p&gt;dbWriteTable(condb, table_name, dat, overwrite = TRUE)&lt;/p&gt;
&lt;h2 id="multiple-passes"&gt;multiple passes&lt;/h2&gt;
&lt;p&gt;for (p in 2:passes) {
message(&amp;quot;Pass number: &amp;quot;, p, &amp;quot;, Progress:&amp;quot;, round(p / passes, 2) * 100, &amp;quot;%&amp;quot;)
read_delim(delim_file, delim,
col_names = col_names, col_types = col_types,
skip = (p - 1) * batch_size + 1, n_max = batch_size, progress = FALSE
) %&amp;gt;%
as.data.frame() %&amp;gt;%
dbWriteTable(condb, table_name, ., append = TRUE)
}&lt;/p&gt;
&lt;p&gt;if (remaining) {
read_delim(delim_file, delim,
col_names = col_names, col_types = col_types,
skip = p * batch_size + 1, n_max = remaining, progress = FALSE
) %&amp;gt;%
as.data.frame() %&amp;gt;%
dbWriteTable(condb, table_name, ., append = TRUE)
}&lt;/p&gt;
&lt;h2 id="close-the-database-connection"&gt;close the database connection&lt;/h2&gt;
&lt;p&gt;dbDisconnect(condb)
}&lt;/code&gt;&lt;/pre&gt;&lt;/p&gt;
&lt;p&gt;The function may look intimidating, but it is fairly straightforward. It reads the file in chunks of batch_size rows using read_delim from readr, inserts each chunk into the SQLite database, and repeats the process through the final line.&lt;/p&gt;
&lt;p&gt;The function arguments are:&lt;/p&gt;
&lt;p&gt;&lt;strong&gt;delim_file&lt;/strong&gt;: path to the delimited file, in this case the census dataset. &lt;strong&gt;delim&lt;/strong&gt;: delimiter character, a semicolon here. &lt;strong&gt;sqlite_file&lt;/strong&gt;: path to the SQLite database, ending in “.sqlite3”. &lt;strong&gt;table_name&lt;/strong&gt;: name of the table within the SQLite database. &lt;strong&gt;batch_size&lt;/strong&gt;: number of records to read in each iteration. &lt;strong&gt;OS&lt;/strong&gt;: the operating system, either “Windows” or “Unix”; the Unix option was tested on Linux, not macOS.&lt;/p&gt;
&lt;/div&gt;
&lt;div id="mover-la-csv-o-txt-a-la-base-de-datos-sqllite" class="section level4"&gt;
&lt;h4&gt;Move the CSV or TXT file into SQLite&lt;/h4&gt;
&lt;p&gt;We are ready to use the function. To avoid running out of RAM, I will transfer the dataset in batches of one million rows. It should finish in 18 iterations.&lt;/p&gt;
&lt;p&gt;&lt;strong&gt;This will take a few minutes—time to make some mate!&lt;/strong&gt;&lt;/p&gt;
&lt;pre class="r"&gt;&lt;code&gt;read_delim2sqlite(delim_file, delim=&amp;quot;;&amp;quot;, sqlite_file = sqlite_file_censo,
table_name = table_name_censo, batch_size=1000000)&lt;/code&gt;&lt;/pre&gt;
&lt;pre&gt;&lt;code&gt;## Total records: 17574003&lt;/code&gt;&lt;/pre&gt;
&lt;pre&gt;&lt;code&gt;## Total Passes to complete: 17&lt;/code&gt;&lt;/pre&gt;
&lt;pre&gt;&lt;code&gt;## Parsed with column specification:
## cols(
## .default = col_integer(),
## P18 = col_character()
## )&lt;/code&gt;&lt;/pre&gt;
&lt;pre&gt;&lt;code&gt;## See spec(...) for full column specifications.&lt;/code&gt;&lt;/pre&gt;
&lt;pre&gt;&lt;code&gt;## Pass number: 2, Progress:12%&lt;/code&gt;&lt;/pre&gt;
&lt;pre&gt;&lt;code&gt;## Pass number: 3, Progress:18%&lt;/code&gt;&lt;/pre&gt;
&lt;pre&gt;&lt;code&gt;## Pass number: 4, Progress:24%&lt;/code&gt;&lt;/pre&gt;
&lt;pre&gt;&lt;code&gt;## Pass number: 5, Progress:29%&lt;/code&gt;&lt;/pre&gt;
&lt;pre&gt;&lt;code&gt;## Pass number: 6, Progress:35%&lt;/code&gt;&lt;/pre&gt;
&lt;pre&gt;&lt;code&gt;## Pass number: 7, Progress:41%&lt;/code&gt;&lt;/pre&gt;
&lt;pre&gt;&lt;code&gt;## Pass number: 8, Progress:47%&lt;/code&gt;&lt;/pre&gt;
&lt;pre&gt;&lt;code&gt;## Pass number: 9, Progress:53%&lt;/code&gt;&lt;/pre&gt;
&lt;pre&gt;&lt;code&gt;## Pass number: 10, Progress:59%&lt;/code&gt;&lt;/pre&gt;
&lt;pre&gt;&lt;code&gt;## Pass number: 11, Progress:65%&lt;/code&gt;&lt;/pre&gt;
&lt;pre&gt;&lt;code&gt;## Pass number: 12, Progress:71%&lt;/code&gt;&lt;/pre&gt;
&lt;pre&gt;&lt;code&gt;## Pass number: 13, Progress:76%&lt;/code&gt;&lt;/pre&gt;
&lt;pre&gt;&lt;code&gt;## Pass number: 14, Progress:82%&lt;/code&gt;&lt;/pre&gt;
&lt;pre&gt;&lt;code&gt;## Pass number: 15, Progress:88%&lt;/code&gt;&lt;/pre&gt;
&lt;pre&gt;&lt;code&gt;## Pass number: 16, Progress:94%&lt;/code&gt;&lt;/pre&gt;
&lt;pre&gt;&lt;code&gt;## Pass number: 17, Progress:100%&lt;/code&gt;&lt;/pre&gt;
&lt;p&gt;We only need to run this lengthy conversion once. It creates a .sqlite3 file that is slightly smaller than the CSV.&lt;/p&gt;
&lt;div class="figure"&gt;
&lt;img src="https://www.juliancabezas.com/img/archivo_censo_sqlite.JPG" alt="SQLite file created" /&gt;
&lt;p class="caption"&gt;SQLite file created&lt;/p&gt;
&lt;/div&gt;
&lt;/div&gt;
&lt;/div&gt;
&lt;div id="analizar-y-recoger-datos-con-dplyr" class="section level3"&gt;
&lt;h3&gt;Analyse and retrieve data with dplyr&lt;/h3&gt;
&lt;p&gt;To start retrieving data from the table, first connect to the database.&lt;/p&gt;
&lt;pre class="r"&gt;&lt;code&gt;con &amp;lt;- dbConnect(SQLite(), sqlite_file_censo)
con&lt;/code&gt;&lt;/pre&gt;
&lt;pre&gt;&lt;code&gt;## &amp;lt;SQLiteConnection&amp;gt;
## Path: C:\Censo\censo2017.sqlite3
## Extensions: TRUE&lt;/code&gt;&lt;/pre&gt;
&lt;p&gt;We can inspect the connection details and the table name assigned to the dataset.&lt;/p&gt;
&lt;pre class="r"&gt;&lt;code&gt;dbListTables(con)&lt;/code&gt;&lt;/pre&gt;
&lt;pre&gt;&lt;code&gt;## [1] &amp;quot;censo2017&amp;quot;&lt;/code&gt;&lt;/pre&gt;
&lt;p&gt;Create an object representing the data. It does not load the dataset into RAM: it is only a connection, which is why the total row count is not shown.&lt;/p&gt;
&lt;pre class="r"&gt;&lt;code&gt;datos_db &amp;lt;- tbl(con, &amp;quot;censo2017&amp;quot;)
datos_db&lt;/code&gt;&lt;/pre&gt;
&lt;pre&gt;&lt;code&gt;## # Source: table&amp;lt;censo2017&amp;gt; [?? x 48]
## # Database: sqlite 3.22.0 [C:\Censo\censo2017.sqlite3]
## REGION PROVINCIA COMUNA DC AREA ZC_LOC ID_ZONA_LOC NVIV NHOGAR
## &amp;lt;int&amp;gt; &amp;lt;int&amp;gt; &amp;lt;int&amp;gt; &amp;lt;int&amp;gt; &amp;lt;int&amp;gt; &amp;lt;int&amp;gt; &amp;lt;int&amp;gt; &amp;lt;int&amp;gt; &amp;lt;int&amp;gt;
## 1 15 152 15202 1 2 6 13225 1 1
## 2 15 152 15202 1 2 6 13225 3 1
## 3 15 152 15202 1 2 6 13225 3 1
## 4 15 152 15202 1 2 6 13225 3 1
## 5 15 152 15202 1 2 6 13225 3 1
## 6 15 152 15202 1 2 6 13225 9 1
## 7 15 152 15202 1 2 6 13225 9 1
## 8 15 152 15202 1 2 6 13225 9 1
## 9 15 152 15202 1 2 6 13225 9 1
## 10 15 152 15202 1 2 6 13225 10 1
## # ... with more rows, and 39 more variables: PERSONAN &amp;lt;int&amp;gt;, P07 &amp;lt;int&amp;gt;,
## # P08 &amp;lt;int&amp;gt;, P09 &amp;lt;int&amp;gt;, P10 &amp;lt;int&amp;gt;, P10COMUNA &amp;lt;int&amp;gt;, P10PAIS &amp;lt;int&amp;gt;,
## # P11 &amp;lt;int&amp;gt;, P11COMUNA &amp;lt;int&amp;gt;, P11PAIS &amp;lt;int&amp;gt;, P12 &amp;lt;int&amp;gt;, P12COMUNA &amp;lt;int&amp;gt;,
## # P12PAIS &amp;lt;int&amp;gt;, P12A_LLEGADA &amp;lt;int&amp;gt;, P12A_TRAMO &amp;lt;int&amp;gt;, P13 &amp;lt;int&amp;gt;,
## # P14 &amp;lt;int&amp;gt;, P15 &amp;lt;int&amp;gt;, P15A &amp;lt;int&amp;gt;, P16 &amp;lt;int&amp;gt;, P16A &amp;lt;int&amp;gt;,
## # P16A_OTRO &amp;lt;int&amp;gt;, P17 &amp;lt;int&amp;gt;, P18 &amp;lt;chr&amp;gt;, P19 &amp;lt;int&amp;gt;, P20 &amp;lt;int&amp;gt;,
## # P21M &amp;lt;int&amp;gt;, P21A &amp;lt;int&amp;gt;, P10PAIS_GRUPO &amp;lt;int&amp;gt;, P11PAIS_GRUPO &amp;lt;int&amp;gt;,
## # P12PAIS_GRUPO &amp;lt;int&amp;gt;, ESCOLARIDAD &amp;lt;int&amp;gt;, P16A_GRUPO &amp;lt;int&amp;gt;,
## # REGION_15R &amp;lt;int&amp;gt;, PROVINCIA_15R &amp;lt;int&amp;gt;, COMUNA_15R &amp;lt;int&amp;gt;,
## # P10COMUNA_15R &amp;lt;int&amp;gt;, P11COMUNA_15R &amp;lt;int&amp;gt;, P12COMUNA_15R &amp;lt;int&amp;gt;&lt;/code&gt;&lt;/pre&gt;
&lt;p&gt;Now we can use dplyr verbs. For example, we might retrieve a vector of ages to make a histogram. In the census dataset, age is stored in P09.&lt;/p&gt;
&lt;pre class="r"&gt;&lt;code&gt;edades &amp;lt;- datos_db %&amp;gt;%
select(P09)
&lt;p&gt;edades&lt;/code&gt;&lt;/pre&gt;&lt;/p&gt;
&lt;pre&gt;&lt;code&gt;## # Source: lazy query [?? x 1]
## # Database: sqlite 3.22.0 [C:\Censo\censo2017.sqlite3]
## P09
## &amp;lt;int&amp;gt;
## 1 73
## 2 78
## 3 78
## 4 52
## 5 44
## 6 39
## 7 35
## 8 13
## 9 12
## 10 65
## # ... with more rows&lt;/code&gt;&lt;/pre&gt;
&lt;p&gt;Everything looks fine, but the object is still a connection. R is using lazy evaluation: it performs only part of the operation to avoid unnecessary work.&lt;/p&gt;
&lt;p&gt;We can also confirm this in the environment:&lt;/p&gt;
&lt;div class="figure"&gt;
&lt;img src="https://www.juliancabezas.com/img/archivo_censo_con.JPG" /&gt;
&lt;/div&gt;
&lt;p&gt;As you can see, even the row count is unavailable:&lt;/p&gt;
&lt;pre class="r"&gt;&lt;code&gt;nrow(edades)&lt;/code&gt;&lt;/pre&gt;
&lt;pre&gt;&lt;code&gt;## [1] NA&lt;/code&gt;&lt;/pre&gt;
&lt;p&gt;To bring this column into RAM, use &lt;strong&gt;collect&lt;/strong&gt;. I recommend &lt;strong&gt;always&lt;/strong&gt; using it when you need a complete in-memory object rather than a database connection.&lt;/p&gt;
&lt;pre class="r"&gt;&lt;code&gt;edades&amp;lt;-collect(edades)
&lt;p&gt;nrow(edades)&lt;/code&gt;&lt;/pre&gt;&lt;/p&gt;
&lt;pre&gt;&lt;code&gt;## [1] 17574003&lt;/code&gt;&lt;/pre&gt;
&lt;pre class="r"&gt;&lt;code&gt;edades&lt;/code&gt;&lt;/pre&gt;
&lt;pre&gt;&lt;code&gt;## # A tibble: 17,574,003 x 1
## P09
## &amp;lt;int&amp;gt;
## 1 73
## 2 78
## 3 78
## 4 52
## 5 44
## 6 39
## 7 35
## 8 13
## 9 12
## 10 65
## # ... with 17,573,993 more rows&lt;/code&gt;&lt;/pre&gt;
&lt;p&gt;Now it works!&lt;/p&gt;
&lt;p&gt;Another example: P09 represents age and P08 represents sex. Suppose we want the average age of men (sex = 1) by region. Do not forget &lt;strong&gt;collect&lt;/strong&gt;!&lt;/p&gt;
&lt;pre class="r"&gt;&lt;code&gt;hombres_edad &amp;lt;- datos_db %&amp;gt;%
filter(P08==1) %&amp;gt;%
group_by(REGION) %&amp;gt;%
summarise(Edad_media = mean(P09))
&lt;p&gt;hombres_edad &amp;lt;- collect(hombres_edad)&lt;/code&gt;&lt;/pre&gt;&lt;/p&gt;
&lt;pre&gt;&lt;code&gt;## Warning: Missing values are always removed in SQL.
## Use `AVG(x, na.rm = TRUE)` to silence this warning&lt;/code&gt;&lt;/pre&gt;
&lt;pre class="r"&gt;&lt;code&gt;hombres_edad&lt;/code&gt;&lt;/pre&gt;
&lt;pre&gt;&lt;code&gt;## # A tibble: 16 x 2
## REGION Edad_media
## &amp;lt;int&amp;gt; &amp;lt;dbl&amp;gt;
## 1 1 32.1
## 2 2 33.0
## 3 3 33.7
## 4 4 34.6
## 5 5 35.8
## 6 6 35.7
## 7 7 35.9
## 8 8 35.0
## 9 9 35.3
## 10 10 35.1
## 11 11 34.2
## 12 12 35.9
## 13 13 34.2
## 14 14 35.7
## 15 15 33.5
## 16 16 36.8&lt;/code&gt;&lt;/pre&gt;
&lt;p&gt;We can now work with large structured datasets, retrieve subsets and perform analyses.&lt;/p&gt;
&lt;p&gt;I hope you enjoyed this tutorial. Feel free to get in touch!&lt;/p&gt;
&lt;div class="figure"&gt;
&lt;img src="https://media.giphy.com/media/3hvmlYNsOTFWE/giphy.gif" /&gt;
&lt;/div&gt;
&lt;/div&gt;</description></item><item><title>A function for parallel processing on Windows and servers (R parallelisation, part 2)</title><link>https://www.juliancabezas.com/en/post/parallel-cluster-function/</link><pubDate>Thu, 24 Jan 2019 00:00:00 +0000</pubDate><guid>https://www.juliancabezas.com/en/post/parallel-cluster-function/</guid><description>&lt;div id="TOC"&gt;
&lt;ul&gt;
&lt;li&gt;&lt;a href="#el-problema-crear-clusters-es-tedioso"&gt;The problem: creating clusters is tedious&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href="#la-solucion-una-funcion-para-clear-clusters"&gt;The solution: a function to create clusters&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href="#un-pequeno-y-reciclado-ejemplo"&gt;A small, recycled example&lt;/a&gt;&lt;/li&gt;
&lt;/ul&gt;
&lt;/div&gt;
&lt;p&gt;This short tutorial is the second part of my introduction to parallelising R on Windows. &lt;a href="https://www.juliancabezas.com/en/post/parallel-r-windows/"&gt;I recommend reading the first part&lt;/a&gt;.&lt;/p&gt;
&lt;div id="el-problema-crear-clusters-es-tedioso" class="section level2"&gt;
&lt;h2&gt;The problem: creating clusters is tedious&lt;/h2&gt;
&lt;p&gt;Creating clusters can be tedious: loading packages one by one and exporting datasets is slow, and it is easy to forget a crucial package. We want a more convenient approach. We may also be working on a server without Internet access, which complicates package installation.&lt;/p&gt;
&lt;/div&gt;
&lt;div id="la-solucion-una-funcion-para-clear-clusters" class="section level2"&gt;
&lt;h2&gt;The solution: a function to create clusters&lt;/h2&gt;
&lt;p&gt;I created a function, available in &lt;a href="https://github.com/juliancabezas/random_functions/blob/master/create_cluster"&gt;this GitHub file&lt;/a&gt;. To give credit where it is due, it is inspired by &lt;a href="https://github.com/nathanvan/parallelsugar"&gt;parallelsugar&lt;/a&gt;, with a few modifications.&lt;/p&gt;
&lt;p&gt;The function creates a cluster with &lt;em&gt;n&lt;/em&gt; nodes, specified by the &lt;em&gt;nnodes&lt;/em&gt; argument.&lt;/p&gt;
&lt;p&gt;create_cluster is designed to work on servers. When they have no Internet connection, packages can be installed in a shared folder and loaded from there. Use &lt;em&gt;libroute&lt;/em&gt; to specify the library path.&lt;/p&gt;
&lt;pre class="r"&gt;&lt;code&gt;create_cluster &amp;lt;- function(nnodes, libroute = NULL) {
&lt;h1 id="nnodes-number-of-nodes"&gt;nnodes: Number of nodes&lt;/h1&gt;
&lt;h1 id="libroute-provided-library-route-different-of-the-default-library"&gt;libroute: Provided library route (different of the default library)&lt;/h1&gt;
&lt;p&gt;tryCatch({
cl &amp;lt;- parallel::makeCluster(nnodes) # Create an empty cluster with n nodes&lt;/p&gt;
&lt;pre&gt;&lt;code&gt;loaded.package.names &amp;amp;lt;- c( # Find the loaded packages in the session
sessionInfo()$basePkgs, # base packages
names(sessionInfo()$otherPkgs)
) # aditional loaded packages
# Send the packages and the complete environment to the cluster
parallel::clusterExport(cl,
&amp;amp;quot;loaded.package.names&amp;amp;quot;, # loaded packages
envir = environment()
) # &amp;amp;quot;Environment&amp;amp;quot;
## Charge the packages in all nodes
if (is.null(libroute)) { # Use default library location
parallel::parLapply(cl, 1:length(cl), function(xx) {
lapply(loaded.package.names, function(yy) {
library(yy, character.only = TRUE)
})
})
} else {
# If we provide a library location use it (useful when working in remote machines)
parallel::parLapply(cl, 1:length(cl), function(xx) {
lapply(loaded.package.names, function(yy) {
library(yy, character.only = TRUE, lib.loc = libroute)
})
})
}
&lt;/code&gt;&lt;/pre&gt;
&lt;p&gt;})
return(cl) # The result is the cluster
}&lt;/code&gt;&lt;/pre&gt;&lt;/p&gt;
&lt;/div&gt;
&lt;div id="un-pequeno-y-reciclado-ejemplo" class="section level2"&gt;
&lt;h2&gt;A small, recycled example&lt;/h2&gt;
&lt;p&gt;In the example from the previous tutorial, we used &lt;em&gt;clusterCall&lt;/em&gt; to load &lt;em&gt;randomForest&lt;/em&gt;. Here, if the package and any other dependencies are already loaded in our session, we can export them to the nodes in one step.&lt;/p&gt;
&lt;p&gt;First, load the package and create the function:&lt;/p&gt;
&lt;pre class="r"&gt;&lt;code&gt;library(randomForest)
&lt;p&gt;rf_boot_fun &amp;lt;- function(iter, data, n_sample) {
ind &amp;lt;- sample(1:nrow(data), size = n_sample)
data_sample &amp;lt;- data[ind, ]&lt;/p&gt;
&lt;h1 id="fit-the-random-forest-model"&gt;Fit the Random Forest model&lt;/h1&gt;
&lt;p&gt;model1 &amp;lt;- randomForest(Sepal.Length ~ Petal.Length, data = data_sample)&lt;/p&gt;
&lt;h1 id="extract-the-pseudo-r2"&gt;Extract the pseudo-R2&lt;/h1&gt;
&lt;p&gt;R2 &amp;lt;- model1$rsq[length(model1$rsq)]&lt;/p&gt;
&lt;h1 id="extract-the-mse"&gt;Extract the MSE&lt;/h1&gt;
&lt;p&gt;MSE &amp;lt;- model1$mse[length(model1$mse)]
results &amp;lt;- c(&amp;quot;R2&amp;quot; = R2, &amp;quot;MSE&amp;quot; = MSE)
return(results)
}&lt;/code&gt;&lt;/pre&gt;&lt;/p&gt;
&lt;p&gt;Now I can apply the function with &lt;em&gt;parLapply&lt;/em&gt;.&lt;/p&gt;
&lt;pre class="r"&gt;&lt;code&gt;library(parallel)
&lt;h1 id="define-a-library-path-optional"&gt;Define a library path (optional)&lt;/h1&gt;
&lt;p&gt;ruta_libreria &amp;lt;- &amp;quot;C:/Users/Julian/Documents/R/win-library/3.5&amp;quot;&lt;/p&gt;
&lt;h1 id="this-can-simply-be-nuevoclus-create_cluster4"&gt;This can simply be nuevoclus&amp;lt;-create_cluster(4)&lt;/h1&gt;
&lt;p&gt;nuevoclus &amp;lt;- create_cluster(4, ruta_libreria)&lt;/p&gt;
&lt;h1 id="10000-iterations-of-our-function-using-the-iris-dataset"&gt;10,000 iterations of our function using the &amp;quot;iris&amp;quot; dataset&lt;/h1&gt;
&lt;p&gt;results &amp;lt;- parLapply(nuevoclus, 1:10000, rf_boot_fun, data = iris, n_sample = 100)&lt;/p&gt;
&lt;p&gt;stopCluster(nuevoclus) # Close the cluster; do not forget this step&lt;/p&gt;
&lt;h1 id="the-first-two-results"&gt;The first two results&lt;/h1&gt;
&lt;p&gt;head(results, n = 2)&lt;/code&gt;&lt;/pre&gt;&lt;/p&gt;
&lt;pre&gt;&lt;code&gt;## [[1]]
## R2 MSE
## 0.778244 0.174310
##
## [[2]]
## R2 MSE
## 0.6912495 0.1865890&lt;/code&gt;&lt;/pre&gt;
&lt;p&gt;Done—parallelisation is much easier now!&lt;/p&gt;
&lt;div class="figure"&gt;
&lt;img src="https://media.giphy.com/media/uTuLngvL9p0Xe/giphy.gif" /&gt;
&lt;/div&gt;
&lt;p&gt;&lt;strong&gt;I hope you find this useful!&lt;/strong&gt;&lt;/p&gt;
&lt;p&gt;&lt;strong&gt;I welcome your comments and questions!&lt;/strong&gt;&lt;/p&gt;
&lt;/div&gt;</description></item><item><title>Parallelising R on Windows with the parallel package</title><link>https://www.juliancabezas.com/en/post/parallel-r-windows/</link><pubDate>Mon, 07 Jan 2019 00:00:00 +0000</pubDate><guid>https://www.juliancabezas.com/en/post/parallel-r-windows/</guid><description>&lt;div id="TOC"&gt;
&lt;ul&gt;
&lt;li&gt;&lt;a href="#por-que-paralelizar"&gt;Why parallelise?&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href="#convertir-tu-loop-en-una-funcion"&gt;Turn your loop into a function&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href="#usando-la-funcion-lapply"&gt;Using lapply&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href="#el-paquete-parallel"&gt;The parallel package&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href="#un-ejemplo-mas-realista-usando-machine-learning"&gt;A more realistic example: machine learning&lt;/a&gt;&lt;/li&gt;
&lt;/ul&gt;
&lt;/div&gt;
&lt;div id="por-que-paralelizar" class="section level2"&gt;
&lt;h2&gt;Why parallelise?&lt;/h2&gt;
&lt;p&gt;You have been asked to run code that takes 20 hours to finish. It applies the same function to different parameters or datasets many, &lt;strong&gt;many&lt;/strong&gt; times.&lt;/p&gt;
&lt;p&gt;That might not seem like a problem, but requests often arrive &lt;del&gt;yesterday&lt;/del&gt; without 20 hours of notice. If you are wondering why your PC or workstation has 4, 8 or 16 cores, this tutorial is for you!&lt;/p&gt;
&lt;p&gt;Parallelising R processes can be difficult. Add &lt;del&gt;bad luck&lt;/del&gt; Windows to the equation, and it can feel almost impossible.&lt;/p&gt;
&lt;p&gt;In this tutorial (&lt;strong&gt;the first on this website!&lt;/strong&gt;), we will introduce parallel processing in R on Windows. We will start with the &lt;strong&gt;apply&lt;/strong&gt; family, explore the parallel package and create a Windows cluster, before finishing with a common &lt;em&gt;machine learning&lt;/em&gt; application.&lt;/p&gt;
&lt;div id="nivel-de-este-tutorial" class="section level4"&gt;
&lt;h4&gt;Tutorial level&lt;/h4&gt;
&lt;p&gt;This tutorial assumes you are an intermediate or advanced R user.&lt;/p&gt;
&lt;/div&gt;
&lt;/div&gt;
&lt;div id="convertir-tu-loop-en-una-funcion" class="section level2"&gt;
&lt;h2&gt;Turn your loop into a function&lt;/h2&gt;
&lt;p&gt;We will use the &lt;strong&gt;iris&lt;/strong&gt; dataset, familiar to many R users. It contains petal and sepal length and width measurements for three flower species.&lt;/p&gt;
&lt;pre class="r"&gt;&lt;code&gt;data(iris)
knitr::kable(head(iris))&lt;/code&gt;&lt;/pre&gt;
&lt;table&gt;
&lt;thead&gt;
&lt;tr class="header"&gt;
&lt;th align="right"&gt;Sepal.Length&lt;/th&gt;
&lt;th align="right"&gt;Sepal.Width&lt;/th&gt;
&lt;th align="right"&gt;Petal.Length&lt;/th&gt;
&lt;th align="right"&gt;Petal.Width&lt;/th&gt;
&lt;th align="left"&gt;Species&lt;/th&gt;
&lt;/tr&gt;
&lt;/thead&gt;
&lt;tbody&gt;
&lt;tr class="odd"&gt;
&lt;td align="right"&gt;5.1&lt;/td&gt;
&lt;td align="right"&gt;3.5&lt;/td&gt;
&lt;td align="right"&gt;1.4&lt;/td&gt;
&lt;td align="right"&gt;0.2&lt;/td&gt;
&lt;td align="left"&gt;setosa&lt;/td&gt;
&lt;/tr&gt;
&lt;tr class="even"&gt;
&lt;td align="right"&gt;4.9&lt;/td&gt;
&lt;td align="right"&gt;3.0&lt;/td&gt;
&lt;td align="right"&gt;1.4&lt;/td&gt;
&lt;td align="right"&gt;0.2&lt;/td&gt;
&lt;td align="left"&gt;setosa&lt;/td&gt;
&lt;/tr&gt;
&lt;tr class="odd"&gt;
&lt;td align="right"&gt;4.7&lt;/td&gt;
&lt;td align="right"&gt;3.2&lt;/td&gt;
&lt;td align="right"&gt;1.3&lt;/td&gt;
&lt;td align="right"&gt;0.2&lt;/td&gt;
&lt;td align="left"&gt;setosa&lt;/td&gt;
&lt;/tr&gt;
&lt;tr class="even"&gt;
&lt;td align="right"&gt;4.6&lt;/td&gt;
&lt;td align="right"&gt;3.1&lt;/td&gt;
&lt;td align="right"&gt;1.5&lt;/td&gt;
&lt;td align="right"&gt;0.2&lt;/td&gt;
&lt;td align="left"&gt;setosa&lt;/td&gt;
&lt;/tr&gt;
&lt;tr class="odd"&gt;
&lt;td align="right"&gt;5.0&lt;/td&gt;
&lt;td align="right"&gt;3.6&lt;/td&gt;
&lt;td align="right"&gt;1.4&lt;/td&gt;
&lt;td align="right"&gt;0.2&lt;/td&gt;
&lt;td align="left"&gt;setosa&lt;/td&gt;
&lt;/tr&gt;
&lt;tr class="even"&gt;
&lt;td align="right"&gt;5.4&lt;/td&gt;
&lt;td align="right"&gt;3.9&lt;/td&gt;
&lt;td align="right"&gt;1.7&lt;/td&gt;
&lt;td align="right"&gt;0.4&lt;/td&gt;
&lt;td align="left"&gt;setosa&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;
&lt;p&gt;Suppose a colleague wants to fit a linear model relating petals to sepals. Because the sample is small (150 individuals), you are asked to run 20,000 bootstrap iterations to examine the model coefficients and R&lt;sup&gt;2&lt;/sup&gt;.&lt;/p&gt;
&lt;p&gt;We can do this with a loop that draws a sample of 100 individuals (two-thirds of the dataset), fits a linear model, and saves the R&lt;sup&gt;2&lt;/sup&gt; and coefficients in a data frame.&lt;/p&gt;
&lt;pre class="r"&gt;&lt;code&gt;# Create an empty data frame to store the results
results &amp;lt;- data.frame()
&lt;p&gt;system.time({
for (i in 1:20000) {
# Draw a sample of 100 numbers between 1 and 150
ind &amp;lt;- sample(1:nrow(iris), size = 100)
iris_sample &amp;lt;- iris[ind, ]
# Fit a linear model using petal and sepal measurements
model1 &amp;lt;- lm(iris_sample[, &amp;quot;Sepal.Length&amp;quot;] ~ iris_sample[, &amp;quot;Petal.Length&amp;quot;])&lt;/p&gt;
&lt;pre&gt;&lt;code&gt;# Extract R2 and the coefficients
R2 &amp;amp;lt;- summary(model1)$r.squared
coef &amp;amp;lt;- coefficients(model1)
# Combine everything in the results data frame
results &amp;amp;lt;- rbind(results, c(R2, coef))
&lt;/code&gt;&lt;/pre&gt;
&lt;p&gt;}
})&lt;/code&gt;&lt;/pre&gt;&lt;/p&gt;
&lt;pre&gt;&lt;code&gt;## user system elapsed
## 20.90 0.03 20.95&lt;/code&gt;&lt;/pre&gt;
&lt;p&gt;The algorithm takes just over 20 seconds. That may be acceptable now, but what if tomorrow you need to do the same thing with half a million observations?&lt;/p&gt;
&lt;p&gt;The problem is that the loop uses only one processor core, running one iteration after another. This is not very efficient.&lt;/p&gt;
&lt;p&gt;The first step towards parallelisation is to turn the loop into a function. Here the only argument will be an iteration vector, “iter”, which lets us request more iterations.&lt;/p&gt;
&lt;pre class="r"&gt;&lt;code&gt;lm_boot_fun &amp;lt;- function(iter) {
ind &amp;lt;- sample(1:nrow(iris), size = 100)
iris_sample &amp;lt;- iris[ind, ]
model1 &amp;lt;- lm(iris_sample[, &amp;quot;Sepal.Length&amp;quot;] ~ iris_sample[, &amp;quot;Petal.Length&amp;quot;])
R2 &amp;lt;- summary(model1)$r.squared
coef &amp;lt;- coefficients(model1)
# Create a vector of results
results &amp;lt;- c(&amp;quot;R2&amp;quot; = R2, coef)
# Return the function output
return(results)
}&lt;/code&gt;&lt;/pre&gt;
&lt;p&gt;Let us test the function. As the following code shows, it returns a result regardless of the argument we pass.&lt;/p&gt;
&lt;pre class="r"&gt;&lt;code&gt;result &amp;lt;- lm_boot_fun(1)
result&lt;/code&gt;&lt;/pre&gt;
&lt;pre&gt;&lt;code&gt;## R2 (Intercept)
## 0.7554039 4.3353075
## iris_sample[, &amp;quot;Petal.Length&amp;quot;]
## 0.3999710&lt;/code&gt;&lt;/pre&gt;
&lt;pre class="r"&gt;&lt;code&gt;result &amp;lt;- lm_boot_fun(56876)
result&lt;/code&gt;&lt;/pre&gt;
&lt;pre&gt;&lt;code&gt;## R2 (Intercept)
## 0.7393922 4.3426829
## iris_sample[, &amp;quot;Petal.Length&amp;quot;]
## 0.3939433&lt;/code&gt;&lt;/pre&gt;
&lt;p&gt;Passing a vector still returns just one result, because “results” is overwritten on every iteration.&lt;/p&gt;
&lt;pre class="r"&gt;&lt;code&gt;result &amp;lt;- lm_boot_fun(1:10)
result&lt;/code&gt;&lt;/pre&gt;
&lt;pre&gt;&lt;code&gt;## R2 (Intercept)
## 0.7732318 4.2822007
## iris_sample[, &amp;quot;Petal.Length&amp;quot;]
## 0.4256894&lt;/code&gt;&lt;/pre&gt;
&lt;/div&gt;
&lt;div id="usando-la-funcion-lapply" class="section level2"&gt;
&lt;h2&gt;Using lapply&lt;/h2&gt;
&lt;p&gt;To keep all the results, we can use lapply.&lt;/p&gt;
&lt;p&gt;lapply belongs to the apply family (apply, sapply, vapply, lapply and mapply), one of R’s most powerful tools—and one I wish I had learned earlier. This tutorial is not about the whole family, so we will focus on lapply.&lt;/p&gt;
&lt;p&gt;lapply applies another function, such as the one we created, to a list or vector of arguments, and returns a list of the same length.&lt;/p&gt;
&lt;pre class="r"&gt;&lt;code&gt;iter &amp;lt;- seq(1, 20000)
system.time({
results &amp;lt;- lapply(iter, lm_boot_fun)
})&lt;/code&gt;&lt;/pre&gt;
&lt;pre&gt;&lt;code&gt;## user system elapsed
## 15.05 0.00 15.07&lt;/code&gt;&lt;/pre&gt;
&lt;p&gt;Processing time has already improved a little, but we want to make it much faster. One detail to handle is that lapply returns a list.&lt;/p&gt;
&lt;pre class="r"&gt;&lt;code&gt;head(results, n = 3)&lt;/code&gt;&lt;/pre&gt;
&lt;pre&gt;&lt;code&gt;## [[1]]
## R2 (Intercept)
## 0.7795166 4.2839537
## iris_sample[, &amp;quot;Petal.Length&amp;quot;]
## 0.4123361
##
## [[2]]
## R2 (Intercept)
## 0.7683068 4.2132627
## iris_sample[, &amp;quot;Petal.Length&amp;quot;]
## 0.4311986
##
## [[3]]
## R2 (Intercept)
## 0.7909086 4.2600338
## iris_sample[, &amp;quot;Petal.Length&amp;quot;]
## 0.4163103&lt;/code&gt;&lt;/pre&gt;
&lt;p&gt;To convert this list into a data frame, use unlist to build a matrix and then convert it. Here we specify three columns.&lt;/p&gt;
&lt;pre class="r"&gt;&lt;code&gt;results &amp;lt;- data.frame(matrix(unlist(results), ncol = 3, byrow = T))
head(results, n = 2)&lt;/code&gt;&lt;/pre&gt;
&lt;pre&gt;&lt;code&gt;## X1 X2 X3
## 1 0.7795166 4.283954 0.4123361
## 2 0.7683068 4.213263 0.4311986&lt;/code&gt;&lt;/pre&gt;
&lt;p&gt;You can find more elegant solutions in &lt;a href="https://stackoverflow.com/questions/4227223/r-list-to-data-frame"&gt;this Stack Overflow discussion&lt;/a&gt;.&lt;/p&gt;
&lt;/div&gt;
&lt;div id="el-paquete-parallel" class="section level2"&gt;
&lt;h2&gt;The parallel package&lt;/h2&gt;
&lt;p&gt;The parallel package is included with R. It provides parallel versions of functions in the apply family. Before using them, we need to find out how many cores are available.&lt;/p&gt;
&lt;pre class="r"&gt;&lt;code&gt;library(parallel)
numCores &amp;lt;- detectCores()
numCores&lt;/code&gt;&lt;/pre&gt;
&lt;pre&gt;&lt;code&gt;## [1] 8&lt;/code&gt;&lt;/pre&gt;
&lt;p&gt;We have eight logical cores! As a rule of thumb for this example, we will use half of them (four). On a Unix-like system, such as Linux or macOS, we could use mclapply. However, mclapply relies on &lt;em&gt;forking&lt;/em&gt;, which creates a copy of the process and is available only on Unix-like systems.&lt;/p&gt;
&lt;pre class="r"&gt;&lt;code&gt;iter &amp;lt;- seq(1, 20000)
system.time({
results &amp;lt;- mclapply(iter, lm_boot_fun, mc.cores = 7)
})&lt;/code&gt;&lt;/pre&gt;
&lt;pre&gt;&lt;code&gt;## Error in mclapply(iter, lm_boot_fun, mc.cores = 7): &amp;#39;mc.cores&amp;#39; &amp;gt; 1 is not supported on Windows&lt;/code&gt;&lt;/pre&gt;
&lt;pre&gt;&lt;code&gt;## Timing stopped at: 0 0 0&lt;/code&gt;&lt;/pre&gt;
&lt;p&gt;On Windows, we will create a cluster manually using makeCluster. The resulting object becomes an argument to parLapply. In this case, we will use four cores.&lt;/p&gt;
&lt;pre class="r"&gt;&lt;code&gt;cl &amp;lt;- makeCluster(4) # Create the cluster
system.time(
results &amp;lt;- parLapply(cl, iter, lm_boot_fun)
)&lt;/code&gt;&lt;/pre&gt;
&lt;pre&gt;&lt;code&gt;## user system elapsed
## 0.03 0.00 4.76&lt;/code&gt;&lt;/pre&gt;
&lt;pre class="r"&gt;&lt;code&gt;stopCluster(cl) # Close the cluster; do not forget this step&lt;/code&gt;&lt;/pre&gt;
&lt;p&gt;&lt;strong&gt;Wow!&lt;/strong&gt; We have substantially reduced the bootstrap processing time. We can enjoy a cup of mate—after shutting down the cluster with stopCluster.&lt;/p&gt;
&lt;p&gt;&lt;strong&gt;Great!&lt;/strong&gt; &lt;img src="https://media.giphy.com/media/4xpB3eE00FfBm/giphy.gif" /&gt;&lt;/p&gt;
&lt;p&gt;We could stop here, but this is a very simple example. In practice, we often use datasets stored on our computers and five to ten packages.&lt;/p&gt;
&lt;div id="complejizando-la-funcion" class="section level3"&gt;
&lt;h3&gt;Making the function more complex&lt;/h3&gt;
&lt;p&gt;Let us pretend iris is a dataset stored on our computer. We will also add two arguments to the function: the dataset and the sample size.&lt;/p&gt;
&lt;pre class="r"&gt;&lt;code&gt;# Write the iris dataset to disk
write.csv(iris, file = &amp;quot;iris.csv&amp;quot;)
&lt;h1 id="read-the-dataset-created-above"&gt;Read the dataset created above&lt;/h1&gt;
&lt;p&gt;datos_pc &amp;lt;- read.csv(&amp;quot;iris.csv&amp;quot;)&lt;/p&gt;
&lt;p&gt;lm_boot_fun2 &amp;lt;- function(iter, data, n_sample) {&lt;/p&gt;
&lt;h1 id="data-and-n_sample-are-now-function-arguments"&gt;data and n_sample are now function arguments&lt;/h1&gt;
&lt;p&gt;ind &amp;lt;- sample(1:nrow(data), size = n_sample)
data_sample &amp;lt;- data[ind, ]
model1 &amp;lt;- lm(data_sample[, &amp;quot;Sepal.Length&amp;quot;] ~ data_sample[, &amp;quot;Petal.Length&amp;quot;])
R2 &amp;lt;- summary(model1)$r.squared
coef &amp;lt;- coefficients(model1)
results &amp;lt;- c(&amp;quot;R2&amp;quot; = R2, coef)
return(results)
}&lt;/code&gt;&lt;/pre&gt;&lt;/p&gt;
&lt;p&gt;Now we can repeat the operation using the additional arguments of parLapply.&lt;/p&gt;
&lt;pre class="r"&gt;&lt;code&gt;cl &amp;lt;- makeCluster(4) # Create the cluster
system.time(
results &amp;lt;- parLapply(cl, iter, lm_boot_fun2, data = datos_pc, n_sample = 100)
)&lt;/code&gt;&lt;/pre&gt;
&lt;pre&gt;&lt;code&gt;## user system elapsed
## 0.01 0.00 4.52&lt;/code&gt;&lt;/pre&gt;
&lt;pre class="r"&gt;&lt;code&gt;stopCluster(cl) # Close the cluster; do not forget this step&lt;/code&gt;&lt;/pre&gt;
&lt;p&gt;Suppose your manager walks in and says the client wants a Random Forest—perhaps because it sounds impressive. Let us adapt the function using the randomForest package and save the pseudo-R&lt;sup&gt;2&lt;/sup&gt; and MSE.&lt;/p&gt;
&lt;pre class="r"&gt;&lt;code&gt;library(randomForest)
&lt;p&gt;rf_boot_fun &amp;lt;- function(iter, data, n_sample) {
ind &amp;lt;- sample(1:nrow(data), size = n_sample)
data_sample &amp;lt;- data[ind, ]&lt;/p&gt;
&lt;h1 id="fit-the-random-forest-model"&gt;Fit the Random Forest model&lt;/h1&gt;
&lt;p&gt;model1 &amp;lt;- randomForest(Sepal.Length ~ Petal.Length, data = data_sample)&lt;/p&gt;
&lt;h1 id="extract-the-pseudo-r2"&gt;Extract the pseudo-R2&lt;/h1&gt;
&lt;p&gt;R2 &amp;lt;- model1$rsq[length(model1$rsq)]&lt;/p&gt;
&lt;h1 id="extract-the-mse"&gt;Extract the MSE&lt;/h1&gt;
&lt;p&gt;MSE &amp;lt;- model1$mse[length(model1$mse)]
results &amp;lt;- c(&amp;quot;R2&amp;quot; = R2, &amp;quot;MSE&amp;quot; = MSE)
return(results)
}&lt;/code&gt;&lt;/pre&gt;&lt;/p&gt;
&lt;p&gt;Now let us use &lt;em&gt;parLapply&lt;/em&gt;.&lt;/p&gt;
&lt;pre class="r"&gt;&lt;code&gt;library(randomForest)
cl &amp;lt;- makeCluster(4) # Create the cluster
system.time(
results &amp;lt;- parLapply(cl, iter, rf_boot_fun, data = datos_pc, n_sample = 100)
)&lt;/code&gt;&lt;/pre&gt;
&lt;pre&gt;&lt;code&gt;## Error in checkForRemoteErrors(val): 4 nodes produced errors; first error: could not find function &amp;quot;randomForest&amp;quot;&lt;/code&gt;&lt;/pre&gt;
&lt;pre&gt;&lt;code&gt;## Timing stopped at: 0 0 0&lt;/code&gt;&lt;/pre&gt;
&lt;pre class="r"&gt;&lt;code&gt;stopCluster(cl) # Close the cluster; do not forget this step&lt;/code&gt;&lt;/pre&gt;
&lt;p&gt;&lt;strong&gt;What happened?&lt;/strong&gt; I already called &lt;em&gt;library(randomForest)&lt;/em&gt;!&lt;/p&gt;
&lt;div class="figure"&gt;
&lt;img src="https://media.giphy.com/media/DVkZv0SltX1h6/giphy.gif" /&gt;
&lt;/div&gt;
&lt;p&gt;The &lt;em&gt;randomForest&lt;/em&gt; package is loaded in our current session. However, &lt;em&gt;makeCluster&lt;/em&gt; opens &lt;em&gt;n&lt;/em&gt; new R sessions, each starting from scratch. We must load the packages in every session. In some cases, we must also export datasets and functions.&lt;/p&gt;
&lt;p&gt;To load &lt;em&gt;randomForest&lt;/em&gt; on the cluster nodes, use &lt;em&gt;clusterCall&lt;/em&gt;. It runs a function on every node; here, we use it to load the package on all of them.&lt;/p&gt;
&lt;pre class="r"&gt;&lt;code&gt;cl &amp;lt;- makeCluster(4) # Create the cluster
clusterCall(cl, function() library(randomForest))&lt;/code&gt;&lt;/pre&gt;
&lt;pre class="r"&gt;&lt;code&gt;system.time(
results &amp;lt;- parLapply(cl, iter, rf_boot_fun, data = datos_pc, n_sample = 100)
)&lt;/code&gt;&lt;/pre&gt;
&lt;pre&gt;&lt;code&gt;## user system elapsed
## 0.02 0.00 137.75&lt;/code&gt;&lt;/pre&gt;
&lt;pre class="r"&gt;&lt;code&gt;stopCluster(cl) # Close the cluster; do not forget this step&lt;/code&gt;&lt;/pre&gt;
&lt;/div&gt;
&lt;/div&gt;
&lt;div id="un-ejemplo-mas-realista-usando-machine-learning" class="section level2"&gt;
&lt;h2&gt;A more realistic example: machine learning&lt;/h2&gt;
&lt;p&gt;Now for a more realistic example: model &lt;em&gt;tuning&lt;/em&gt;. This usually means choosing the best value of one or more parameters according to a performance statistic, such as &lt;em&gt;R&lt;sup&gt;2&lt;/sup&gt;&lt;/em&gt; or &lt;em&gt;RMSE&lt;/em&gt;. Tuning uses the training or calibration dataset, while the model is evaluated on validation data.&lt;/p&gt;
&lt;p&gt;We will use iris as our training dataset and assume we have only sepal measurements, to make the task a little harder.&lt;/p&gt;
&lt;p&gt;We will use a &lt;em&gt;Support Vector Machine&lt;/em&gt; (SVM) to predict species from flower measurements. This model works well for classification, but can depend heavily on the cost and gamma parameters when using a radial kernel. We will try to optimise both.&lt;/p&gt;
&lt;p&gt;The &lt;strong&gt;e1071&lt;/strong&gt; package includes a &lt;em&gt;tune&lt;/em&gt; function that uses &lt;em&gt;cross-validation&lt;/em&gt;. Here we will use &lt;em&gt;10-fold&lt;/em&gt; cross-validation with three repetitions. This example comes from a real project with a large data matrix, where each model run took about an hour and a half. Without parallelisation, tuning would have taken days.&lt;/p&gt;
&lt;p&gt;Enough talking—let us look at the model!&lt;/p&gt;
&lt;pre class="r"&gt;&lt;code&gt;library(e1071)
svm_model &amp;lt;- svm(Species ~ Sepal.Length + Sepal.Width, data = iris)
summary(svm_model)&lt;/code&gt;&lt;/pre&gt;
&lt;pre&gt;&lt;code&gt;##
## Call:
## svm(formula = Species ~ Sepal.Length + Sepal.Width, data = iris)
##
##
## Parameters:
## SVM-Type: C-classification
## SVM-Kernel: radial
## cost: 1
## gamma: 0.5
##
## Number of Support Vectors: 86
##
## ( 10 40 36 )
##
##
## Number of Classes: 3
##
## Levels:
## setosa versicolor virginica&lt;/code&gt;&lt;/pre&gt;
&lt;p&gt;The default cost is 1 and gamma is 0.5. Let us examine the &lt;em&gt;performance&lt;/em&gt;.&lt;/p&gt;
&lt;pre class="r"&gt;&lt;code&gt;library(caret)
confusionMatrix(iris$Species, predict(svm_model))&lt;/code&gt;&lt;/pre&gt;
&lt;pre&gt;&lt;code&gt;## Confusion Matrix and Statistics
##
## Reference
## Prediction setosa versicolor virginica
## setosa 49 1 0
## versicolor 0 36 14
## virginica 0 12 38
##
## Overall Statistics
##
## Accuracy : 0.82
## 95% CI : (0.749, 0.8779)
## No Information Rate : 0.3467
## P-Value [Acc &amp;gt; NIR] : &amp;lt; 2.2e-16
##
## Kappa : 0.73
## Mcnemar&amp;#39;s Test P-Value : NA
##
## Statistics by Class:
##
## Class: setosa Class: versicolor Class: virginica
## Sensitivity 1.0000 0.7347 0.7308
## Specificity 0.9901 0.8614 0.8776
## Pos Pred Value 0.9800 0.7200 0.7600
## Neg Pred Value 1.0000 0.8700 0.8600
## Prevalence 0.3267 0.3267 0.3467
## Detection Rate 0.3267 0.2400 0.2533
## Detection Prevalence 0.3333 0.3333 0.3333
## Balanced Accuracy 0.9950 0.7980 0.8042&lt;/code&gt;&lt;/pre&gt;
&lt;p&gt;Accuracy is 0.82. In many cases that is not enough, so let us try to improve it by tuning the parameters.&lt;/p&gt;
&lt;p&gt;We create a function that returns model performance for a given cost and gamma. Notice that I call &lt;em&gt;tune&lt;/em&gt; with only one parameter combination at a time. It could accept a list of parameters, but that would be harder to parallelise.&lt;/p&gt;
&lt;pre class="r"&gt;&lt;code&gt;tune_svm_par &amp;lt;- function(costo, gamma) {
control &amp;lt;- tune.control(
nrepeat = 3, repeat.aggregate = mean,
sampling = &amp;quot;cross&amp;quot;, sampling.aggregate = mean,
sampling.dispersion = sd,
cross = 10
)
&lt;p&gt;svm_cv &amp;lt;- tune(&amp;quot;svm&amp;quot;,
train.x = iris[, c(&amp;quot;Sepal.Length&amp;quot;, &amp;quot;Sepal.Width&amp;quot;)],
train.y = iris$Species,
kernel = &amp;quot;radial&amp;quot;,
scale = TRUE,
type = &amp;quot;C-classification&amp;quot;,
tunecontrol = control,
ranges = list(cost = costo, gamma = gamma)
)&lt;/p&gt;
&lt;p&gt;return(svm_cv$performances)
}&lt;/code&gt;&lt;/pre&gt;&lt;/p&gt;
&lt;p&gt;For parallelisation, we will use &lt;em&gt;clusterMap&lt;/em&gt;, which accepts more than one varying argument. It is the equivalent of &lt;em&gt;mapply&lt;/em&gt; in the apply family.&lt;/p&gt;
&lt;pre class="r"&gt;&lt;code&gt;costos.test &amp;lt;- seq(0.2, 4, by = 0.1)
gammas.test &amp;lt;- seq(0.2, 4, by = 0.1)
&lt;p&gt;grilla &amp;lt;- expand.grid(&amp;quot;costos&amp;quot; = costos.test, &amp;quot;gammas&amp;quot; = gammas.test)
nrow(grilla)&lt;/code&gt;&lt;/pre&gt;&lt;/p&gt;
&lt;pre&gt;&lt;code&gt;## [1] 1521&lt;/code&gt;&lt;/pre&gt;
&lt;p&gt;There are 1,521 combinations. This &lt;em&gt;should&lt;/em&gt; take a while.&lt;/p&gt;
&lt;pre class="r"&gt;&lt;code&gt;t1 &amp;lt;- Sys.time()
&lt;p&gt;cl &amp;lt;- makeCluster(4) # Create the cluster
clusterCall(cl, function() library(e1071))&lt;/p&gt;
&lt;p&gt;list.errores &amp;lt;- clusterMap(
cl = cl, fun = tune_svm_par,
costo = grilla$costos,
gamma = grilla$gammas,
SIMPLIFY = FALSE, RECYCLE = FALSE
)&lt;/p&gt;
&lt;p&gt;library(dplyr)
errores.bind &amp;lt;- bind_rows(list.errores)&lt;/p&gt;
&lt;p&gt;stopCluster(cl)&lt;/p&gt;
&lt;p&gt;t2 &amp;lt;- Sys.time()&lt;/code&gt;&lt;/pre&gt;&lt;/p&gt;
&lt;pre class="r"&gt;&lt;code&gt;print(t2 - t1)&lt;/code&gt;&lt;/pre&gt;
&lt;pre&gt;&lt;code&gt;## Time difference of 52.99884 secs&lt;/code&gt;&lt;/pre&gt;
&lt;p&gt;That hardly took any time! Now let us find the cost and gamma combination with the lowest error.&lt;/p&gt;
&lt;pre class="r"&gt;&lt;code&gt;knitr::kable(subset(errores.bind, errores.bind$error == min(errores.bind$error)))&lt;/code&gt;&lt;/pre&gt;
&lt;table&gt;
&lt;thead&gt;
&lt;tr class="header"&gt;
&lt;th&gt;&lt;/th&gt;
&lt;th align="right"&gt;cost&lt;/th&gt;
&lt;th align="right"&gt;gamma&lt;/th&gt;
&lt;th align="right"&gt;error&lt;/th&gt;
&lt;th align="right"&gt;dispersion&lt;/th&gt;
&lt;/tr&gt;
&lt;/thead&gt;
&lt;tbody&gt;
&lt;tr class="odd"&gt;
&lt;td&gt;885&lt;/td&gt;
&lt;td align="right"&gt;2.8&lt;/td&gt;
&lt;td align="right"&gt;2.4&lt;/td&gt;
&lt;td align="right"&gt;0.18&lt;/td&gt;
&lt;td align="right"&gt;0.1044681&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;
&lt;p&gt;Let us test the SVM with those parameters.&lt;/p&gt;
&lt;pre class="r"&gt;&lt;code&gt;library(e1071)
svm_model &amp;lt;- svm(Species ~ Sepal.Length + Sepal.Width, data = iris, cost = 2.6, gamma = 2)
library(caret)
confusionMatrix(iris$Species, predict(svm_model))&lt;/code&gt;&lt;/pre&gt;
&lt;pre&gt;&lt;code&gt;## Confusion Matrix and Statistics
##
## Reference
## Prediction setosa versicolor virginica
## setosa 50 0 0
## versicolor 0 36 14
## virginica 0 10 40
##
## Overall Statistics
##
## Accuracy : 0.84
## 95% CI : (0.7714, 0.8947)
## No Information Rate : 0.36
## P-Value [Acc &amp;gt; NIR] : &amp;lt; 2.2e-16
##
## Kappa : 0.76
## Mcnemar&amp;#39;s Test P-Value : NA
##
## Statistics by Class:
##
## Class: setosa Class: versicolor Class: virginica
## Sensitivity 1.0000 0.7826 0.7407
## Specificity 1.0000 0.8654 0.8958
## Pos Pred Value 1.0000 0.7200 0.8000
## Neg Pred Value 1.0000 0.9000 0.8600
## Prevalence 0.3333 0.3067 0.3600
## Detection Rate 0.3333 0.2400 0.2667
## Detection Prevalence 0.3333 0.3333 0.3333
## Balanced Accuracy 1.0000 0.8240 0.8183&lt;/code&gt;&lt;/pre&gt;
&lt;p&gt;Training accuracy increased from 0.82 to 0.84. It may not look like much, but a two-percentage-point improvement from a procedure that took less than a minute—thanks to parallelisation—is a good result.&lt;/p&gt;
&lt;p&gt;&lt;strong&gt;Thank you for reading to the end of this tutorial!&lt;/strong&gt; &lt;strong&gt;I welcome your comments and questions.&lt;/strong&gt;&lt;/p&gt;
&lt;/div&gt;</description></item><item><title>Predicting Vascular Plant Richness in a Heterogeneous Wetland Using Spectral and Textural Features and a Random Forest Algorithm</title><link>https://www.juliancabezas.com/en/publication/lecs-riqueza/</link><pubDate>Sun, 01 May 2016 00:00:00 +0000</pubDate><guid>https://www.juliancabezas.com/en/publication/lecs-riqueza/</guid><description/></item><item><title>Evaluation of impacts of management in an anthropogenic peatland using field and remote sensing data</title><link>https://www.juliancabezas.com/en/publication/lecs-carbono/</link><pubDate>Tue, 01 Sep 2015 00:00:00 +0000</pubDate><guid>https://www.juliancabezas.com/en/publication/lecs-carbono/</guid><description/></item><item><title>How many measurements are needed to estimate accurate daily and annual soil respiration fluxes? Analysis using data from a temperate rainforest</title><link>https://www.juliancabezas.com/en/publication/lecs-respiracion/</link><pubDate>Tue, 01 Sep 2015 00:00:00 +0000</pubDate><guid>https://www.juliancabezas.com/en/publication/lecs-respiracion/</guid><description/></item><item><title>Reconstructing the vegetation disturbance history of a biodiversity hotspot in central chile using Landsat, Bfast and Landtrendr</title><link>https://www.juliancabezas.com/en/publication/igars-samovar/</link><pubDate>Tue, 01 Sep 2015 00:00:00 +0000</pubDate><guid>https://www.juliancabezas.com/en/publication/igars-samovar/</guid><description/></item><item><title>Experience and education</title><link>https://www.juliancabezas.com/en/experience/</link><pubDate>Mon, 01 Jan 0001 00:00:00 +0000</pubDate><guid>https://www.juliancabezas.com/en/experience/</guid><description/></item></channel></rss>