Basic Principles and Advantages vs Stata

R differs fundamentally from Stata in several important ways:

First, R allows you to work with multiple objects in memory simultaneously (data sets, functions, libraries, vectors), while Stata limits you to a single data set in memory. This flexibility enables more complex workflows and reduces the need for constant data loading/saving.

R is much closer to “real” programming languages like Python. Functions and packages are written in the same language as the one you use to code, making it intuitive to create your own tools. This seamless integration between user code and package development creates a lower barrier to expanding your capabilities.

The R community is massive, free, and open-source, offering thousands of specialized packages compared to Stata’s more centralized ecosystem. When you encounter problems, you’ll find extensive help online through forums, blogs, and documentation, all in a generally nice and supportive vibe (vs infamous Nick Cox…).

R’s versatility allows you to do almost everything without leaving the environment. You can run other programs installed on your computer, connect to other languages (notably Python), write websites, books, or interactive dashboards. This extensibility means your skills transfer across various data science domains.

Finally, R offers a much nicer coding environment with features like code completion, syntax highlighting, proper indentation, and faster code writing. It also tends to work better with AI assistants!

The Main Packages

Principles

R was written a long time ago and comes with many commands out of the box, known as “base” commands. The packages that are always loaded include:

• base
• stats
• And some others

You can identify which package a command comes from by typing ?<command> — the package name will show in curly brackets <command> {package}.

You could do 90% of your work with these base packages, but the R community has created additional packages to make your life easier. A pedantic but useful point: base commands are generally faster/more efficient, unless a package was specifically designed for speed. Sometimes packages prioritize convenience over speed, but other times they genuinely help you do things much more efficiently. I personally mix base commands with package commands all the time.

When looking for performance, search for “functions implemented in C” as a marker of speed.

To use additional packages, you need to:

1. Install them once: install.packages("package") (and again when you update R)
2. Load them every time you start a script: library(package)

As you load libraries, they “cover” the names of functions in your environment (this means if two packages have functions with the same name, the most recently loaded package’s version will be used). You can always access the function of a package you have installed using package::command() to specify exactly which package’s implementation you want.

Input/Output (Mostly Input)

For working with various file formats, several packages excel:

data.table functions:

• fread for CSVs, JSON, and many other formats
• Sometimes fread doesn’t work well with very large files or quirky encodings, in which case you’ll need one of the packages below

tidyverse:

• readxl for Excel files
• readr for CSV files and other formats
  • More robust than fread in some cases, though sometimes slower

Data Manipulation

Two main ecosystems dominate R’s data manipulation landscape:

tidyverse (dplyr, tidyr):

• Generally considered the “easy” way, though sometimes not using the most efficient commands
• Heavy with lots of dependencies (i.e., other packages required to make it work well)
• Includes useful supplementary tools:
  • stringr for string manipulation (learn regex!)
  • lubridate for dates and time manipulation
• Tip: don’t load the whole tidyverse! Only load what you need

data.table:

• Designed for efficiency and speed
• Avoids making copies (memory-efficient), uses low-level C functions, can run on multiple threads
• Steeper learning curve, but worth the investment
• Somewhat similar to SQL syntax
• Can be used with dplyr style by running data.table in the background using dtplyr

As you can tell, data.table is my go-to :-).

Plotting

R offers different approaches to data visualization:

Base functions:

• Simple commands like plot(vector1, vector2) or hist(vector) for quick and easy plots

ggplot2:

• Part of the tidyverse ecosystem
• Takes time to learn but is incredibly powerful for complex visualizations
• I ask Google (or LLMs) for most of the things I want to do
• A helpful resource with examples: ggplot2 essentials
• Can also create maps, though I personally prefer tmap (a purely taste-based preference)

Geo

For geographic data analysis and visualization:

sf for vector data:

• Handles buffers, intersections, spatial joins

terra for raster data:

• Processes grid-based spatial data

Both may require some external packages installed on your computer (easily manageable with Homebrew).

tmap for mapping:

• Currently transitioning to version 4.0, so commands might look strange
• I highly recommend the book Geocomputation with R as both a learning resource and reference

Other Useful Features

= vs <-

• You can use both = or <- to assign a value to a variable
• I usually use = because it’s faster to type and more consistent with other programming languages

Pipes:

• You can pipe the result of a function into another function using |> (base pipe) or %>% (tidyverse pipe)
• This creates cleaner code for multiple data transformations in a row
• Makes code much more readable by reducing nested function calls

# Without pipes (nested calls, hard to read)
head(filter(select(data, x, y), x > 10), 5)

# With pipes (much clearer)
data |> 
  select(x, y) |> 
  filter(x > 10) |> 
  head(5)

Vectorization:

• Many R functions accept vectors as input and efficiently apply operations to the entire vector
• When a function doesn’t accept vectors, you can use the *apply family of functions
• This approach is typically faster than explicit loops

# Vectorized operation (fast)
x <- 1:1000
y <- x^2 + 2*x + 1

# Using apply for custom function
nums <- 1:10
squared <- sapply(nums, function(x) x^2)

Building your own functions:

• Creating custom functions in R is straightforward
• You can examine how almost any function works by typing its name without the parentheses in the console

# Define your own function
calculate_bmi <- function(weight_kg, height_m) {
  bmi <- weight_kg / (height_m^2)
  return(bmi)
}

# Use it
my_bmi <- calculate_bmi(70, 1.75)

# See function definition
head  # Type without parentheses to see how it works

Object persistence:

• To use a data set in different scripts, save objects with saveRDS() and load them with readRDS()
• This preserves all attributes of the object as they were when saved
• Please avoid saving CSVs for intermediate data whenever possible!

# Save an R object
complex_dataframe <- data.frame(x = 1:10, y = letters[1:10])
saveRDS(complex_dataframe, "data/my_dataframe.rds")

# Later, in another script
complex_dataframe <- readRDS("data/my_dataframe.rds")

Advanced Topics

Parallel Computing

One disadvantage of R is that it’s single-threaded by default, while Stata can be natively multithreaded (depending on the license). What does this mean?

• Your computer likely has 4-12 cores (processing units that perform calculations)
• Natively, R is single-threaded: operations happen one at a time (sequentially)
• As data sets grow larger, distributing operations across multiple cores can significantly reduce computation time

Solutions include:

• The future package ecosystem for “manual” parallel computing
• data.table installed from source to enable multi-threading for specific operations

Large Data Sets

R, like most programming languages, primarily works with data “in-memory” (what your RAM can hold). This approach is fast but creates limitations with very large datasets (>10 million rows).

For memory-efficient processing:

• Use packages designed to minimize RAM usage (like data.table)
• For data too large to fit in memory, consider database-inspired approaches:
  • duckdb or polars can work with “on-disk” data without loading everything into RAM
  • These tools can even handle data split across multiple files

Web Scraping

While not my specialty, R offers solid web scraping capabilities through:

• rvest, xml2, and beautifulsoup packages
• Python might be better for complex scraping projects

Learn More

Books

These resources have been invaluable in my R journey:

• R for Data Science — The perfect starting point
• Advanced R — Excellent for understanding how things work under the hood
• Geocomputation with R — Essential for spatial analysis

Websites and Blogs

For staying current and solving problems:

• data.table documentation
• Most packages have websites with “vignettes” that demonstrate functionality
• Stack Overflow (an essential resource for troubleshooting)
• R-bloggers
• Twitter/X, Bluesky (follow R developers and data scientists)

Pro Tips

Read the help files and error messages carefully! R documentation typically contains far more information than Stata’s, including:

• Expected data types for function arguments
• Return value formats
• Detailed examples

Most errors can be solved by carefully reading the error message, which usually tells you exactly what went wrong. This approach has saved me countless hours of frustration.

I hope this guide helps your transition from Stata to R! The learning curve may feel steep at first, but the flexibility and power you’ll gain are well worth the effort. If you have questions or suggestions, feel free to reach out.

The Steps

1. Open the terminal (Cmd+space, type “terminal”, press enter)

2. Install Homebrew

   /bin/bash -c "$(curl -fsSL https://raw.githubusercontent.com/Homebrew/install/HEAD/install.sh)"

   Enter your password (no characters will show: this is normal) and press enter. Say yes/press enter when required.

   Homebrew is a package manager: it lets you download programs, their dependencies (other programs they rely on to work) and helps you keep them up to date.

3. Install yt-dlp

   brew install yt-dlp

   This will install the program and all its dependencies.

4. Install ffmpeg

   brew install ffmpeg

   This is a program that converts content in different audiovisual formats.

5. Download your content (full yt-dlp options). These websites are supported.

   1. If you just want the audio, a command like the following should do the job (don’t forget to input the URL!):

      yt-dlp --extract-audio --ignore-errors --no-warnings --no-check-certificate --audio-format mp3 --embed-thumbnail --restrict-filenames -o '%(title)s.%(ext)s' [url_of_video_or_audio]

      Importantly, all the options are optional, i.e., the command will download you something! By default yt-dlp will download the video if it’s a video, and an audio file if it’s a radio show or song. Quick explanation of the options in this case:

      • --extract-audio: only extract the audio
      • --ignore-errors --no-warnings --no-check-certificate: technical stuff so that the program doesn’t crash/complains too much.
      • --audio-format mp3: convert output to mp3
      • --embed-thumbnail: use the thumbnail of the video as the “album cover”
      • --restrict-filenames: clean the filenames of weird characters and spaces
      • -o '%(title)s.%(ext)s': specify the naming of the output. Here it will take the title of the video/audio file, add a “.” and write the file extension. An example will be: Chic_–_Soup_for_One.mp3. More on outputs (for example, what variables you can extract from the video and add to the filename) can be found here.

      • Note: you can also pass a YouTube playlist URL, and it will download all the content of the playlist! I usually then modify the output format as follows:

        -o '%(playlist)s/%(playlist_index)03d_%(title)s.%(ext)s'

        This downloads each video/audio of a given playlist in a playlist-specific folder, and adds the index (i.e., order) of the video within the playlist at the start of the filename.

   2. If you want a video, here is a good command to start with:

      yt-dlp --restrict-filenames --write-subs --convert-subs srt --embed-subs --remux-video mkv -o '%(title)s.%(ext)s' [url_of_video]

      This will download the video and all subtitles and make it an mkv (container) file. It is also possible to get multiple audio streams (languages) but I have never tried: search for --audio-multistreams in the option page and on Google (adding yt-dlp to your query, of coure) ;-].

6. Controlling where the files will be downloaded. There are two different ways of doing this.
   1. [preferred method] In the terminal, navigate to the destination folder using cd path/to/folder. cd stands for “current directory” where the terminal is “working from”. Then just run one of the above commands and the audio or video file should appear in that folder.

   2. Instead of navigating to destination directory, pass the full path of the destination folder to the -o (output) option:

      -o 'full/path/to/dir/%(title)s.%(ext)s'

      The path should be from the root of your machine. To get any folder’s full path, select it in the Finder and type Cmd+Alt+c: the full path will be in your clipboard, and you just need to paste it (Cmd+v) anywhere. You can also right-click on a folder, keep Alt pressed, and click on “Copy “[directory]” as Pathname”.

I’m starting a repository of GPX routes of (usually) self-supported bike races with fixed routes. These routes can serve as inspiration for my own bike trips, or as reference for anyone out there curious about them. So far I’m following these races (links point to official websites):

• Accursed Race
• Atlas Mountain Race
• French Divide
• Hellenic Mountain Race
• Leman Gravel Challenge
• Pan Celtic Race
• Pedals de Foc Non Stop
• Silk Road Mountain Race
• Tour Divide
• Transpyrenees (Transiberica)

Planned additions: none for now — get in touch if you want to see other race routes!

Cool races that do not follow a fixed route:

• Transcontinental Race

Many other great bikepacking routes that are not races can be found on Bikepacking.com.

Files are provided in GPX (ideal for GPS devices) and GeoJSON (more practical for GIS work and mapping). To visualise the files, you can simply drag-and-drop them into QGIS — let me know if you spot any mistakes and/or files not working!

Accursed Race

• 2024: GPX, GeoJSON

Atlas Mountain Race

The route looks pretty similar from year to year… but let me know if you would like to see more years.

• 2024: GPX, GeoJSON

French Divide

• 2023: GPX, GeoJSON
• 2022: GPX, GeoJSON

Hellenic Mountain Race

• 2024: GPX, GeoJSON
• 2023: GPX, GeoJSON

Leman Gravel Challenge

• 2024: GPX, GeoJSON

Pan Celtic Race

• 2023: GPX, GeoJSON
• 2022: GPX, GeoJSON

Pedals de Foc Non Stop

Stable route.

• 2022: GPX, GeoJSON

Silk Road Mountain Race

• 2024: GPX, GeoJSON
• 2023: GPX, GeoJSON
• 2022: GPX, GeoJSON

Tour Divide

The route has been fairly stable over the years — here is the 2023 vintage.

• 2023: GPX, GeoJSON

Transpyrenees (Transiberica)

• 2023: GPX, GeoJSON

That was until my colleague Matteo took the time the other day to teach me his own protocol to install and maintain a healthy Python ecosystem. This post will walk the reader through the steps that helped me achieve Python sanity on macOS.

The Steps

These commands are to be run in the terminal. They assume that you have Homebrew installed (which I recommend — some reasons why): install instructions are available here.

Controversial Step 0: if installed, remove Anaconda completely and clean your .zshrc of all export PATHs created by Anaconda or conda. More details on Anaconda’s help page. Not taking a stance for or against Anaconda/conda here, just advising to clean up old layers of Python before laying the sturdy foundations!

1. Install Python from Homebrew brew install python
2. Make sure that the export PATH in .zshrc is set to the one prescribed by Homebrew and that there are no other export paths export PATH="/usr/local/opt/python/libexec/bin:$PATH"
3. Restart the terminal session
4. Navigate to the directory of the project cd path/to/project
5. Install virtualenv with pip pip install virtualenv
6. Create a new virtual environment virtualenv [environment-name]
7. Activate (source) the new environment source [environment-name]/bin/activate
8. Install required packages with pip pip install [package]
9. If necessary later on, upgrade packages pip install [packages] --upgrade
10. Once done managing the environment, deactivate it deactivate

Spyder setup

Accustomed to RStudio and other statistical IDEs, Spyder is a close enough cousin to ease my transition into Python. A few further steps are necessary to link it with the custom environment created above.

1. Install spyder-kernels in the virtual environment pip install spyder-kernels
   • Depending on your version of Spyder, it might ask you to install a specific version. Try with the version displayed first pip install spyder-kernels==2.3.0, for example
2. In Spyder preferences
   1. Go to the Python Interpreter tab and select “Use the following Python interpreter”.
   2. Click on the file icon
   3. In the Finder prompt, navigate to your virtual environment, the bin folder and select the pyhton file.
   4. Click OK and restart Spyder
3. Check if it works by import [package] a package installed in your virtual environment using the IPython console in Spyder

Important: if you create a new virtual environment, you will have to make sure that it has spyder-kernels installed (step 1) and re-link Spyder to that new environment (step 2).

Updating and sharing your environment

1. Export a list of packages with their version to a text file pip freeze > requirements.txt
2. Update the packages listed in requirements.txt pip install --upgrade -r requirements.txt
3. Send requirements.txt to someone, and they just need to pip install -r requirements.txt to install all the packages

If nothing works

Take a deep breath and ask your knowledgeable friends and colleagues to take a look at your computer and help you figure things out. There is light at the end of the Python tunnel!

I love Simon Caldwell’s Sunset radio show on FBI. It’s jsut the kind of electronic music that can fuel my entire day. Large portion’s of my time is spent on FBI’s website, methodically clicking away to get the next show. Unfortunately for hardcore fans like myself, FBI doesn’t provide a podcast of Simon’s shows, and we’re reduced to navigate the clumsy and laggy website.

That’s until I decided to take matters in my own hands and build my own Sunset with Simon Caldwell “radio”. In this post, I’ll start by trying to understand FBI’s website structure. I will use R, my language of choice since a few years — both because that’s what I’m most comfortable with, but also to check how it will perform in less data-centered tasks. Let’s get started!

Code

library(xml2)
library(rvest)
library(stringr)
library(data.table)
options(timeout=10000) # increase download timeout threshold

First, let’s get all links on the show’s main landing page.

mainpage = "https://fbiradio.com/945fm/programs/sunset-simon-caldwell/"
pg = read_html(mainpage)

links.all = html_attr(html_elements(pg, "a"), "href")

Check if we see any .mp3 links.

grep(".*\\.mp3", links.all, value = T)
# character(0)

No luck — that would have been too easy!

Inspecting the “play” button, we notice that clicking on it provokes an event. Looking at the network activity (in the browser’s developer tools) when clicking the play button, we see that it GETs an mp3 file which we can copy the url.

mp3_url = "https://d27rxetjl76nhc.cloudfront.net/ondemand/2022/02/07/202202071800_1_1_Sunset_-_Simon_Caldwell.mp3"

Let’s try to decompose the URL in intelligible parts: notice the year after the /ondemand/ part.

year = gsub(".+/([0-9]{4})/.+", "\\1", mp3_url)
month = gsub(".+/[0-9]{4}/([0-9]{2})/.+", "\\1", mp3_url)
day = gsub(".+/[0-9]{4}/[0-9]{2}/([0-9]{2}).+", "\\1", mp3_url)
print(paste(year, month, day))
# [1] "2022 02 07"

Using some regex, gsub identifies the numbers that represent each date element. Notice the digits in parenthesis which form the group that may be extracted with the second argument \\1. Notice as well that we can tell regex how many digits we expect in curly brackets. After many years fighting it, I finally started to “get” and enjoy regex — do yourself a favor and get your hands dirty too! Once you start getting comfortable with it’s inner workings, you’ll find out it’s pretty powerful. There are great online tools to get acquainted with the beast: I use regexr.com a lot to test regex code.

Back to our URL: notice how the date is repeated (all concatenated) after the day element we extracted. It seems to be followed by the show’s start time, 18.00. Let’s extract that with more regex.

ymd = paste0(year, month, day)
hour = gsub(str_glue(".+/{ymd}([0-9]4)_.+"), "\\1", mp3_url)
hour
# [1] "1800"

str_glue from stringr is great because it let’s you refer to variables directly in the string using curly brackets. However, since our regex code also needs the curly brackets to know how many digits to expect, I had to double the curly brackets around the 4.

The rest of the mp3 name doesn’t seem to have much to it and we may store it as is.

ymdh = paste0(year, month, day, hour)
stub = gsub(str_glue(".+/{ymdh}(.+\\.mp3)$"), "\\1", mp3_url)
stub
# [1] "_1_1_Sunset_-_Simon_Caldwell.mp3"

Let’s review the pieces we have so far:

year
month
day
ymdh
stub
# [1] "2022"
# [1] "02"
# [1] "07"
# [1] "202202071800"
# [1] "_1_1_Sunset_-_Simon_Caldwell.mp3"

We should probably add the very first part of the URL as well.

trunk = gsub(str_glue("^(.+/){year}/{month}/{day}/{ymdh}(.+\\.mp3)$"), "\\1", mp3_url)
print(str_glue("{trunk}{year}/{month}/{day}/{ymdh}{stub}"))
# https://d27rxetjl76nhc.cloudfront.net/ondemand/2022/02/07/202202071800_1_1_Sunset_-_Simon_Caldwell.mp3

Alright, now that we have the building blocks, let’s try to got other shows at earlier dates. We start manually inputing different values for the dates and see if anything comes up.

month = "01"
day = "31"

test_url = str_glue("{trunk}{year}/{month}/{day}/{year}{month}{day}{hour}{stub}")
test_url
# https://d27rxetjl76nhc.cloudfront.net/ondemand/2022/01/31/202201311800_1_1_Sunset_-_Simon_Caldwell.mp3

file_name = str_glue("{year}-{month}-{day}_simon-sunset.mp3")

# create "shows" directory if missing 
if(!dir.exists("shows")) {dir.create("shows")}
download.file(test_url, destfile = file.path(file_name), mode = "wb")

It worked! In next post, we’ll see how far back in time we can go using some date-time packages for R.

In the example below, I am matching each trip pickup coordinates (points) with one of 263 taxi zones (polygons). In the end, I want to know how many taxi trips started in each zones. To find out to which taxi zone each pair of coordinates belongs, I use st_intersect from the sf package, and geos_intersects_matrix from geos. Surprisingly, the benchmark reveals that when intersecting points and polygons sf is about 40% faster compared to the data.table+geos team.

I am in no position to say what is happening behind the scenes here, but found it interesting to report and contrast with Grant’s findings: the advantages of using data.table+geos might depend on the type of spatial operation. There might also be a better way to implement the data.table+geos version in this case that I overlooked — if you have ideas, please let me know!

library(sf)
library(geos)
library(data.table)
library(microbenchmark)
options(timeout=10000)

rm(list = ls())
gc()

url_trips = "https://s3.amazonaws.com/nyc-tlc/trip+data/yellow_tripdata_2015-01.csv"

download.file(url = url_trips, destfile = "trip_records.csv", mode = "wb") # takes some time and is pretty heavy!
trips.dt = fread("trip_records.csv", nrows = 100000) # enough rows to benchmark performance

# Keep only pickup (origin) coordinate columns
cols_drop = grep("pickup_l", names(trips.dt), value = T, invert = T)
trips.dt[, (cols_drop) := NULL]
trips.dt = trips.dt[!(pickup_longitude == 0 | pickup_latitude == 0)] # obviously invalid coordinates

# Generate geom
trips.dt[, `:=`(o_geom = geos_make_point(pickup_longitude, 
                                         pickup_latitude, 
                                         crs=4326))]

# Generate sf
trips.sf = st_as_sf(trips.dt)
trips.sf = st_transform(trips.sf, 4326)

# Get polygons to intersect
url_zones = "https://s3.amazonaws.com/nyc-tlc/misc/taxi_zones.zip"
download.file(url_zones, destfile = "zones_polygons.zip", mode = "wb")
unzip("zones_polygons.zip", exdir = file.path("zones"))

# sf version
zones.sf = st_read(file.path("zones", "taxi_zones.shp"))
zones.sf = st_transform(zones.sf, 4326)

# geom version
zones.dt = as.data.table(zones.sf)[, geom := as_geos_geometry(geometry, crs = 4326)]
zones.dt[, geometry := NULL]

# Functions to test
geos.dt = function(x) {
  geos_intersects_matrix(zones.dt[, geom], trips.dt[, o_geom])
}

sf = function(x) {
  st_intersects(zones.sf, trips.sf)
}

# Benchmark
microbenchmark(geos = geos.dt(),
               sf = sf(),
               times = 10)

# Unit: seconds
#  expr      min       lq     mean   median       uq      max neval
#  geos 3.406296 3.823752 4.388384 4.226851 4.992151 6.192083    10
#    sf 2.431363 2.493557 2.894264 2.870200 3.118318 3.941235    10

1. Foreword
2. The list
   1. Society
   2. Personal growth, psychology, philosophy
   3. Cities, architecture
   4. The environment
   5. Unclassified stuff: science, history, computers, etc.

Foreword

The holiday season is the perfect time to comfortably sit by the fire with a cup of hot Christmas tea and a great article to read. Here are few suggestions from links I collected in 2021.

The Internet is a wonderland of interesting content. Everyday, millions of people write, record and share excellent pieces of work, with the potential of expanding one’s mindset, knowledge and curiosity on many topics. Throughout the year, I encounter or receive many such fascinating links. Some I share, some I keep for later, some I read right away. But there is always too much content, and a lot ends up in a dusty digital drawer.

But it doesn’t have to be this way, so I decided to browse that dusty drawer and make a Best-Of-2021 list of links I enjoyed the most. Each link has its own value, and made the cut for different reasons: some taught me something new, others showed me a perspective I didn’t know about before, and some are just entertaining. Of course, the list is not exhaustive, missing articles I read but failed to save, for example, but it’s a pretty nice selection already.

I hope you find something in those links, too, and that it starts new conversations, or enriches old ones we’ve had. If you have your own Best Of 2021, I would love to read through it as well!

Thanks to all those who’ve sent me great content, but also to all the thriving online communities that make the Internet a great place — this list is yours.

If you have an issue accessing any article behind a paywall, you may try this tool — otherwise just get in touch and I’ll be happy to help.

The list

The list kept on growing as I was writing the post, and now contains 33 links. To help the reader, ★ identify top picks.

Society

• ★ Private Schools Have Become Truly Obscene An in-depth dive into U.S. private schools and how they worsen economic and social divides. Resonates with this 2014 article The Mobility Myth which also presents the amazing work of economist Raj Chetty and his team on social mobility.
• ★ Can Progressives Be Convinced That Genetics Matters? Behavior geneticist Kathryn Paige Harden tells the story of her research and how difficult it is to maintain a scientific middle-ground in an increasingly polarized public discourse. She has an interesting discussion with Sam Harris on his podcast (listen directly here).
• David Graeber’s Possible Worlds The life and work of anarchist anthropologist David Graeber who passed away in September 2020.
• Adam Curtis Explains It All Adam Curtis is a British filmmaker interested in the intersection between history, ideology and social psychology. His movies are like collages, carving out new ways of understanding the world and its changes. The article focuses on his latest movie.

Personal growth, psychology, philosophy

• ★ The Real Benefits Of Staying Off Social Media There is a lot written about our contemporary relationship with technology, and this article summarizes the evidence quite well, speaking from lived experience. This other one is a good follow-up article: ★ Pursue High-quality Leisure.
• For New Year’s Resolutions, Never Think You’re Too Old to Become a Beginner Don’t be afraid to start something new and be bad at it. This article also talks about the importance of being bad at something: The Mental Benefits of Being Terrible at Something.
• Learning Curve: Learning to Suffer A small essay on how suffering is part of life and how endurance sports can help us get used to the mental challenges that suffering causes.
• Is the Western way of raising kids weird? A different perspective on how children are raised around the world.
• Identify a Remarkable Trait in Anyone. Then Either Copy or Avoid It Very short, everything is in the title, but great advice to remind oneself from time to time.
• Advice for Young Scientists—and Curious People in General Selected extracts from Nobel Prize-winning biologist Peter Medawar.

Cities, architecture

• ★ How are Rome’s monuments still standing? Talks about the special type of concrete that the Romans used to build, which lasted almost 2,000 years, versus 100 years for modern concrete. The future of concrete might be in Roman recipes!
• ★ Unboxing the hidden politics of SimCity Don’t know if you ever played SimCity, but this video goes “behind the curtain” and exposes the underlying ideology behind one of the world’s most successful strategy video game.
• Why Tokyo Works The title is pretty self-explanatory: what are some of the key ingredients that make one of the biggest cities in the world work pretty well? Hint: public transportation.
• How a dearth of tourists has transformed Barcelona’s property market Looks at the effect of the pandemic on Barcelona’s real-estate market. (Use the tool linked above to access.)
• Why New York’s Billionaires’ Row Is Half Empty New York recently built some of the most crazy skyscrapers, and here’s an analysis of the economics behind such luxurious buildings.
• As birth rates fall, animals prowl in our abandoned ‘ghost villages’ How some regions are emptying and what does that mean for human settlements and biodiversity.
• Habitat 67, Montreal’s ‘failed dream’ A short history and critique of the highly innovative (at the time) Habitat 67, a housing project built for the 1967 Universal Exhibition in Montreal.
• How Early Megacities Emerged From the Jungles of Cambodia The story of old cities in South-East Asia and what led to their abandonment.
• When Is the Revolution in Architecture Coming? A critique of contemporary architecture being too cold and disconnected from people’s lives. Disclaimers: he is not an architect (at all), and I don’t necessarily agree with all his arguments, but I found his perspective interesting.

The environment

• ★ A dead battery dilemma The issues linked with increasing the share of electric vehicles, in particular producing and recycling their batteries.
• Cycling is ten times more important than electric cars for reaching net-zero cities Of course, it comes to no surprises that bicycles will save the world!
• Biodiversity decline will require millions of years to recover The biodiversity decline that we are creating is not temporary and will have dramatic consequences for centuries to come.
• Living Mediterranean Report This report documents the collapse of Mediterranean biodiversity.

Unclassified stuff: science, history, computers, etc.

• ★ Thirty-six Thousand Feet Under the Sea The fascinating adventure of 21st century explorers and how their crazy personalities collide.
• ★ 21 old films from 1895 to 1902 colorized and upscaled in 60 fps, with sound Denis Shiryaev has been uploading some of the first ever made films completely restored in 4K and colorized. I find it really amazing to watch the oldest moving images in such high quality — puts things in perspective.
• Stromatolites: The Earth’s oldest living lifeforms A story about very old living organisms and how the Earth became what it is today.
• 400-year-old Greenland shark ‘longest-living vertebrate’ Just amazing how long an animal may live: this shark was already 150 years old when the U.S. became a country.
• The digital natives are not who you think it is A humorous comment on the new “digital native” buzzword: true digital natives are the ones who know how computers work, as opposed to those who know how to scroll and make TikTok videos.
• The Body’s Most Embarrassing Organ Is an Evolutionary Marvel A history of the anus, which apparently is an amazing organ.
• Are We on the Verge of Chatting with Whales? The article starts by proposing a definition of language (always an interesting challenge) and how some biologists are using machine learning to try to communicate with sperm whales.
• CERN 2019 WorldWideWeb Rebuild A team at the CERN recreated the original World Wide Web browser. A true delight to be able to browse the Web as it originally looked like.

I am reposting this here as a personal reminder, a form of archive and to spread all this good advice wider. Hope this inspires and helps some people out there!

So I hear you’re thinking about doing a PhD in Economics! Well if that’s true, then you must be a masochist!

There are many people that are willing to offer free advice on how best to complete the enduring task of writing a PhD dissertation, and I am not going to write out my top ten tips as to how to be successful. Mainly because I don’t have such a list! This is an individualized journey, a process of self-exploration, and your development is an internal one that should not be compared to your classmates.

Here are a couple sources from successful academics that I have found to be insightful to various degrees.

1. Writing Tips for Ph.D. Students: John Cochrane writes an entertaining paper on how to write a research paper.
2. Ph.D. Thesis Research – Where do I start?, Don Davis
3. Successful Paper Seminar, Don Davis
4. An unofficial guidebook for PhD students in economics and education (plenty of extra links included)
5. Research Template: This is a template that I have created, with significant influence from Alex Eble’s research idea template (his website is also full of useful links).
6. How to structure the introduction and abstract of a paper, Claudia Sahm
7. Doing Research, Paul Niehaus
8. How to find the right questions, Esther Duflo
9. My Experienced Advice to Grads in Economics, Ariel Rubinstein
10. Pep Talk on Research, David Weil

Hope you enjoy the resources and you will do an amazing job contributing to the frontiers of Economic knowledge!

Using the right packages

When working with large datasets with millions of observations, you can quickly run out of memory. The first step is to make sur you are working with data.tables instead of data.frames. data.tables processes are much faster and more memory-efficient than most other in-memory data management packages.

Remove and garbage collect

Second, be sure to remove unused objects: use the rm(<object>) or rm(list = c('<object1>', '<object2>', ...)) if you have multiple objects. Bear in mind, however, that rm(...) just removes the link to the data stored in memory. After removing, be sure to garbage collect orphan data using gc(): this “physically” erases all unlinked objects from your RAM, actually freeing space for the next job.¹ This video is a nice and short introduction to garbage collection for non-programmers. A typical use of rm(...) and gc() in my scripts is shown below (see the Clean-up section). In the same vein, restarting the RStudio session between memory-heavy scripts might give you some extra legroom.²

### Some loading and transformations ###
# Load SF in memory from storage
grid_panel.sf = readRDS('2_data/2_constructed/the_grid_3.0.Rds')
# Transform to data.table for faster manipulation
grid_panel.dt = as.data.table(grid_panel.sf)
# Keep only some variables, keep only one row per ID
grid_work.dt = grid_panel.dt[unique(id), .(id, cell_index, geometry)]
# Change the name of variables
setnames(grid_work.dt, 'id', 'cell_id')
# Transform back to SF
grid_work.sf = st_as_sf(grid_work.dt)

### Clean-up ###
rm(list = c('grid_panel.sf', 'grid_panel.dt', 'grid_work.dt'))
gc()

Slice it up

Finally, you may slice up your data and perform the computation in a loop, and re-assemble the pieces in a final step. If you work with multi-years datasets for example, you might have to perform some operations year-by-year, and bind all the years back together once the computations are completed.

Can’t slice it? Automate script writing

Beware, however, that some operations are better executed outside a loop: if the method you use takes advantage of parallelized computation, a loop will restrict that ability. Therefore, packages like data.table (when properly installed on Mac) and r5r work at their full potential outside loops.

What if you have a serie of operations using these packages that would fit perfectly in a loop? My last trick for these cases is to write a script containing all the operations you wish to perform sequentially as functions. Then, using a loop, you can create almost instantly an arbitrary number of scripts that load your functions and perform the operations. In a master script, copy-paste the lines that run each sub-script using scource(...) (a step that could also be automated in another script: script mania!), and you get the full power of parallelized methods in a loop-y fashion. Below is an example that illustrates that last “trick”.

#~~~~~~~~~~~~~~~~~~~~~~~~~~~#
####       INTRO         ####
#~~~~~~~~~~~~~~~~~~~~~~~~~~~#

# Matrix of stations, for each year

# Each route is independently computed for each year,
# since we have yearly street network maps.

#~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~#
####       PACKAGES         ####
#~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~#

library(tidyverse)
library(sf)
library(data.table)
library(tmap)
options(java.parameters = '-Xmx8G') # can't be larger than your RAM
library(r5r)
library(tictoc)

### Clean all variables and garbage collect
rm(list = ls())
gc()

#~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~#
####       FUNCTIONS         ####
#~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~#

#~~~~~~~~~~~~~~~~~~~~~~~~~~#
####       CODE         ####
#~~~~~~~~~~~~~~~~~~~~~~~~~~#

####~~~~~~~~~~~~~~~~~ Write the content of each script ~~~~~~~~~~~~~~~~~####

# Running each script individually enables us to use the full power of the Java
# machine, computing routes in parallel across multiple cores.
# Inside a loop, only a single core would be used.

routing.path = file.path('1_scripts', '1_data-prep', '1_routing')

if(!dir.exists(routing.path)){
  dir.create(routing.path)
}

#### Content of individual scripts ####

for (y in 2013:2019){
  
  script.path = file.path(routing.path, str_glue('routing_{y}.R'))
  
  if (!file.exists(script.path)) {
    file.create(script.path)
  }
  
  script = file(script.path)
  content = c('rm(list = ls())',
              'gc()',
              "source(file.path('1_scripts', '1_data-prep', '1_routing', 'routing-fncts.R'))", # routing-fncts.R contains all the commands necessary to perform the computations.
              str_glue("r5r_full.process({y})"),
              str_glue("routing.path = '{routing.path}'")) # Load the routing path to be able to load the next routing script in master.
  writeLines(content, script)
  close(script)
  
}

####~~~~~~~~~~~~~~~~~ Run each script individually ~~~~~~~~~~~~~~~~~####

source(file.path(routing.path, 'routing_2013.R'))
source(file.path(routing.path, 'routing_2014.R'))
source(file.path(routing.path, 'routing_2015.R'))
source(file.path(routing.path, 'routing_2016.R'))
source(file.path(routing.path, 'routing_2017.R'))
source(file.path(routing.path, 'routing_2018.R'))
source(file.path(routing.path, 'routing_2019.R'))

# Clean up
rJava::.jgc(R.gc = TRUE)
gc()

In R, the excellent areal package lets us do areal interpolation and control the parameters described below. The (also great) sf package does have an st_interpolate_aw method, but it lacks some features areal implements.

The “distribution” of values across spatial units takes two parameters (i.e., takes place in two dimensions), described below.

Extensive vs Intensive

Extensive distribution spreads the value of a variable across the overlapping features. This used for count variables (population, number of trees, etc).

Intensive distribution produces a spatially weighted average of the variable across the overlapping features. This is used for rates, averages and other already transformed variables (asthma rate, median income, etc)

For extensive only: sum vs total

Sum assumes that 100% of the source data should be distributed to the target features. The total area of the source is thus $A_j=\sum A_{ij}$, the sum of all the overlapping (intersected) areas.

In practice, that method is used when features do not perfectly overlap, but one still wishes to distribute the entirety of a feature’s value to the overlapping features (because of an imperfectly matching datasets, for example).

Total assumes that, if a source feature is not 100% overlapped by target features, then only the overlapped proportion should be distributed. For example, “if a source feature is only covered by 99.88% of the target features, only 99.88% of the source target’s data should be allocated to target features in the interpolation”. $A_j$ is thus the original area of the source feature, not the sum of the intersected areas as in sum.

In practice, that method is used when one is certain the features overlap perfectly, or when only a proportion of the variable (relative to the overlapped area) needs to be allocated, because that’s what makes sense in that particular context.

This is the “distribution dimension” where areal’s aw_interpolate() differs from sf’s st_interpolate_aw(): the former offers both “sum” and “total” options, while the latter only supports the “total” distribution. areal has the advantage of making the distinction explicit, but will yield the same results as sf if the “total” option is selected.

More details

The areal homepage has detailed explanations and visual descriptions of the steps involved in areal interpolation. See also the reference on st_interpolate_aw() for more details on sf’s implementation.