library(httr2)
library(rvest)
library(readr)
html <- request("https://en.wikipedia.org/wiki/List_of_recessions_in_the_United_States") |>
req_user_agent("wikimill-project/0.1") |>
req_perform() |>
resp_body_html()
tables <- html_table(html)
recessions <- Filter(\(t) any(grepl("GDP decline", names(t))), tables)[[1]]
names(recessions) <- c(
"name", "period_range", "duration", "time_since_previous",
"peak_unemployment", "gdp_decline", "characteristics"
)
write_csv(recessions, "data/recessions.csv")
head(recessions)Scrape: recessions
The List of recessions in the United States table has a flat header, so renaming the columns is the only reshaping it needs.
In R, httr2 fetches the page, rvest::html_table() parses every table on it, and Filter() keeps the one with “GDP decline” in its header.
In Python, requests fetches the page, then pandas.read_html() finds every table and match= picks out the same one.
import io
import pandas as pd
import requests
url = "https://en.wikipedia.org/wiki/List_of_recessions_in_the_United_States"
html = requests.get(url, headers={"User-Agent": "wikimill-project/0.1"}).text
tables = pd.read_html(io.StringIO(html), match="GDP decline")
recessions = tables[0]
recessions.columns = [
"name", "period_range", "duration", "time_since_previous",
"peak_unemployment", "gdp_decline", "characteristics"
]
recessions.to_csv("data/recessions.csv", index=False)
recessions.head()Both versions take about the same code. One difference shows up in the raw text: html_table() joins text across a <br> tag without a space (3 years1 month), while read_html() inserts one. The cleaning step only parses period_range and the percentages, so both CSVs produce the same cleaned table.
The code above is shown for reference only. Run Rscript src/scrape_recessions.R or python src/scrape_recessions.py from the project root to refresh data/recessions.csv. See clean_recessions.qmd for the next step.