library(httr2)
library(dplyr)
library(readr)
fetch_category_members <- function(category) {
resps <- request("https://en.wikipedia.org/w/api.php") |>
req_user_agent("wikimill-project/0.1") |>
req_url_query(
action = "query", list = "categorymembers",
cmtitle = paste0("Category:", category), cmlimit = 500,
format = "json"
) |>
req_perform_iterative(
next_req = iterate_with_cursor(
"cmcontinue",
resp_param_value = \(resp) resp_body_json(resp)$continue$cmcontinue
),
max_reqs = Inf
)
titles <- resps_data(resps, \(resp) {
vapply(resp_body_json(resp)$query$categorymembers, \(m) m$title, character(1))
})
tibble(title = titles)
}
categories <- c(
"Defunct_shopping_malls_in_the_United_States" = "Defunct",
"Demolished_shopping_malls_in_the_United_States" = "Demolished",
"Abandoned_shopping_malls_in_the_United_States" = "Abandoned"
)
malls <- bind_rows(lapply(names(categories), \(cat) {
fetch_category_members(cat) |> mutate(status = categories[[cat]])
}))
write_csv(malls, "data/defunct_malls.csv")
head(malls)Scrape: malls
A Wikipedia category has no table (it’s just a list of page links), so rvest::html_table() and pandas.read_html() have nothing to parse. The MediaWiki API’s categorymembers endpoint returns the same list as structured JSON instead, and handles pagination with a cmcontinue token for categories over 500 members.
The code below fetches the defunct malls category and its two subcategories, then labels each page with the category it came from.
In R, httr2 sends the API request, and req_perform_iterative() with iterate_with_cursor() follows the cmcontinue token until the category runs out of pages. resps_data() then pulls the titles out of every response.
In Python, requests sends the API request inside a while loop that copies the cmcontinue token into the next request until a response has no continue key.
import pandas as pd
import requests
API = "https://en.wikipedia.org/w/api.php"
def fetch_category_members(category):
params = {
"action": "query", "list": "categorymembers",
"cmtitle": f"Category:{category}", "cmlimit": 500,
"format": "json"
}
members = []
while True:
resp = requests.get(API, params=params, headers={"User-Agent": "wikimill-project/0.1"}).json()
members.extend(resp["query"]["categorymembers"])
if "continue" not in resp:
break
params["cmcontinue"] = resp["continue"]["cmcontinue"]
return pd.DataFrame(members)[["title"]]
categories = {
"Defunct_shopping_malls_in_the_United_States": "Defunct",
"Demolished_shopping_malls_in_the_United_States": "Demolished",
"Abandoned_shopping_malls_in_the_United_States": "Abandoned",
}
malls = pd.concat(
[fetch_category_members(cat).assign(status=label) for cat, label in categories.items()],
ignore_index=True
)
malls.to_csv("data/defunct_malls.csv", index=False)
malls.head()httr2 has the edge
httr2::req_perform_iterative()withiterate_with_cursor()handles thecmcontinueloop for us, andhttr2also shipsreq_throttle()andreq_retry()for polite, rate-limited requests.With
requests, the pagination loop, throttling, and retries are all code we write ourselves.
These chunks are shown for reference only (eval: false), so rendering the site doesn’t hit Wikipedia. Run Rscript src/scrape_malls.R or python src/scrape_malls.py from the project root to refresh data/defunct_malls.csv. See clean_malls.qmd for the next step.