Skip to main content

Scrape: social platforms

The “List of most popular social platforms” page has three tables: a 1-row stub, a 5-row table of video-calling apps with a No. ranking column, and the 36-row table we actually want. All three share overlapping column names, so instead of matching on a header string we pick the table that has Country[b] but not No..

In R, httr2 fetches the page, rvest::html_table() parses every table on it, and Filter() keeps the one with a Country[b] column and no No. column.

library(httr2)
library(rvest)
library(readr)

html <- request("https://en.wikipedia.org/wiki/List_of_most_popular_social_platforms") |>
  req_user_agent("wikimill-project/0.1") |>
  req_perform() |>
  resp_body_html()

tables <- html_table(html)
social <- Filter(\(t) "Country[b]" %in% names(t) && !"No." %in% names(t), tables)[[1]]
names(social) <- c("name", "company", "country", "launched", "monthly_active_users", "other_metrics")
write_csv(social, "data/social_platforms.csv")
head(social)

In Python, requests fetches the page, pandas.read_html() parses every table, and next() returns the first one that passes the same check.

import io
import pandas as pd
import requests

url = "https://en.wikipedia.org/wiki/List_of_most_popular_social_platforms"
html = requests.get(url, headers={"User-Agent": "wikimill-project/0.1"}).text

tables = pd.read_html(io.StringIO(html))
social = next(t for t in tables if "Country[b]" in t.columns and "No." not in t.columns)
social.columns = ["name", "company", "country", "launched", "monthly_active_users", "other_metrics"]
social.to_csv("data/social_platforms.csv", index=False)
social.head()
NoteWhich language should I use?

Both versions pick the table by its column names, so the code is nearly line for line the same:

  • rvest::html_table() returns a list of tibbles, and Filter() tests each one’s names().

  • pandas.read_html() returns a list of DataFrames, and a generator expression tests each one’s columns.

These chunks are shown for reference only (eval: false), so rendering the site doesn’t hit Wikipedia. Run Rscript src/scrape_social.R or python src/scrape_social.py from the project root to refresh data/social_platforms.csv. See clean_social.qmd for the next step.