Skip to main content

List sources

The Wikipedia pages below store their data in a bulleted list or a category rather than a table, so they need different scraping methods.

%%{init: {'theme': 'base', 'themeVariables': { 'fontFamily': 'monospace', "fontSize":"16px", "darkMode":false}}}%%

flowchart TB
    cat(["<strong>Wikipedia category</strong><br/>page links, no table"]) -->|"i. Query"| api["MediaWiki API<br/><code>list=categorymembers</code>"]
    api -->|"ii. Paginate"| json["JSON response<br/><code>cmcontinue</code>"]
    json -->|"iii. Save raw"| csv[("<code>data/</code><br><code>*.csv</code>")]
    csv -->|"iv. Dedupe + label"| clean["<code>dplyr</code> or <code>pandas</code> dedupe + count"]
    clean -->|"v. Save clean"| cleancsv[("<code>data/</code><br><code>*_clean.csv</code>")]

    style cat fill:#FFFFFF,color:#000000,stroke:#333,stroke-width:1px
    style api fill:#4CBB9D,color:#FFFFFF,rx:5,ry:5
    style json fill:#FFFFFF,color:#000000,stroke:#333,stroke-width:1px
    style csv fill:#FFFFFF,color:#000000,stroke:#333,stroke-width:1px
    style clean fill:#4CBB9D,color:#FFFFFF,rx:5,ry:5
    style cleancsv fill:#FFFFFF,color:#000000,stroke:#333,stroke-width:1px

Scraped topics