%%{init: {'theme': 'base', 'themeVariables': { 'fontFamily': 'monospace', "fontSize":"16px", "darkMode":false}}}%%
flowchart TB
cat(["<strong>Wikipedia category</strong><br/>page links, no table"]) -->|"i. Query"| api["MediaWiki API<br/><code>list=categorymembers</code>"]
api -->|"ii. Paginate"| json["JSON response<br/><code>cmcontinue</code>"]
json -->|"iii. Save raw"| csv[("<code>data/</code><br><code>*.csv</code>")]
csv -->|"iv. Dedupe + label"| clean["<code>dplyr</code> or <code>pandas</code> dedupe + count"]
clean -->|"v. Save clean"| cleancsv[("<code>data/</code><br><code>*_clean.csv</code>")]
style cat fill:#FFFFFF,color:#000000,stroke:#333,stroke-width:1px
style api fill:#4CBB9D,color:#FFFFFF,rx:5,ry:5
style json fill:#FFFFFF,color:#000000,stroke:#333,stroke-width:1px
style csv fill:#FFFFFF,color:#000000,stroke:#333,stroke-width:1px
style clean fill:#4CBB9D,color:#FFFFFF,rx:5,ry:5
style cleancsv fill:#FFFFFF,color:#000000,stroke:#333,stroke-width:1px
List sources
The Wikipedia pages below store their data in a bulleted list or a category rather than a table, so they need different scraping methods.
Category:Defunct shopping malls in the United States: no “List of” article exists for closed malls, so this category is the closest substitute, with Demolished and Abandoned subcategories.
List of defunct department stores of the United States: bulleted by state, with store name, location, and closure notes. The entries need manual structuring into columns.
Timeline of malware: a chronological bulleted list grouped by decade, then year. Covers ILOVEYOU, Melissa, and Code Red.
List of companies affected by the dot-com bubble: a bulleted list of companies, each with a prose description of its IPO, stock performance, or collapse.