Skip to main content

Clean: malls

A mall in the “Demolished” or “Abandoned” subcategory from scrape_malls.qmd is also a member of the parent “Defunct” category, so each title can appear more than once. The code below keeps the most specific status for each mall and counts how many malls fall into each one.

%%{init: {'theme': 'base', 'themeVariables': { 'fontFamily': 'monospace', "fontSize":"16px", "darkMode":false}}}%%

flowchart TB

    raw[("<code>data/defunct_malls.csv</code>")] -->|"i. Order"| status["<code>status</code> factor<br/>Demolished, Abandoned, Defunct"]
    status -->|"ii. Sort"| arr["Most specific<br/>status first"]
    arr -->|"iii. Deduplicate"| dist["One row per <code>title</code>"]
    dist -->|"iv. Save"| clean[("<code>data/defunct_malls_clean.csv</code>")]

    style raw fill:#FFFFFF,color:#000000,stroke:#333,stroke-width:1px
    style clean fill:#FFFFFF,color:#000000,stroke:#333,stroke-width:1px
    style status fill:#4CBB9D,color:#FFFFFF,rx:5,ry:5
    style arr fill:#4CBB9D,color:#FFFFFF,rx:5,ry:5
    style dist fill:#4CBB9D,color:#FFFFFF,rx:5,ry:5

library(dplyr)
library(readr)

malls_clean <- read_csv("data/defunct_malls.csv") |>
  mutate(status = factor(status, levels = c("Demolished", "Abandoned", "Defunct"))) |>
  arrange(status) |>
  distinct(title, .keep_all = TRUE)

write_csv(malls_clean, "data/defunct_malls_clean.csv")

malls_clean |> count(status, sort = TRUE)
# A tibble: 3 × 2
  status         n
  <fct>      <int>
1 Demolished   416
2 Defunct      278
3 Abandoned     25
import pandas as pd

malls = pd.read_csv("data/defunct_malls.csv")
malls["status"] = pd.Categorical(
    malls["status"], categories=["Demolished", "Abandoned", "Defunct"], ordered=True
)
malls_clean = (
    malls.sort_values("status", kind="stable")
    .drop_duplicates(subset="title")
)
malls_clean.to_csv("data/defunct_malls_clean.csv", index=False)
malls_clean["status"].value_counts()
status
Demolished    416
Defunct       278
Abandoned      25
Name: count, dtype: int64
NoteEither works

An ordered factor in R and an ordered Categorical in pandas both sort the statuses by specificity. arrange() is always stable; in pandas, sort_values() needs kind="stable" so ties keep their original order before drop_duplicates().

See viz_malls.qmd for the chart built on this.