Skip to main content

Clean: recessions

The recessions CSV from scrape_recessions.qmd stores its numbers as text. The code below extracts the start year and the percentages, labels the dot-com, financial crisis, and COVID-19 recessions by era, and keeps recessions since 1945 with a recorded GDP decline.

It also tags each recession with the generation entering the job market when it happened. Each generation’s entry window is its Pew Research birth-year bracket plus 22 (the typical career-entry age), and a recession is joined to every generation whose window overlaps its start and end years. A recession that straddled a boundary would show up under both generations; none of the post-1945 recessions do.

The generation table also carries the columns for drawing each entry window as a shaded band on the timeline:

It also carries each generation’s working years, drawn as a horizontal bar above the bands. A bar starts when the generation’s oldest members turned 18 (the end of high school) and ends when its youngest members reach the Social Security full retirement age for their birth year:

period_range holds the start and end month as text (e.g., “December 2007 – June 2009”, sometimes with footnote markers). The code strips the footnotes, then parses each month into start_date and end_date (the first of the month, matching NBER’s monthly peak and trough dates). start_year and end_year come from those dates, and label combines the name with the year range for the chart.

The Wikipedia names mostly repeat the dates (“Recession of 1960–1961”), so we replace name with one word for the main cause or context, drawn from each row’s characteristics text:

%%{init: {'theme': 'base', 'themeVariables': { 'fontFamily': 'monospace', "fontSize":"16px", "darkMode":false}}}%%

flowchart TB

    raw[("<code>data/recessions.csv</code>")] -->|"i. Parse"| dates["<code>start_date</code>, <code>end_date</code><br/>from <code>period_range</code>"]
    dates -->|"ii. Rename"| names["One-word <code>name</code><br/>and <code>label</code>"]
    names -->|"iii. Extract"| pct["GDP decline, unemployment,<br/>and <code>era</code>"]
    pct -->|"iv. Filter"| filt["Since 1945 with<br/>a GDP decline"]
    gen{{"Generations table<br/>birth years + 18 and + 22"}} -->|"v. Overlap join"| join["Recession matched to<br/>generation entry window"]
    filt -->|"v. Overlap join"| join
    join -->|"vi. Save"| clean[("<code>data/recessions_clean.csv</code>")]

    style raw fill:#FFFFFF,color:#000000,stroke:#333,stroke-width:1px
    style clean fill:#FFFFFF,color:#000000,stroke:#333,stroke-width:1px
    style gen fill:#FFFFFF,color:#000000,stroke:#333,stroke-width:1px
    style dates fill:#4CBB9D,color:#FFFFFF,rx:5,ry:5
    style names fill:#4CBB9D,color:#FFFFFF,rx:5,ry:5
    style pct fill:#4CBB9D,color:#FFFFFF,rx:5,ry:5
    style filt fill:#4CBB9D,color:#FFFFFF,rx:5,ry:5
    style join fill:#4CBB9D,color:#FFFFFF,rx:5,ry:5

library(dplyr)
library(readr)
library(stringr)

generations <- tibble(
  generation = c(
    "Greatest Generation", "Silent Generation", "Baby Boomers",
    "Generation X", "Millennials", "Generation Z"
  ),
  birth_start = c(1901L, 1928L, 1946L, 1965L, 1981L, 1997L),
  birth_end = c(1927L, 1945L, 1964L, 1980L, 1996L, 2012L)
) |>
  mutate(
    entry_start = birth_start + 22L,
    entry_end = birth_end + 22L,
    generation_label = paste0(generation, "\nentered ", entry_start, "–", entry_end),
    band_label = sub(" ", "\n", generation),
    band_start = as.Date(paste0(pmax(entry_start, 1944L), "-01-01")),
    band_end = as.Date(paste0(pmin(entry_end, 2025L) + 1L, "-01-01")),
    band_mid = band_start + (band_end - band_start) / 2,
    shade = row_number() %% 2 == 0,
    workforce_start = birth_start + 18L,
    workforce_end = birth_end + 18L,
    retirement_age = case_when(birth_end <= 1937 ~ 65L, birth_end <= 1954 ~ 66L, TRUE ~ 67L),
    retirement_year = birth_end + retirement_age,
    bar_start = as.Date(paste0(pmax(workforce_start, 1944L), "-01-01")),
    bar_end = as.Date(paste0(pmin(retirement_year, 2025L) + 1L, "-01-01")),
    generation = factor(generation, levels = generation),
    generation_label = factor(generation_label, levels = generation_label)
  )

recessions <- read_csv("data/recessions.csv") |>
  mutate(
    period_range = str_remove_all(period_range, "\\[\\d+\\]"),
    start_date = as.Date(paste("1", str_extract(period_range, "^[A-Za-z]+ \\d{4}")), "%d %B %Y"),
    end_date = as.Date(paste("1", str_extract(period_range, "[A-Za-z]+ \\d{4}$")), "%d %B %Y"),
    start_year = as.integer(format(start_date, "%Y")),
    end_year = as.integer(format(end_date, "%Y")),
    name = case_when(
      start_year == 1945 ~ "Demobilization",
      start_year == 1948 ~ "Tightening",
      start_year == 1953 ~ "Korea",
      start_year == 1957 ~ "Eisenhower",
      start_year == 1960 ~ "Rates",
      start_year == 1969 ~ "Vietnam",
      start_year == 1973 ~ "Oil",
      start_year == 1980 ~ "Volcker",
      start_year == 1981 ~ "Iran",
      start_year == 1990 ~ "Gulf",
      start_year == 2001 ~ "Dotcom",
      start_year == 2007 ~ "Subprime",
      start_year == 2020 ~ "Pandemic",
      TRUE ~ name
    ),
    gdp_decline_pct = as.numeric(str_extract(gdp_decline, "[\\d.]+")),
    peak_unemployment_pct = as.numeric(str_extract(peak_unemployment, "[\\d.]+")),
    era = case_when(
      start_year %in% 2001 ~ "Dot-com bust",
      start_year %in% 2007:2009 ~ "Financial crisis",
      start_year %in% 2020 ~ "COVID-19",
      TRUE ~ "Other"
    ),
    label = if_else(
      start_year == end_year,
      paste0(name, " (", start_year, ")"),
      paste0(name, " (", start_year, "–", end_year, ")")
    )
  ) |>
  filter(start_year >= 1945, !is.na(gdp_decline_pct)) |>
  inner_join(generations, by = join_by(overlaps(start_year, end_year, entry_start, entry_end)))
write_csv(recessions, "data/recessions_clean.csv")
recessions
## # A tibble: 13 × 32
##    name  period_range duration time_since_previous peak_unemployment gdp_decline
##    <chr> <chr>        <chr>    <chr>               <chr>             <chr>      
##  1 Demo… February 19… 8 months 6 years 8 months    5.2%[59] (1946)   −12.7%     
##  2 Tigh… November 19… 11 mont… 3 years 1 month     7.9% (October 19… −1.7%      
##  3 Korea July 1953 –… 10 mont… 3 years 9 months    6.1% (September … −2.6%      
##  4 Eise… August 1957… 8 months 3 years 3 months    7.5% (July 1958)  −3.7%      
##  5 Rates April 1960 … 10 mont… 2 years             7.1% (May 1961)   −1.6%      
##  6 Viet… December 19… 11 mont… 8 years 10 months   6.1% (December 1… −0.6%      
##  7 Oil   November 19… 1 year … 3 years             9.0% (May 1975)   −3.2%      
##  8 Volc… January 198… 6 months 4 years 10 months   7.8% (July 1980)  −2.2%      
##  9 Iran  July 1981 –… 1 year … 1 year              10.8% (November … −2.7%      
## 10 Gulf  July 1990 –… 8 months 7 years 8 months    7.8% (June 1992)  −1.4%      
## 11 Dotc… March 2001 … 8 months 10 years            6.3% (June 2003)  −0.3%      
## 12 Subp… December 20… 1 year … 6 years 1 month     10.0% (October 2… −5.1%[79]  
## 13 Pand… February 20… 2 months 10 years 8 months   14.7% (April 202… −19.2%[86] 
## # ℹ 26 more variables: characteristics <chr>, start_date <date>,
## #   end_date <date>, start_year <int>, end_year <int>, gdp_decline_pct <dbl>,
## #   peak_unemployment_pct <dbl>, era <chr>, label <chr>, generation <fct>,
## #   birth_start <int>, birth_end <int>, entry_start <int>, entry_end <int>,
## #   generation_label <fct>, band_label <chr>, band_start <date>,
## #   band_end <date>, band_mid <date>, shade <lgl>, workforce_start <int>,
## #   workforce_end <int>, retirement_age <int>, retirement_year <int>, …
import pandas as pd

generations = pd.DataFrame({
    "generation": [
        "Greatest Generation", "Silent Generation", "Baby Boomers",
        "Generation X", "Millennials", "Generation Z"
    ],
    "birth_start": [1901, 1928, 1946, 1965, 1981, 1997],
    "birth_end": [1927, 1945, 1964, 1980, 1996, 2012],
})
generations["entry_start"] = generations["birth_start"] + 22
generations["entry_end"] = generations["birth_end"] + 22
generations["generation_label"] = (
    generations["generation"] + "\nentered "
    + generations["entry_start"].astype(str) + "–" + generations["entry_end"].astype(str)
)
generations["band_label"] = generations["generation"].str.replace(" ", "\n", n=1)
generations["band_start"] = pd.to_datetime(generations["entry_start"].clip(lower=1944).astype(str) + "-01-01")
generations["band_end"] = pd.to_datetime((generations["entry_end"].clip(upper=2025) + 1).astype(str) + "-01-01")
generations["band_mid"] = (
    generations["band_start"] + (generations["band_end"] - generations["band_start"]) / 2
).dt.floor("D")
generations["shade"] = (generations.index + 1) % 2 == 0
generations["workforce_start"] = generations["birth_start"] + 18
generations["workforce_end"] = generations["birth_end"] + 18
generations["retirement_age"] = 67
generations.loc[generations["birth_end"] <= 1954, "retirement_age"] = 66
generations.loc[generations["birth_end"] <= 1937, "retirement_age"] = 65
generations["retirement_year"] = generations["birth_end"] + generations["retirement_age"]
generations["bar_start"] = pd.to_datetime(generations["workforce_start"].clip(lower=1944).astype(str) + "-01-01")
generations["bar_end"] = pd.to_datetime((generations["retirement_year"].clip(upper=2025) + 1).astype(str) + "-01-01")

names = {
    1945: "Demobilization", 1948: "Tightening", 1953: "Korea", 1957: "Eisenhower",
    1960: "Rates", 1969: "Vietnam", 1973: "Oil", 1980: "Volcker", 1981: "Iran",
    1990: "Gulf", 2001: "Dotcom", 2007: "Subprime", 2020: "Pandemic",
}
eras = {
    2001: "Dot-com bust", 2007: "Financial crisis", 2008: "Financial crisis",
    2009: "Financial crisis", 2020: "COVID-19",
}

recessions = pd.read_csv("data/recessions.csv")
recessions["period_range"] = recessions["period_range"].str.replace(r"\[\d+\]", "", regex=True)
recessions["start_date"] = pd.to_datetime(
    recessions["period_range"].str.extract(r"^([A-Za-z]+ \d{4})")[0], format="%B %Y"
)
recessions["end_date"] = pd.to_datetime(
    recessions["period_range"].str.extract(r"([A-Za-z]+ \d{4})$")[0], format="%B %Y"
)
recessions["start_year"] = recessions["start_date"].dt.year
recessions["end_year"] = recessions["end_date"].dt.year
recessions["gdp_decline_pct"] = pd.to_numeric(
    recessions["gdp_decline"].str.extract(r"([\d.]+)")[0], errors="coerce"
)
recessions["peak_unemployment_pct"] = pd.to_numeric(
    recessions["peak_unemployment"].str.extract(r"([\d.]+)")[0], errors="coerce"
)
recessions = recessions.dropna(subset=["start_year", "end_year", "gdp_decline_pct"])
recessions = recessions[recessions["start_year"] >= 1945].astype({"start_year": int, "end_year": int})
recessions["name"] = recessions["start_year"].map(names).fillna(recessions["name"])
recessions["era"] = recessions["start_year"].map(eras).fillna("Other")
years = recessions["start_year"].astype(str).where(
    recessions["start_year"] == recessions["end_year"],
    recessions["start_year"].astype(str) + "–" + recessions["end_year"].astype(str)
)
recessions["label"] = recessions["name"] + " (" + years + ")"

# pandas has no overlap join: cross join every pair, then keep the overlaps
recessions = (
    recessions.merge(generations, how="cross")
    .query("start_year <= entry_end and end_year >= entry_start")
)
recessions.to_csv("data/recessions_clean.csv", index=False)
recessions.head()
              name                  period_range  ...  bar_start    bar_end
0   Demobilization  February 1945 – October 1945  ... 1944-01-01 1993-01-01
6       Tightening  November 1948 – October 1949  ... 1944-01-01 1993-01-01
13           Korea          July 1953 – May 1954  ... 1946-01-01 2012-01-01
19      Eisenhower      August 1957 – April 1958  ... 1946-01-01 2012-01-01
25           Rates    April 1960 – February 1961  ... 1946-01-01 2012-01-01

[5 rows x 32 columns]
Tipdplyr has the edge

join_by(overlaps()) matches each recession to the generation entry window it overlaps in one step. pandas has no overlap (non-equi) join, so the Python version cross-joins every recession with every generation (13 × 6 rows) and then filters. That’s fine at this size, but the intermediate table grows with the product of the two row counts.

See viz_recessions.qmd for the chart built on this.