---------------------------------------------------------------------- This is the API documentation for the nostalgia_index library. ---------------------------------------------------------------------- ## Dataclasses Dataclass definitions NostalgiaIndex(window: 'int' = 28, baseline_months: 'int' = 12, spike_factor: 'float' = 3.0) -> None Compute and hold the nostalgia index for one or more topics. Parameters ---------- window Rolling-median window in days (default 28). baseline_months Number of months at the start of the series used to define 100. spike_factor Multiple of the trailing 90-day median that counts as a "revival event". Examples -------- >>> import pandas as pd >>> d = pd.date_range("2016-01-01", periods=730) >>> v = pd.DataFrame({"date": d, "article": "A", "views": 100}) >>> t = pd.DataFrame({"date": d, "views": 1_000_000}) >>> NostalgiaIndex().compute(v, t).nostalgia_index.iloc[-1] np.float64(100.0) ## Functions Utility functions fetch_project_total(project: 'str' = 'en.wikipedia', start: 'dt.date' = datetime.date(2015, 7, 1), end: 'dt.date | None' = None, granularity: 'str' = 'daily', **kw) -> 'pd.DataFrame' Fetch total daily views for a project (``/aggregate/`` endpoint), used as the denominator. Parameters ---------- project Wikimedia project, e.g. ``"en.wikipedia"``. start, end Inclusive date range. granularity ``"daily"`` or ``"monthly"``. Returns ------- pd.DataFrame Columns ``date``, ``views``. fetch_views(article: 'str', start: 'dt.date' = datetime.date(2015, 7, 1), end: 'dt.date | None' = None, project: 'str' = 'en.wikipedia', granularity: 'str' = 'daily') -> 'pd.DataFrame' Fetch daily pageviews for a single Wikipedia article. Parameters ---------- article Article title, e.g. ``"Tamagotchi"``. Spaces are converted to underscores and the title is URL-encoded for you. Redirects are resolved first. start, end Inclusive date range. Data is available from 2015-07-01. project Wikimedia project, e.g. ``"en.wikipedia"``. granularity ``"daily"`` or ``"monthly"``. Returns ------- pd.DataFrame Columns ``date``, ``article``, ``views``. Empty (with a warning) if the article has no data for the range. Examples -------- >>> df = fetch_views("Furby", start=dt.date(2020, 1, 1), end=dt.date(2020, 12, 31)) # doctest: +SKIP >>> df.views.sum() > 0 # doctest: +SKIP True run_app(port: int = 8000, launch_browser: bool = True) -> None Launch the Nostalgia Index Shiny app. Parameters ---------- port Port to serve on. launch_browser Open a browser tab on start. ---------------------------------------------------------------------- This is the CLI documentation for the package. ---------------------------------------------------------------------- ## CLI: nostalgia ``` Usage: nostalgia [OPTIONS] COMMAND [ARGS]... Nostalgia Index: Wikipedia pageviews as a cultural pulse. Options: --help Show this message and exit. Commands: fetch Compute the index for TOPICS and write a CSV. run Launch the Shiny app. ``` ### nostalgia fetch ``` Usage: nostalgia fetch [OPTIONS] TOPICS... Compute the index for TOPICS and write a CSV. Options: --out PATH Output CSV path. --window INTEGER Rolling median days. [default: 28] --help Show this message and exit. ``` ### nostalgia run ``` Usage: nostalgia run [OPTIONS] Launch the Shiny app. Options: --port INTEGER Port to serve on. [default: 8000] --help Show this message and exit. ``` ---------------------------------------------------------------------- This is the User Guide documentation for the package. ---------------------------------------------------------------------- ### Quickstart This page gets you from installation to a computed index in a few commands. The command line is the fastest way to start. The commands below install the package, compute the index for `Tamagotchi` and `Furby` (written to `index.csv`), and launch the Shiny app. ```bash pip install nostalgia_index nostalgia fetch Tamagotchi Furby --out index.csv nostalgia run ``` From Python, the same steps take three calls. `fetch_views()` downloads daily pageviews for one article, `fetch_project_total()` downloads total English Wikipedia traffic to use as the denominator, and `NostalgiaIndex().compute()` combines them. ```python from nostalgia_index import fetch_views, fetch_project_total, NostalgiaIndex views = fetch_views("Furby") totals = fetch_project_total() idx = NostalgiaIndex().compute(views, totals) ``` ```{mermaid} %%{init: {'theme': 'base', 'themeVariables': { 'fontFamily': 'monospace', "darkMode":true}}}%% flowchart LR API(["Wikimedia API"]) subgraph Pkg["nostalgia_index"] C[["client
fetch_views
fetch_project_total"]] K[("disk cache")] I[["index
NostalgiaIndex"]] end CSV[/"index.csv"/] APP("Shiny app") API --> C C <-.-> K C --> I I --> CSV I --> APP style Pkg fill:#3b383f,color:#f6f0c6,stroke:#6c6b71,stroke-width:1px,rx:10,ry:10 style C fill:#ffbe00,color:#1e1c21,rx:5,ry:5 style K fill:#ffbe00,color:#1e1c21,rx:5,ry:5 style I fill:#ffbe00,color:#1e1c21,rx:5,ry:5 style API fill:#ffbe00,color:#1e1c21,rx:5,ry:5 style CSV fill:#ffbe00,color:#1e1c21,rx:5,ry:5 style APP fill:#ffbe00,color:#1e1c21,rx:5,ry:5 ``` The result in `idx` is a DataFrame with one row per article per day. The `nostalgia_index` column holds the scaled index, where 100 equals the article's level during its first 12 months. The first request for each article goes to the Wikimedia API; repeat requests read from a local cache on disk. ### Methodology Raw pageviews have two problems as a measure of cultural interest. First, total Wikipedia traffic has drifted since 2015, so a flat view count can hide a real rise or fall in interest. Second, topics live on very different scales: a famous athlete draws far more views than a discontinued toy. The index fixes both problems by normalizing twice. ## How is the index calculated? Each article goes through the three steps shown below. ```{mermaid} %%{init: {'theme': 'base', 'themeVariables': { 'fontFamily': 'monospace', "darkMode":true}}}%% flowchart LR V[/"article views"/] T[/"total project views"/] P["views per million"] S["28 day rolling median"] B{{"baseline median
first 12 months"}} X["nostalgia_index"] R("revival events
raw index > 3x trailing 90 day median") V --> P T --> P P --> S S --> X B --> X P -.-> R style V fill:#ffbe00,color:#1e1c21,rx:5,ry:5 style T fill:#ffbe00,color:#1e1c21,rx:5,ry:5 style P fill:#ffbe00,color:#1e1c21,rx:5,ry:5 style S fill:#ffbe00,color:#1e1c21,rx:5,ry:5 style B fill:#ffbe00,color:#1e1c21,rx:5,ry:5 style X fill:#f4640d,color:#1e1c21,rx:5,ry:5 style R fill:#ffbe00,color:#1e1c21,rx:5,ry:5 ``` ``` views_per_million = article_views / total_project_views x 1e6 smoothed = rolling 28 day median nostalgia_index = smoothed / baseline_median x 100 ``` The first step divides an article's daily views by total project views that day (English Wikipedia by default), which removes drift in overall traffic. The second step smooths the series with a rolling median, which is less sensitive to a single viral day than a mean. The third step divides by the baseline median and multiplies by 100, so every topic starts on the same scale. The baseline is the median of the smoothed series over its first 12 months. An index of 100 means a topic draws the same share of Wikipedia traffic as it did during its baseline period; an index of 200 means it draws twice that share. ## What counts as a revival? A revival event is a day where the unsmoothed index exceeds 3x its trailing 90 day median. The unsmoothed index matters here, because the rolling median would flatten the very spikes we're trying to detect. All three thresholds are arguments to `NostalgiaIndex()`: `window` (28 days), `baseline_months` (12), and `spike_factor` (3.0). ## Caveats Keep a few limitations in mind when reading the index. The Wikimedia pageviews data starts on 2015-07-01, so nothing earlier is available. Only `agent=user` traffic is used, which excludes spiders. Redirects are resolved through the MediaWiki API, so a misspelled title like `Tamagochi` is counted as `Tamagotchi`. ::: {.callout-warning} Topics without an article in 2015 have no true baseline. Their index is relative to their own first year of data, not to a shared starting point. ::: ## Worked example The example below runs the index on the sample data in `fixtures/`, so it doesn't call the API. `summary()` returns one row per topic with its current index, peak value and date, change versus baseline, and revival count. ```{python} import pandas as pd from nostalgia_index import NostalgiaIndex views = pd.read_csv("fixtures/views.csv", parse_dates=["date"]) totals = pd.read_csv("fixtures/totals.csv", parse_dates=["date"]) ni = NostalgiaIndex() idx = ni.compute(views, totals) ni.summary(idx).round(1) ``` `revivals()` lists each revival event with its unsmoothed index, the trailing median it was compared against, and the resulting multiple. ```{python} ni.revivals(idx).head() ``` ## Recap The nostalgia index divides pageviews by total Wikipedia traffic, smooths them with a rolling median, and scales each topic to 100 at its baseline. Revival events flag days where the unsmoothed index jumps above three times its recent median. Read results for topics created after 2015 with care, since their baseline is their first year rather than 2015. ### Running the app The Shiny app wraps the same pipeline in an interface for comparing topics side by side. Launch it from the command line (the port defaults to 8000). ```bash nostalgia run --port 8000 ``` You can also launch it from Python with `run_app()`. ```python from nostalgia_index import run_app run_app(port=8000) ``` ## Computing an index The sidebar holds every input the index needs. Work through it from top to bottom. ```{mermaid} %%{init: {'theme': 'base', 'themeVariables': { 'fontFamily': 'monospace', "darkMode":true}}}%% flowchart TD subgraph In["Sidebar inputs"] TP["Topics"] W["Smoothing"] D["Date range"] end GO(["Compute index"]) subgraph Re["Reactive"] F[["fetch_views
fetch_project_total"]] IX[["NostalgiaIndex.compute"]] end subgraph Out["Outputs"] VB["Value boxes"] CH["Index chart"] RT["Revival table"] CG["Compare grid"] end TP --> GO W --> GO D --> GO GO --> F --> IX IX --> VB IX --> CH IX --> RT IX --> CG style In fill:#3b383f,color:#f6f0c6,stroke:#6c6b71,stroke-width:1px,rx:10,ry:10 style Re fill:#3b383f,color:#f6f0c6,stroke:#6c6b71,stroke-width:1px,rx:10,ry:10 style Out fill:#3b383f,color:#f6f0c6,stroke:#6c6b71,stroke-width:1px,rx:10,ry:10 style TP fill:#ffbe00,color:#1e1c21,rx:5,ry:5 style W fill:#ffbe00,color:#1e1c21,rx:5,ry:5 style D fill:#ffbe00,color:#1e1c21,rx:5,ry:5 style GO fill:#ffbe00,color:#1e1c21,rx:5,ry:5 style F fill:#ffbe00,color:#1e1c21,rx:5,ry:5 style IX fill:#ffbe00,color:#1e1c21,rx:5,ry:5 style VB fill:#ffbe00,color:#1e1c21,rx:5,ry:5 style CH fill:#ffbe00,color:#1e1c21,rx:5,ry:5 style RT fill:#ffbe00,color:#1e1c21,rx:5,ry:5 style CG fill:#ffbe00,color:#1e1c21,rx:5,ry:5 ``` 1. Choose articles in Topics, or click a preset button (Sports, Games, TV, and so on) to load a curated set. To add an article that isn't listed, type its title in Add an article and click Add. The app resolves redirects and warns you if no article exists. 2. Set Smoothing (days) to control the rolling median window (7 to 90 days, default 28). 3. Set the Date range. Data is available from 2015-07-01. 4. Click Compute index. The app doesn't fetch or recompute anything until you click Compute index, so you can adjust several settings first. ::: {.callout-note} The baseline is the first 12 months of the date range you select. Moving the start date moves the baseline, so the same topic can show different index values across date ranges. ::: ## Starter topics The topic dropdown starts with 92 articles in 8 categories. Michael Jordan and Nintendo 64 are selected by default, and each category has a preset button that selects its whole group. | Category | Count | Topics | | --- | --- | --- | | Sports | 17 | Michael Jordan, Chicago Bulls, Space Jam, Shaquille O'Neal, Ken Griffey Jr., Sammy Sosa, Mark McGwire, Derek Jeter, Brett Favre, Dale Earnhardt, Tiger Woods, Mike Tyson, Attitude Era, WCW Monday Nitro, 1996 Summer Olympics, 1992 United States men's Olympic basketball team, Topps | | Games | 21 | Oregon Trail II, Super Mario Bros., Nintendo 64, GoldenEye 007, Mario Kart 64, Super Smash Bros., Sega Genesis, Game Boy, Game Boy Color, PlayStation (console), Pokémon Red, Blue, and Yellow, Mortal Kombat, Street Fighter II, NBA Jam, Madden NFL, Tony Hawk's Pro Skater, Halo: Combat Evolved, Counter-Strike, StarCraft, RuneScape, Xbox network | | TV | 13 | Dawson's Creek, Glee (TV series), One Tree Hill (TV series), Gossip Girl, Dragon Ball Z, Toonami, Cartoon Network, Nickelodeon, Teenage Mutant Ninja Turtles, Mighty Morphin Power Rangers, Cowboy Bebop, Beavis and Butt-Head, Jackass (franchise) | | Music | 15 | Taylor Swift, Beyoncé, Backstreet Boys, Michael Jackson, Eminem, Weezer, The Pussycat Dolls, Alanis Morissette, Britney Spears, The Eras Tour, Blink-182, Linkin Park, Limp Bizkit, Green Day, Wu-Tang Clan | | Films | 6 | Mean Girls, Barbie (film), Star Wars, The Matrix, Fight Club, American Pie (film) | | Books | 3 | Harry Potter, Twilight (novel series), The Hunger Games | | Toys | 8 | Lego, Nerf, Hot Wheels, Transformers, Beyblade, Magic: The Gathering, Yu-Gi-Oh!, Milk caps (game) | | Tech | 9 | Instant camera, Phonograph record, Blockbuster (retailer), Napster, LimeWire, AIM (software), Myspace, Mountain Dew, Jolt Cola | Type any other article title into the box and click Add to include it. ## Reading the results The value boxes across the top show the topic with the highest peak, the biggest comeback (the largest current change versus baseline), and the total number of revival events. The tabs below them hold the index chart, the revival events table, a comparison grid, and a short summary of the methodology. ## Recap Launch the app with `nostalgia run` or `run_app()`, pick your topics and settings in the sidebar, and click Compute index. Keep the start date in mind when comparing runs, because it sets the baseline. ### Ethics and rate limits The nostalgia index pulls its data from the [Wikimedia REST API](https://wikimedia.org/api/rest_v1/), a free public service. This page covers how the package keeps its requests polite and how to make them identify you. ## Identifying your requests Wikimedia's [User-Agent policy](https://foundation.wikimedia.org/wiki/Policy:Wikimedia_Foundation_User-Agent_Policy) asks every client to send a descriptive `User-Agent` header with contact information. The package sends one with every request, but the default contains placeholder contact details. Edit `USER_AGENT` in `nostalgia_index.client` to use your own project URL and email. ## Keeping request volume low The package requests only `agent=user` data, which excludes spiders. Historical pageviews don't change, so responses are cached on disk and each article and date range is downloaded only once. The cache lives in `~/.cache/nostalgia_index`; set the `NOSTALGIA_CACHE_DIR` environment variable to store it somewhere else. ```{mermaid} %%{init: {'theme': 'base', 'themeVariables': { 'fontFamily': 'monospace', "darkMode":true}}}%% flowchart TD REQ(["fetch_views"]) HIT{"in disk cache?"} CACHE[("cached frame")] CALL["GET with User-Agent
agent=user"] ST{"status"} BACK["exponential backoff
up to 4 retries"] NF["empty frame + warning"] OK["parse and cache"] REQ --> HIT HIT -- yes --> CACHE HIT -- no --> CALL --> ST ST -- "429 / 5xx" --> BACK --> CALL ST -- 404 --> NF ST -- 200 --> OK style REQ fill:#ffbe00,color:#1e1c21,rx:5,ry:5 style HIT fill:#ffbe00,color:#1e1c21,rx:5,ry:5 style CACHE fill:#ffbe00,color:#1e1c21,rx:5,ry:5 style CALL fill:#ffbe00,color:#1e1c21,rx:5,ry:5 style ST fill:#ffbe00,color:#1e1c21,rx:5,ry:5 style BACK fill:#ffbe00,color:#1e1c21,rx:5,ry:5 style NF fill:#ffbe00,color:#1e1c21,rx:5,ry:5 style OK fill:#ffbe00,color:#1e1c21,rx:5,ry:5 ``` When the API responds with a 429 (too many requests) or a server error (500, 502, 503, or 504), the request is retried with exponential backoff, up to four times. ## Recap Replace the placeholder contact details in `USER_AGENT` before making regular requests. The disk cache and backoff retries keep load on Wikimedia low, and `NOSTALGIA_CACHE_DIR` controls where cached responses are stored. ### Desktop installer (plan) This page is a plan for shipping the Nostalgia Index app as a desktop installer (`.dmg`, `.exe`, `.AppImage`) with the [shinyelectron](https://r-pkg.thecoatlessprofessor.com/shinyelectron/) R package (version 0.2.1 or later). Nothing described here exists in the repository yet. shinyelectron only understands an app directory (a folder with an `app.py`). The package is not an app directory, so the plan generates a thin launcher folder that imports the package, vendors a copy of the package next to it, and hands that folder to `shinyelectron::export()`. The package source stays as it is. ```{mermaid} %%{init: {'theme': 'base', 'themeVariables': { 'fontFamily': 'monospace', "darkMode":true}}}%% flowchart TD PKG["src/nostalgia_index/"] --> SC(["scaffold_py.py"]) SC --> APP["electron/app/"] APP --> CHK(["app_check()"]) CHK --> EXP(["export()"]) EXP --> DIST["electron/dist/electron-app/dist/"] ``` ## Entry point The app already exposes a module level `App` object, so no source changes are needed. ```python # src/nostalgia_index/app/app.py app = App(app_ui, server, static_assets=Path(__file__).parent / "www") ``` The launcher imports it from `nostalgia_index.app.app`. `run_app()` is not used, because shinyelectron owns the port and the server process. The app name and version (`nostalgia_index`, `0.1.0`) come from `[project]` in `pyproject.toml`. ## Build machine shinyelectron is an R package, so the build machine needs R (4.4 or later) and Node (22 or later, with npm 11.5 or later). End users need neither. ```r shinyelectron::sitrep_shinyelectron() shinyelectron::install_nodejs() # only if Node is missing ``` ## Runtime strategy The plan uses the `bundled` strategy, which ships a portable Python with the installer (roughly 150 to 300 MB). The other strategies were ruled out for these reasons: * `shinylive`: `httpx` cannot open sockets inside Pyodide, and `diskcache` relies on SQLite file locking. The app could not reach the Wikimedia API. * `system`: it requires users to have Python on their `PATH`. * `container`: there are no heavy system libraries to justify Docker. * `auto-download`: it is a reasonable fallback if installer size becomes a problem, since the app needs internet access anyway. The app is never fully offline. Every index computation calls the Wikimedia REST API, so the installer removes the Python requirement, not the network requirement. ## Scaffold the launcher Run the scaffold script from the package root. It is idempotent and should be re-run before every build to refresh the vendored copy. ```bash python tools/scaffold_py.py \ --strategy=bundled \ --module=nostalgia_index.app.app \ --attr=app \ --platforms=mac,win,linux ``` It creates the following files: ``` electron/ ├── app/ │ ├── app.py # from nostalgia_index.app.app import app │ ├── requirements.txt # [project].dependencies │ ├── nostalgia_index/ # vendored copy (gitignored) │ └── _shinyelectron.yml # name, version, strategy, dependencies └── build.R # app_check() + export() into electron/dist ``` Two repository changes go with it. `electron/dist/` and `electron/app/nostalgia_index/` are added to `.gitignore`, and `electron/` is excluded from the sdist. ```toml [tool.hatch.build.targets.sdist] exclude = ["electron"] ``` ## Dependency checks * `shinywidgets` and `plotly`: these are pure Python and install from `requirements.txt` into the bundled runtime. * `custom.css`: it is resolved with `Path(__file__).parent`, so it loads from the vendored copy regardless of the working directory. * Cache: `diskcache` writes to `~/.cache/nostalgia_index` (or `NOSTALGIA_CACHE_DIR`). This resolves on all three platforms, but the Windows path should be confirmed in a test build. * User-Agent: `USER_AGENT` in `client.py` still holds placeholder contact details. These must be real before a public release (see [Ethics and rate limits](03-ethics-and-rate-limits.qmd)). * Python pin: set `dependencies.python.version` in `_shinyelectron.yml` to a version that satisfies `requires-python >= 3.10`. ## Validate and build ```bash Rscript electron/build.R ``` `build.R` runs `app_check("electron/app")` and then `export()`. Only the contents of `electron/dist/electron-app/dist/` get shipped. If the packaged app fails to start, launch it with `SHINYELECTRON_DEBUG=1` to see runtime detection, dependency installs, and server startup. ## Signing and CI macOS installers only build on macOS, and `bundled` cannot cross-build, so each platform builds on its own CI runner. * Copy `build-installers.yml` from the py-shinyelectron-pkg skill assets into `.github/workflows/`, and copy `scaffold_py.py` into `tools/` (the workflow calls it there). * Signing uses `sign = TRUE` with the `CSC_LINK`, `CSC_KEY_PASSWORD`, `APPLE_ID`, `APPLE_APP_SPECIFIC_PASSWORD`, and `APPLE_TEAM_ID` secrets. Secrets go in the job level `env:` block, never in files. * Trigger the workflow on version tags so the installer version tracks `pyproject.toml`. ## Checklist ``` - [ ] Preflight the build machine (R, Node, npm) - [ ] Add tools/scaffold_py.py and run it with --strategy=bundled - [ ] Update .gitignore and the hatch sdist exclude - [ ] Pin dependencies.python.version - [ ] Replace the placeholder User-Agent contact details - [ ] Run electron/build.R locally and test the installer - [ ] Confirm the cache path on Windows - [ ] Add the CI workflow and signing secrets ``` ## Version bumps The installer version is pinned to the package version. Bump `version` in `pyproject.toml` and re-run the scaffold; never edit the version in `_shinyelectron.yml` by hand.