# Methodology

Raw pageviews have two problems as a measure of cultural interest. First, total Wikipedia traffic has drifted since 2015, so a flat view count can hide a real rise or fall in interest. Second, topics live on very different scales: a famous athlete draws far more views than a discontinued toy. The index fixes both problems by normalizing twice.


# How is the index calculated?

Each article goes through the three steps shown below.


*\[Rich HTML output -- view on the documentation site\]*


    views_per_million = article_views / total_project_views x 1e6
    smoothed          = rolling 28 day median
    nostalgia_index   = smoothed / baseline_median x 100

The first step divides an article's daily views by total project views that day (English Wikipedia by default), which removes drift in overall traffic. The second step smooths the series with a rolling median, which is less sensitive to a single viral day than a mean. The third step divides by the baseline median and multiplies by 100, so every topic starts on the same scale.

The baseline is the median of the smoothed series over its first 12 months. An index of 100 means a topic draws the same share of Wikipedia traffic as it did during its baseline period; an index of 200 means it draws twice that share.


# What counts as a revival?

A revival event is a day where the unsmoothed index exceeds 3x its trailing 90 day median. The unsmoothed index matters here, because the rolling median would flatten the very spikes we're trying to detect.

All three thresholds are arguments to [NostalgiaIndex()](../reference/NostalgiaIndex.md#nostalgia_index.NostalgiaIndex): `window` (28 days), `baseline_months` (12), and `spike_factor` (3.0).


# Caveats

Keep a few limitations in mind when reading the index. The Wikimedia pageviews data starts on 2015-07-01, so nothing earlier is available. Only `agent=user` traffic is used, which excludes spiders. Redirects are resolved through the MediaWiki API, so a misspelled title like `Tamagochi` is counted as `Tamagotchi`.

> **Warning: Warning**
>
> Topics without an article in 2015 have no true baseline. Their index is relative to their own first year of data, not to a shared starting point.


# Worked example

The example below runs the index on the sample data in `fixtures/`, so it doesn't call the API. [summary()](../reference/NostalgiaIndex.md#nostalgia_index.NostalgiaIndex.summary) returns one row per topic with its current index, peak value and date, change versus baseline, and revival count.


``` python
import pandas as pd
from nostalgia_index import NostalgiaIndex

views = pd.read_csv("fixtures/views.csv", parse_dates=["date"])
totals = pd.read_csv("fixtures/totals.csv", parse_dates=["date"])
ni = NostalgiaIndex()
idx = ni.compute(views, totals)
ni.summary(idx).round(1)
```


    /tmp/ipykernel_36375/1864511853.py:8: UserWarning: obj.round has no effect with datetime, timedelta, or period dtypes. Use obj.dt.round(...) instead.
      ni.summary(idx).round(1)


|     | article    | current | peak  | peak_date  | pct_vs_baseline | revivals |
|-----|------------|---------|-------|------------|-----------------|----------|
| 0   | Furby      | 82.0    | 107.0 | 2015-11-21 | -18.0           | 1        |
| 1   | Tamagotchi | 182.8   | 189.4 | 2019-12-23 | 82.8            | 0        |


[revivals()](../reference/NostalgiaIndex.md#nostalgia_index.NostalgiaIndex.revivals) lists each revival event with its unsmoothed index, the trailing median it was compared against, and the resulting multiple.


``` python
ni.revivals(idx).head()
```


|     | date       | article | raw_index  | trailing_median | multiple |
|-----|------------|---------|------------|-----------------|----------|
| 0   | 2017-12-17 | Furby   | 632.203184 | 94.416844       | 6.695873 |


# Recap

The nostalgia index divides pageviews by total Wikipedia traffic, smooths them with a rolling median, and scales each topic to 100 at its baseline. Revival events flag days where the unsmoothed index jumps above three times its recent median. Read results for topics created after 2015 with care, since their baseline is their first year rather than 2015.