◢◤ ARKANE_
SYS_TIME UTC+0
Work with me

Chapter 20

Time Series Analysis

Every model we’ve met so far has assumed rows are independent and identically distributed. Shuffle them and nothing changes. A random 80/20 split is fine. Ordinary k-fold cross-validation works as-is. Time series data breaks that assumption. Today’s value depends on yesterday’s. December’s sales depend on last December’s. A random split leaks the future into training and quietly inflates every reported metric. The model gets to peek at values from after the point it’s supposed to be predicting. This chapter is the toolkit built around the thing we’ve so far ignored: order.

20.1 Motivation

A time series is a sequence of values indexed by time. Almost every real series is a mix of three components: trend (the long-run direction), seasonality (a pattern repeating at a fixed, known period), and residual noise (whatever’s left once trend and seasonality are removed). Classical decomposition splits a series into exactly these pieces, additively (y=trend+seasonal+residualy = \text{trend} + \text{seasonal} + \text{residual}), or multiplicatively when the seasonal swing grows with the level of the series. Once this is understood, the modelling task becomes clear. Forecast the trend. Forecast the seasonal pattern. Treat the residual as irreducible uncertainty.

20.2 Exposition

20.2.1 Stationarity

Most classical forecasting models (ARIMA and its relatives) assume the series is stationary. Its mean, variance, and autocorrelation don’t change over time, so the bit of history a model is fit to generalises to the next stretch. Most real series aren’t stationary on their own. They trend, and their seasonal amplitude often grows too. Differencing (subtracting each value from the one before it, yt′=yt−yt−1y_t' = y_t - y_{t-1}) removes a linear trend. Seasonal differencing (yt′=yt−yt−my_t' = y_t - y_{t-m}) removes seasonality at period mm. The Augmented Dickey-Fuller (ADF) test checks whether differencing worked. Its null hypothesis is that the series has a unit root (is non-stationary). A small p-value (usually below 0.05) lets that null be rejected.

20.2.2 Autocorrelation and Partial Autocorrelation

Autocorrelation (ACF) at lag kk measures how similar yty_t is to yt−ky_{t-k}. Partial autocorrelation (PACF) measures the same relationship with the effect of all shorter lags already removed: the direct connection, not the connection mediated through intervening lags. These aren’t just diagnostics. This is how model order gets chosen. A sharp PACF cutoff after lag pp suggests an AR(pp) term. A sharp ACF cutoff after lag qq suggests an MA(qq) term. A slowly decaying ACF is the signature of non-stationarity: a signal to difference before fitting anything.

20.2.3 ARIMA: AR, I, MA

ARIMA(p,d,qp,d,q) combines three ideas. AR(pp), autoregressive: today’s value is a linear combination of the pp previous values plus noise, capturing momentum. I(dd), integrated: difference the series dd times to make it stationary before fitting. d=1d=1 handles a linear trend. d=2d=2 handles a trend that’s accelerating (rare in practice). MA(qq), moving average: today’s value is a linear combination of the qq previous forecast errors, capturing short-lived shocks that take a few periods to wash out. SARIMA adds a second, seasonal (P,D,Q,m)(P,D,Q,m) block on top of the same three ideas, applied at the seasonal lag mm instead of lag 1. Use it whenever ACF/PACF or the decomposition shows a repeating pattern the plain ARIMA terms can’t capture.

20.2.4 Exponential Smoothing

A simpler approach, from a simpler time. Or the Honey Badger of forecasting models, taking on beasts much bigger than itself. You decide! The idea is simple: forecast the next value as a weighted average of past values, with weights decaying exponentially into the past. Simple exponential smoothing handles level only. One smoothed average, nudged a little towards each new value as it arrives. Holt’s method (Holt 1957) adds a second smoothed term for trend, so the forecast can keep rising or falling instead of flattening out at the last level. Holt-Winters (Winters 1960) adds a third smoothed term on top of that, one for seasonality repeating at a fixed period. Three components, three running averages: one for level, one for trend, one for season, each updated a little every time a new value comes in. It’s cheaper to fit than ARIMA and a strong baseline. Always fit one before reaching for anything more complex.

20.2.5 Beyond Classical Statistics: ML and Deep Learning

Classical models work directly on a single series and its own history. A different, increasingly common approach treats forecasting as supervised regression. Engineer lag features (yt−1y_{t-1}, yt−7y_{t-7}, yt−30y_{t-30}, …) and rolling statistics as columns, then fit a gradient-boosted tree (Chapter 12) to predict yty_t from them. This folds in external features (price, weather, promotions) that ARIMA has no natural way to use, at the cost of losing ARIMA’s built-in uncertainty intervals. Prophet sits in between: an additive trend-plus-seasonality model fit via curve-fitting rather than the ARIMA machinery, popular for its robustness to missing data and minimal tuning. Deep learning models (LSTMs, temporal convolutional networks, transformer-based forecasters) can model many related series jointly and capture long-range or nonlinear dependencies classical models can’t, at the cost of needing far more data and tuning. They’re usually overkill for a single series with a clear seasonal pattern.

20.3 Trade-offs and Limits

ARIMA is more flexible than exponential smoothing, but needs a stationary series and careful order selection. Exponential smoothing is simpler, needs less tuning, and is often just as accurate in practice. Fit both. Keep whichever generalises better on held-out data. Classical models give prediction intervals for free (well calibrated only when the model’s assumptions hold) and work with very little data. Lag-feature ML models scale better to many related series and easily incorporate external regressors, but need careful leakage-free cross-validation and don’t naturally produce calibrated uncertainty. Use multiplicative decomposition when the seasonal swing grows with the series’ level. Use additive when it stays roughly constant. This is usually obvious when plotted.

20.4 Procedures

Procedure 20.1 (Split Time Series Data Correctly). When to use: every time, never a random split or plain k-fold on time-ordered data.

Listing 20.1: Chronological holdout and TimeSeriesSplit
from sklearn.model_selection import TimeSeriesSplit

horizon = 28
train, test = df.iloc[:-horizon], df.iloc[-horizon:]

tscv = TimeSeriesSplit(n_splits=5, test_size=28)
for train_idx, test_idx in tscv.split(df):
    # Each fold's test block is always chronologically
    # after its train block
    pass

Common mistake: using train_test_split(shuffle=True) or plain KFold on time-indexed data. This lets the model train on future values to predict the past, producing metrics that look great in development and fail in production.

Procedure 20.2 (Test and Achieve Stationarity). When to use: before fitting any ARIMA/SARIMA model.

Listing 20.2: ADF test and differencing
from statsmodels.tsa.stattools import adfuller

def is_stationary(series, alpha=0.05):
    return adfuller(series.dropna())[1] < alpha

diff1 = series.diff()             # removes a linear trend
seasonal_diff = series.diff(7)    # removes weekly seasonality

Common mistake: over-differencing a series that’s already stationary. This introduces artificial oscillation and inflates variance. Re-run the ADF test after each differencing step rather than differencing “just in case.”

Procedure 20.3 (Choose ARIMA/SARIMA Orders). When to use: after confirming the differencing order needed for stationarity.

Listing 20.3: Visual and automated order selection
from statsmodels.graphics.tsaplots import plot_acf, plot_pacf
plot_acf(diff1.dropna(), lags=30)
plot_pacf(diff1.dropna(), lags=30)

import pmdarima as pm
auto_model = pm.auto_arima(
    series, seasonal=True, m=7, stepwise=True, trace=True
)

Procedure 20.4 (Evaluate a Forecast Against a Naive Baseline). When to use: after any forecast, before deciding a model is good enough to ship.

Listing 20.4: Always compare against the naive baseline
def forecast_metrics(y_true, y_pred):
    mae = mean_absolute_error(y_true, y_pred)
    rmse = mean_squared_error(y_true, y_pred) ** 0.5
    return {'MAE': mae, 'RMSE': rmse}

# Or shift(7) for weekly seasonal naive
naive = y_true.shift(1).bfill()

Common mistake: skipping the naive baseline. This chapter’s own Worked Example shows exactly why that’s a mistake: a fitted model can lose to “repeat the last value,” and there’s no way to know that without checking.

20.5 Worked Example

A synthetic two-year daily series (slow upward trend plus a smooth annual sine-wave seasonal component plus moderate Gaussian noise) makes stationarity and forecast evaluation concrete. The raw series fails the ADF test outright: p=0.682p = 0.682, nowhere near significant, cannot reject non-stationarity. That’s exactly as expected given its visible trend. First-differencing fixes this clearly: p≈3×10−27p \approx 3 \times 10^{-27}, far below 0.05.

Decomposing the series confirms what it was built from: trend rising from about 109 to 127 over the two years, seasonal amplitude of roughly 16 units. The pieces are recovered correctly by construction, a useful sanity check before trusting decomposition on a real series where the ground truth isn’t known in advance.

Listing 20.5: ARIMA forecast vs. the naive baseline, over a 28-day horizon
model = SARIMAX(
    train,
    order=(2, 1, 2),
    enforce_stationarity=False,
    enforce_invertibility=False,
)
fitted = model.fit(disp=False)
pred = fitted.get_forecast(steps=28).predicted_mean

naive = pd.Series([train.iloc[-1]] * 28, index=test.index)

Measured, forecasting the final 28 days: an ARIMA(2,1,2) model reaches MAE 7.228 and RMSE 8.664. The naive baseline (repeating the last training value for all 28 days) reaches MAE 4.602 and RMSE 5.713, beating the fitted ARIMA model on both metrics. This is not a mistake in the setup. It’s a real, common outcome worth understanding rather than hiding. Over a 28-day horizon against a 365-day seasonal period and a slow linear trend, the series barely moves in expectation. “Stay near the current level” is already close to the right answer. ARIMA’s extra flexibility, fitting autoregressive and moving-average structure, is mostly available to fit short-term noise correlation it has no real ability to predict out-of-sample. The model isn’t wrong to be more complex. The complexity doesn’t help at this particular horizon, on this particular series. This is exactly why Procedure 20.4 insists on the naive baseline every time. Without it, a perfectly reasonable-looking ARIMA fit with plausible-looking diagnostics would have shipped without anyone noticing it was worse than doing nothing extra at all.

AWS SageMaker Equivalent
import boto3

forecast = boto3.client('forecast')
# Amazon Forecast automatically tries multiple algorithms
# (ARIMA, ETS, DeepAR+, Prophet, CNN-QR) and selects the
# best for the data via AutoML.
# predictor = forecast.create_auto_predictor(
#     PredictorName='demand-forecast',
#     ForecastHorizon=28,
#     ForecastFrequency='D',
#     DataConfig={'DatasetGroupArn': dataset_group_arn},
# )

When to prefer Amazon Forecast over hand-rolled ARIMA: with many related series (hundreds of SKUs, stores, or sensors), where Forecast trains global models across all series and handles algorithm selection automatically. That includes, implicitly, the naive-baseline comparison this chapter insists on doing by hand. For a single series or a handful, statsmodels/pmdarima is faster to iterate on and easier to explain to stakeholders.

Amazon Forecast closed to new customers in 2024. Existing accounts can still use it, and AWS points new users to SageMaker Canvas’s time-series forecasting or SageMaker’s built-in DeepAR algorithm. The same advice applies to those.

20.6 History and Further Reading

Hyndman and Athanasopoulos’s Forecasting: Principles and Practice is the standard modern reference, covering ARIMA, exponential smoothing, and ML-based forecasting side by side (Hyndman and Athanasopoulos 2018). Murphy’s Advanced Topics, Ch. 29, covers the state-space-model machinery underneath ARIMA and exponential smoothing (Murphy 2023). Box, Jenkins, Reinsel, and Ljung’s Time Series Analysis: Forecasting and Control is the original, rigorous treatment of the Box-Jenkins approach to ARIMA (Box et al. 2015).

20.7 How This Connects

This chapter builds on Chapter 13’s cross-validation, though time-series CV works differently: folds must respect chronological order via TimeSeriesSplit, never training on the future to predict the past. It builds on Chapter 9’s EDA, since decomposition and ACF/PACF plots are exploratory analysis specialised for sequential data. It connects to Chapter 19’s unsupervised learning, since anomaly detection on a time series usually starts by modelling the expected pattern with this chapter’s tools and then flagging large residuals. It enables Book 2, Chapter 25’s monitoring, where drift detection is, underneath, a time series problem: is this metric’s distribution changing over time? It’s often confused with sequential or panel data in general. Time series specifically means one variable (or a small related set) observed at regular intervals where order carries information.

This chapter closes Book 1. Book 2 opens with responsible AI and data governance, and from there moves into the neural-network and large-language-model territory that increasingly defines what a Solutions Architect is asked to build.

20.8 Exercises

  1. [W] Explain in your own words why a random 80/20 train/test split is invalid for time series data but fine for the datasets used in earlier chapters.

  2. [W] State what a slowly-decaying ACF (no sharp cutoff at any lag) tells you about a series, and what action it suggests.

  3. [A] Using this chapter’s measured ADF p-values (0.682 raw, roughly 3×10−273 \times 10^{-27} differenced), explain what “reject the null hypothesis” means concretely for the differenced series, using the 0.05 significance threshold.

  4. [A] Using Listing 20.5’s measured results (ARIMA MAE 7.228, naive MAE 4.602), compute the percentage by which the naive baseline outperforms ARIMA, and propose one change to the forecast horizon (longer or shorter) that this chapter’s explanation would predict might reverse that result.

  5. [E] A team reports “our ARIMA model achieves MAE of 7.2 on 28-day demand forecasts” as evidence it’s ready to ship. Using this chapter’s Worked Example, write the question you’d ask before agreeing.

  6. [E] A colleague computes a 7-day rolling mean feature for a lag-feature forecasting model using .rolling(7).mean() without first calling .shift(1). Using Procedure 20.1’s common mistake about leakage, explain what’s wrong.

  7. [R] Read the Hyndman and Athanasopoulos chapter on forecast evaluation (fpp3, “Evaluating forecast accuracy”). Identify one accuracy measure they discuss that this chapter’s reference material does not (beyond MAE, RMSE, MAPE, sMAPE), and explain what situation it’s designed for.

20.9 Mental Model

A time series model is a weather forecaster. It doesn’t know the future. It extrapolates the patterns (trend, seasonality, momentum, mean-reversion) visible in the past. It should be honest about how much less certain it is the further out it forecasts. Stationarity is the assumption that the weather changes, but the rules generating the weather do not. The moment that assumption breaks, a regime shift, a new product launch, a pandemic, every classical forecasting model needs refitting. It was never modelling the cause, only the pattern. And as this chapter’s own Worked Example shows, sometimes the honest, humble forecast, repeat the last value, is exactly the one that should have shipped.

Box, George E. P., Gwilym M. Jenkins, Gregory C. Reinsel, and Greta M. Ljung. 2015. Time Series Analysis: Forecasting and Control. 5th ed. John Wiley & Sons.
Holt, Charles C. 1957. Forecasting Seasonals and Trends by Exponentially Weighted Moving Averages. ONR Memorandum No. 52. Carnegie Institute of Technology.
Hyndman, Rob J., and George Athanasopoulos. 2018. Forecasting: Principles and Practice. 2nd ed. OTexts.
Murphy, Kevin P. 2023. Probabilistic Machine Learning: Advanced Topics. MIT Press.
Winters, Peter R. 1960. “Forecasting Sales by Exponentially Weighted Moving Averages.” Management Science 6 (3): 324–42.
online /books/classical-data-science/ch20-time-series Newcastle, UK v0.1.0 // rss