Khansa, home
Work with me

Jump to

Pages

Projects

Links

Lesson 6 of 8, about 15 minutes

Backtesting time series models and choosing error metrics

One test window is one roll of the dice. A backtest repeats the test on several windows in the past, always training on earlier data and testing on later data. This is how you know a model really works.

By the end: Test any model on several time windows and pick the right error metric.

Walk forward through time

TimeSeriesSplit makes the windows for you. Each round trains on the past and tests on the next 28 days. The training part grows each time.

Python
import numpy as np
import pandas as pd

rng = np.random.default_rng(42)
days = pd.date_range("2024-01-01", periods=365, freq="D")
trend = np.linspace(100, 140, 365)                 # sales slowly grow
week = np.array([-8, -6, -4, 0, 6, 16, 12])[days.dayofweek]  # busy weekends
s = pd.Series(trend + week + rng.normal(0, 4, 365), index=days, name="sales")

from sklearn.model_selection import TimeSeriesSplit
from statsmodels.tsa.holtwinters import ExponentialSmoothing

tscv = TimeSeriesSplit(n_splits=4, test_size=28)
scores = []
for train_idx, test_idx in tscv.split(s):
    train, test = s.iloc[train_idx], s.iloc[test_idx]
    m = ExponentialSmoothing(train, trend="add", seasonal="add",
                             seasonal_periods=7).fit()
    scores.append(float(np.mean(np.abs(test.values - m.forecast(28).values))))

print([round(x, 2) for x in scores])
print("average MAE:", round(np.mean(scores), 2))

You should see

[2.33, 2.93, 3.39, 3.08]
average MAE: 2.93

Four windows, four scores. The scores differ from window to window, which is exactly why one test is not enough.

Which error metric should you use?

MAE is the average miss in your own unit. Easy to explain to anyone.

RMSE also uses your unit but punishes big misses more. Use it when one huge miss is much worse than several small ones.

MAPE is the average miss as a percentage. It is friendly, but it breaks when actual values are close to zero.

MASE compares your error to the error of a simple seasonal naive forecast. A value under 1 means you beat the baseline. It works across different series, so it is the safest score for comparing models.

Python
def mase(train, actual, forecast, m=7):
    scale = np.mean(np.abs(train.values[m:] - train.values[:-m]))
    return float(np.mean(np.abs(actual.values - forecast.values)) / scale)

# values below 1 beat the seasonal naive baseline
print(round(mase(train, test, m.forecast(28)), 2))

You should see

0.75

Below 1, so on the last window the model beats seasonal naive by about 25%.

A forecast is a range, not a line

A good forecast says "about 140, probably between 130 and 150". Check how often the real value falls inside your 95% range. It should be close to 95 times out of 100. If it is far lower, your range is too narrow.

Where you will use this

  • Choosing between ETS, ARIMA and a machine learning model fairly
  • Showing a client that the model was tested on the past, not guessed
  • Comparing forecasts of different products on the same scale with MASE

Common mistake: Do not tune your model on the test window and then report that same score. Tune on the backtest, then keep one final window untouched.

Try it yourself

Add a second model, seasonal naive, to the loop and print both averages. Does ETS win in every window?

Check yourself

MASE is 0.8. Is that good?

Yes. It means the model beats the seasonal naive baseline by about 20%.

Why not use a random train and test split?

It lets the model peek at the future. Time series must be split by time.