Walk forward through time
TimeSeriesSplit makes the windows for you. Each round trains on the past and tests on the next 28 days. The training part grows each time.
import numpy as np
import pandas as pd
rng = np.random.default_rng(42)
days = pd.date_range("2024-01-01", periods=365, freq="D")
trend = np.linspace(100, 140, 365) # sales slowly grow
week = np.array([-8, -6, -4, 0, 6, 16, 12])[days.dayofweek] # busy weekends
s = pd.Series(trend + week + rng.normal(0, 4, 365), index=days, name="sales")
from sklearn.model_selection import TimeSeriesSplit
from statsmodels.tsa.holtwinters import ExponentialSmoothing
tscv = TimeSeriesSplit(n_splits=4, test_size=28)
scores = []
for train_idx, test_idx in tscv.split(s):
train, test = s.iloc[train_idx], s.iloc[test_idx]
m = ExponentialSmoothing(train, trend="add", seasonal="add",
seasonal_periods=7).fit()
scores.append(float(np.mean(np.abs(test.values - m.forecast(28).values))))
print([round(x, 2) for x in scores])
print("average MAE:", round(np.mean(scores), 2))You should see
[2.33, 2.93, 3.39, 3.08] average MAE: 2.93
Four windows, four scores. The scores differ from window to window, which is exactly why one test is not enough.
Which error metric should you use?
MAE is the average miss in your own unit. Easy to explain to anyone.
RMSE also uses your unit but punishes big misses more. Use it when one huge miss is much worse than several small ones.
MAPE is the average miss as a percentage. It is friendly, but it breaks when actual values are close to zero.
MASE compares your error to the error of a simple seasonal naive forecast. A value under 1 means you beat the baseline. It works across different series, so it is the safest score for comparing models.
def mase(train, actual, forecast, m=7):
scale = np.mean(np.abs(train.values[m:] - train.values[:-m]))
return float(np.mean(np.abs(actual.values - forecast.values)) / scale)
# values below 1 beat the seasonal naive baseline
print(round(mase(train, test, m.forecast(28)), 2))You should see
0.75
Below 1, so on the last window the model beats seasonal naive by about 25%.
A forecast is a range, not a line
A good forecast says "about 140, probably between 130 and 150". Check how often the real value falls inside your 95% range. It should be close to 95 times out of 100. If it is far lower, your range is too narrow.