Choose the target before the model
Predicting a price level, a return, volatility, an event probability and a fill probability are different tasks. Define the outcome horizon and the loss function, then choose a baseline that makes sense for that task.
The workbench's ridge example predicts the return from the next bar's open to the following open using close-t features. It compares forecast MSE with a zero-return forecast. Its trading rule and execution costs are evaluated separately.
Keep transformations inside training
Fit scalers, imputers, feature selection and model parameters using training observations only. Apply the fitted transformation to later observations. Full-sample standardisation can expose a historical model to future distribution information even when the target column is correctly shifted.
Respect the label's time span
A training example belongs to a decision time and an outcome interval. If that interval reaches into validation, the training label uses validation information. Purge boundary examples according to the actual horizon. Panel datasets may also need splits that respect shared times across instruments.
The reference ridge routine uses expanding training windows and a two-observation gap for its stated next-open target. Other targets need a gap derived from their own availability times.
Compare an ablation
Add one feature family at a time. Measure the change against the same baseline, periods and evaluation procedure. Inspect where gains arise: particular instruments, regimes, events or isolated extremes. Report unstable results and retain failed experiments.
A sensible experiment sequence
- Constant forecast and a simple historical estimate.
- Linear or regularised linear model.
- Nonlinear model with the same information set.
- Architecture changes only when the failure analysis supports them.
Model complexity increases the opportunities for selection bias. A higher validation score is useful evidence only in the context of the experiment history, cost of implementation and performance on genuinely unseen observations.