Basically Futures

STAGE 4 · LESSON 4 OF 5

Out-of-sample and forward evaluation

A strategy must be challenged on data it was not designed to fit.

THE IDEA~8–12 MINUTES

Understand it

In-sample data help develop a hypothesis; out-of-sample data test it with fewer degrees of freedom. Walk-forward methods repeat this idea over time. Paper trading adds execution and behavior questions but still differs from real fills. Stronger evidence also includes robustness to costs, parameter changes and different market conditions. No single test eliminates uncertainty or guarantees future returns.

WORKED EXAMPLE

After choosing a breakout threshold on 2022–2024 data, reserve 2025 observations untouched for validation. If you optimize based on 2025 too, it is no longer untouched out-of-sample evidence.

Watch for this misconception

Retuning after every disappointing validation test and continuing to call the same dataset 'unseen.'

PRACTICE EXERCISE

Write the intended data split and a rule for when you will stop optimizing.

Expectancy calculator ↗
KNOWLEDGE CHECK

When does an out-of-sample dataset lose its holdout role?

Select the best answer

Check the sources

Hypothetical educational examples. Verify current product specifications and rules before making decisions.