# Train/test split: out-of-sample testing for a backtest

Canonical URL: https://marketheist.io/portfolio/learn/train-test-split
By Jeffrey Lin · Published 2026-11-10 · Updated 2026-11-10 · Data through Sep 2026
Category: Validation

## Key takeaways

- Any setting chosen because it looked strongest on a history tends to look worse on new data: part of its advantage was luck in that particular sample. The test period estimates the result without that luck.
- The split only works if nothing about the test period influenced the design. Choosing settings on the whole history and then splitting it measures nothing.
- Degradation is how much a result drops from the training period to the test period, in percent. MarketHeist's backtester colours a drop under 15% green, 15% to 35% amber and above 35% red.
- One split is one experiment: the test period has its own luck, and markets change between periods. Walk-forward analysis repeats the split through time to get several test periods instead of one.
- MarketHeist's lab picks the trend-filter length with the strongest training Sharpe ratio and applies it, unchanged, to the following test months; with one training window and one test window, it is a train/test split.

## Definition

A train/test split divides a backtest's history into an earlier training period, used to design the strategy and choose its settings, and a later test period, used only once to measure how those settings perform on data they were not fitted to. The drop from training to test results, often called degradation, is a direct measure of how much of the backtest was fitted to noise.

Also known as: out-of-sample test, in-sample vs out-of-sample, holdout test, backtest overfitting, data snooping, performance degradation, IS/OOS split.

## How it works

Backtests are usually built by trying ideas and settings until the result looks good. Each try that is kept because it beat the others carries some luck from the specific history it was tested on: a moving-average length that happened to dodge two crashes, a threshold that happened to sit just above a few bad months. Run on new data, that luck does not repeat, and the result falls back. The more settings tried, the larger the fall tends to be.

A train/test split puts the fall in plain view. Set aside the later part of the history before starting. Design and tune the strategy on the earlier part only, then run the final version once on the later part. The test result is an honest estimate because the test data played no role in the choices. The gap between the two is the degradation.

It works like an exam held back from the practice papers: a student who memorised the practice answers scores well on them and poorly on the exam, while one who learned the subject scores similarly on both. A strategy whose test results are close to its training results learned something general; one whose results collapse memorised the training period.

A split has limits. One test period is one sample, with its own luck: a market regime that suits the strategy can flatter the test, and one that does not can sink a sound idea. The split also costs data, because the test months cannot be used for design. And it is easily spoiled: looking at the test result, changing the strategy and testing again turns the test period into training data. Walk-forward analysis and the deflated Sharpe ratio are two ways to go further.

## Formula

**Degradation of a metric from training to test**

```
D = \max (0,\frac{M_{train} - M_{test}}{|M_{train}|})
```

where:

- D: degradation: 0 when the test result is as good or better, 1 when it falls to zero
- M_train: the metric (Sharpe ratio, CAGR, Calmar, Sortino, Omega or time invested) on the training data
- M_test: the same metric on the test data

This is the backtester's version. For maximum drawdown, where smaller is better, it is (|DD_test| − |DD_train|) / |DD_train|. The lab on this page reports the Sharpe ratio form without the floor at zero, 1 − SR_test / SR_train, so a test period that did better shows as a negative drop.

**Choosing a setting on the training data**

```
N^* = argmax_{N \in {N_\min,…,N_\max}} SR_{train} (N), reported: SR_{test} (N^*)
```

where:

- N: a candidate setting, here the trend filter's moving-average length in months
- N*: the setting with the highest Sharpe ratio on the training data (the shortest on a tie)
- SR_train(N), SR_test(N): Sharpe ratio of the rule with setting N on the training and test months

The training figure SR_train(N*) is the maximum of several noisy estimates, so it is biased upward; the test figure is not, because N* was fixed before the test months were read.

**Sharpe ratio on each side**

```
SR = \frac{\bar{r}}{s} \sqrt{12}
```

where:

- r̄: mean monthly return over the months of that side
- s: sample standard deviation of those monthly returns

Risk-free rate 0, as elsewhere on MarketHeist; months out of the market earn 0.

## Worked example

The classic 60/40 stocks-and-bonds portfolio with a whole-portfolio trend filter, from Feb 2014 to Sep 2026. The first 300 months, 25 years, are the training set; every month after that, through Sep 2026, is the test set, kept out of every choice.

On the training months the strongest trend-filter length, among every length from 4 to 18 months, was 12 months, with a Sharpe ratio of 1.18. That figure is the winner of a contest among fifteen lengths on the same data, so some of it is luck.

On the test months the same rule, unchanged, earned a Sharpe ratio of 0.90: a drop of 24% from training. MarketHeist's backtester colours a drop under 15% green, from 15% to 35% amber and above 35% red, so this one sits in the amber band: the idea held up in part, with a noticeably weaker result than the training figure promised.

Choosing by training Sharpe did no better than a setting fixed in advance by any meaningful margin. The famous 10-month length, never tuned, earned 0.89 over the same test months, close to the tuned choice.

The test set is also where the rule's trade-off is measured honestly. Over the test months the filtered portfolio grew 6.7% a year against 9.2% for simply holding it, with a worst decline of -11.5% against -20.2%: less growth for a shallower fall.

Data through Sep 2026. Reproduce id: 35ebfa0f424bbfd4 (this guide, its lab setup and the data month).

## How MarketHeist computes it

- **Data**: Monthly total returns (dividends reinvested). Funds younger than the period shown are extended backward with older funds, spliced on returns, not prices.
- **Window**: The test months of the worked example and the lab are the stitched test windows, Feb 2014 to Sep 2026 (152 months). The folds are cut from the months after the longest filter length the lab offers (24 months) has its first signal; the last test window runs to the end of the data: months left after the last full window form a fold of their own when there are at least 12 of them, and otherwise are added to the previous test window, so no month is dropped.
- **Portfolio streams**: A fund is its proxy-extended monthly return stream. A built-in portfolio is read as the MarketHeist MCP reads it: its composed stream with its own rebalancing rule and overlay. A custom mix is rebalanced yearly.
- **Walk-forward**: In each fold, every filter length in the lab's range is scored by its Sharpe ratio over the training window and the highest-scoring length (the shortest on a tie) trades the next test window; the test windows are stitched into one series. A rolling fold slides the training window forward by one test window; an anchored fold keeps its start and grows. The comparison filter has a fixed length of 10 months, and buy and hold is measured over the same test months. Decay is 1 minus the stitched test Sharpe ratio over the average training Sharpe ratio of the chosen lengths.
- **Rule tested**: The whole-portfolio trend filter of the trend-filter guide: hold the portfolio while its value is above its N-month moving average, else cash at 0, decided at month-end and applied the next month. The portfolio is first composed with its own rebalancing rule. (`applyPortfolioOverlay` in packages/portfolio/engine/overlay.ts)
- **Window**: Every length is measured over one window that starts after the longest length the lab offers has warmed up, so the training and test months are the same for every candidate and do not move with the range. (`SWEEP_ANCHOR_MONTHS` in content-engine/lab/compute4.ts)
- **Choice on the training months**: For each candidate length in the range, the Sharpe ratio (mean over sample standard deviation of monthly returns, times √12, risk-free rate 0) over the training months; the length with the highest wins, the shortest on a tie. Nothing from the test months is read. (`walkforwardData` in content-engine/lab/compute4.ts)
- **Folds**: Training months [a, b) and test months [b, c), with only complete test windows counted. One training window followed by one test window is a train/test split; more folds slide (or, anchored, grow) the training window by one test window each time, which is walk-forward analysis. Months after the last complete test window are left out. (`walkFolds` in content-engine/lab/compute4.ts)
- **Reference lines**: The fixed 10-month length and buy and hold are measured over exactly the same test months. (`WALK_FIXED_N` in content-engine/lab/compute4.ts)
- **Backtester's train/test panel**: Splits the loaded price history at a chosen percentage (70% training by default, at least 10 bars on each side), runs the same strategy configuration independently on each part, and compares Sharpe, CAGR, maximum drawdown, Calmar, Sortino, Omega and time invested. It does not choose settings for you: the settings in the configuration are tested as they are, and indicators warm up again at the start of the test part. Built-in indicators only. (`TrainTestSplitModal` in apps/backtest/src/ui/TrainTestSplitModal.tsx)
- **Engine functions**: `applyPortfolioOverlay` (packages/portfolio/engine/overlay.ts), `sharpe` (packages/shared/engine/metrics.ts), `metricsFromReturns` (packages/portfolio/engine/portfolio.ts)

## Train/test split (out-of-sample testing) compared with its neighbours

Train/test split and other ways to check a backtest

| Method | How | Test periods | Main weakness |
| --- | --- | --- | --- |
| Full-sample backtest | Design and measure on the same history | None | Measures fit, not skill |
| Train/test split | Design on the earlier part, test once on the later part | One | One sample of luck; spoiled by repeated peeking |
| Walk-forward analysis | Repeat the split, sliding or growing the training window | Several, stitched | Fewer training months per fold; still one history |
| Cross-validation (combinatorial) | Many train/test combinations of blocks of time | Many | Test blocks can precede training blocks, so leakage needs care |
| Parameter plateau | Check that neighbouring settings perform similarly | None, a stability check | A stable plateau can still be fitted to the period |
| Deflated Sharpe ratio | Discount the Sharpe ratio for the number of trials | None, a statistical correction | Needs an honest count of trials |

## Limitations

- One split is a single experiment. A test period that happens to suit the rule, or not, moves the verdict, and the result depends on where the split falls.
- The test is only clean once. Looking at the test result and then revising the strategy, even slightly, makes the test period part of the training data.
- Markets change. A drop from training to test can be overfitting, or a genuine change in how markets behaved; a split alone cannot tell the two apart.
- In the lab only one setting, the filter length, is chosen. A real design process makes many more choices (the rule itself, the assets, the period), each of which adds to the luck in the training result.
- The simulation ignores trading costs and taxes and earns nothing on cash; long histories are extended backward with older funds or indexes.

## Frequently asked questions

**What is out-of-sample testing?**

Measuring a strategy on data that played no part in designing it. In a train/test split, the out-of-sample data is the later test period that was set aside before any design choices were made.

**How much does a backtest usually degrade out of sample?**

Some degradation is normal, because the training figure is the winner of a search and so flattered by luck. The backtester's colour bands treat a drop under 15% as small, 15% to 35% as moderate and above 35% as severe; these are rules of thumb, not statistical tests.

**What ratio of training to test data is common?**

Splits of about 70/30 or 80/20 are common, and the backtester starts at 70% training. A longer test period gives a more reliable verdict but leaves less data for design; with monthly data, a test period of only a few years is a small sample.

**Can the test period be better than the training period?**

Yes. If the test years happen to suit the rule, the test result can beat the training result. That is not proof of robustness either: it is one period's luck in the other direction.

**What is the difference between a train/test split and walk-forward analysis?**

A train/test split has one training period and one test period. Walk-forward analysis repeats the process through time, re-choosing the settings on each new training window and testing each choice on the period that follows, then stitches the test periods into one out-of-sample record.

## References

1. Lo, A. W. and MacKinlay, A. C. (1990) Data-Snooping Biases in Tests of Financial Asset Pricing Models. Review of Financial Studies 3(3).
2. White, H. (2000) A Reality Check for Data Snooping. Econometrica 68(5).
3. Bailey, D. H., Borwein, J. M., López de Prado, M. and Zhu, Q. J. (2014) Pseudo-Mathematics and Financial Charlatanism: The Effects of Backtest Overfitting on Out-of-Sample Performance. Notices of the American Mathematical Society 61(5).
4. Bailey, D. H. and López de Prado, M. (2014) The Deflated Sharpe Ratio: Correcting for Selection Bias, Backtest Overfitting and Non-Normality. Journal of Portfolio Management 40(5).
5. Harvey, C. R., Liu, Y. and Zhu, H. (2016) ... and the Cross-Section of Expected Returns. Review of Financial Studies 29(1).

## Try it

App: https://marketheist.io/backtest/

MCP prompt (MarketHeist MCP server, https://marketheist.io/mcp):

```
Using MarketHeist.io, compare a 60/40 portfolio with 6-, 10- and 12-month trend filters against plain 60/40 (CAGR, max drawdown, Sharpe), then explain how a train/test split would show whether a filter length chosen on earlier years holds up on later years it was not chosen on
```

Historical simulation, not advice.
