Train/test split: out-of-sample testing for a backtest

On this page

Key takeaways

  • Any setting chosen because it looked strongest on a history tends to look worse on new data: part of its advantage was luck in that particular sample. The test period estimates the result without that luck.
  • The split only works if nothing about the test period influenced the design. Choosing settings on the whole history and then splitting it measures nothing.
  • Degradation is how much a result drops from the training period to the test period, in percent. MarketHeist's backtester colours a drop under 15% green, 15% to 35% amber and above 35% red.
  • One split is one experiment: the test period has its own luck, and markets change between periods. Walk-forward analysis repeats the split through time to get several test periods instead of one.
  • MarketHeist's lab picks the trend-filter length with the strongest training Sharpe ratio and applies it, unchanged, to the following test months; with one training window and one test window, it is a train/test split.

Definition

A train/test split divides a backtest's history into an earlier training period, used to design the strategy and choose its settings, and a later test period, used only once to measure how those settings perform on data they were not fitted to. The drop from training to test results, often called degradation, is a direct measure of how much of the backtest was fitted to noise.

Also known as: out-of-sample test, in-sample vs out-of-sample, holdout test, backtest overfitting, data snooping, performance degradation, IS/OOS split.

How it works

Backtests are usually built by trying ideas and settings until the result looks good. Each try that is kept because it beat the others carries some luck from the specific history it was tested on: a moving-average length that happened to dodge two crashes, a threshold that happened to sit just above a few bad months. Run on new data, that luck does not repeat, and the result falls back. The more settings tried, the larger the fall tends to be.

A train/test split puts the fall in plain view. Set aside the later part of the history before starting. Design and tune the strategy on the earlier part only, then run the final version once on the later part. The test result is an honest estimate because the test data played no role in the choices. The gap between the two is the degradation.

It works like an exam held back from the practice papers: a student who memorised the practice answers scores well on them and poorly on the exam, while one who learned the subject scores similarly on both. A strategy whose test results are close to its training results learned something general; one whose results collapse memorised the training period.

A split has limits. One test period is one sample, with its own luck: a market regime that suits the strategy can flatter the test, and one that does not can sink a sound idea. The split also costs data, because the test months cannot be used for design. And it is easily spoiled: looking at the test result, changing the strategy and testing again turns the test period into training data. Walk-forward analysis and the deflated Sharpe ratio are two ways to go further.

Formula

Degradation of a metric from training to test
D=max(0,Mtrain−Mtest|Mtrain|)
D
degradation: 0 when the test result is as good or better, 1 when it falls to zero
M_train
the metric (Sharpe ratio, CAGR, Calmar, Sortino, Omega or time invested) on the training data
M_test
the same metric on the test data

This is the backtester's version. For maximum drawdown, where smaller is better, it is (|DD_test| − |DD_train|) / |DD_train|. The lab on this page reports the Sharpe ratio form without the floor at zero, 1 − SR_test / SR_train, so a test period that did better shows as a negative drop.

Choosing a setting on the training data
N*=argmaxN∈{Nmin,…,Nmax}SRtrain(N),reported:SRtest(N*)
N
a candidate setting, here the trend filter's moving-average length in months
N*
the setting with the highest Sharpe ratio on the training data (the shortest on a tie)
SR_train(N), SR_test(N)
Sharpe ratio of the rule with setting N on the training and test months

The training figure SR_train(N*) is the maximum of several noisy estimates, so it is biased upward; the test figure is not, because N* was fixed before the test months were read.

Sharpe ratio on each side
SR=r¯s12
r̄
mean monthly return over the months of that side
s
sample standard deviation of those monthly returns

Risk-free rate 0, as elsewhere on MarketHeist; months out of the market earn 0.

Worked example

The classic 60/40 stocks-and-bonds portfolio with a whole-portfolio trend filter, from Feb 2014 to Sep 2026. The first 300 months, 25 years, are the training set; every month after that, through Sep 2026, is the test set, kept out of every choice.

On the training months the strongest trend-filter length, among every length from 4 to 18 months, was 12 months, with a Sharpe ratio of 1.18. That figure is the winner of a contest among fifteen lengths on the same data, so some of it is luck.

On the test months the same rule, unchanged, earned a Sharpe ratio of 0.90: a drop of 24% from training. MarketHeist's backtester colours a drop under 15% green, from 15% to 35% amber and above 35% red, so this one sits in the amber band: the idea held up in part, with a noticeably weaker result than the training figure promised.

Choosing by training Sharpe did no better than a setting fixed in advance by any meaningful margin. The famous 10-month length, never tuned, earned 0.89 over the same test months, close to the tuned choice.

The test set is also where the rule's trade-off is measured honestly. Over the test months the filtered portfolio grew 6.7% a year against 9.2% for simply holding it, with a worst decline of -11.5% against -20.2%: less growth for a shallower fall.

Data through Sep 2026. Reproduce id 35ebfa0f424bbfd4 (this guide, its lab setup and the data month).

Interactive lab

Growth of $10,000 over the test months for the trend-filter length chosen on the training months, the fixed 10-month length and buy and hold, with the training and test Sharpe ratios. Move the training and test windows to turn the single split into a walk-forward.

Growth of $10,000 over the test months: walk-forward, fixed 10-month filter and buy and holdClassic Stocks & Bonds 60/40, the 1 test windows of 144 months stitched together (Feb 2014 to Sep 2026). $10,000 grows to $22,729 with the walk-forward choice, $22,174 with a fixed 10-month filter and $30,644 buy and hold.$20k2014201620182020202220242026Walk-forward choiceFixed 10-month filterBuy and hold
Growth of $10,000 on a log scale over the stitched test months. Feb 2014 to Sep 2026.
MeasureWalk-forwardFixed 10 monthsBuy and hold
Sharpe ratio (test months)0.900.890.97
Sharpe ratio in training, average of the chosen lengths1.18
Decay (1 minus test over training)24%
CAGR6.7%6.5%9.2%
Maximum drawdown-11.5%-11.5%-20.2%
Growth of $10,000$22,729$22,174$30,644

Rolling walk-forward: the length from 4 to 18 months with the highest Sharpe ratio in each 300-month training window trades the next 144 months; the test windows are stitched. Risk-free rate 0; cash earns 0. Feb 2014 to Sep 2026.

FoldTraining windowTest windowChosen lengthTraining SharpeTest SharpeTest Sharpe at 10Test Sharpe, buy and hold
1Feb 1989 to Jan 2014 (300)Feb 2014 to Sep 2026 (152)121.180.900.890.97

Months in brackets. The chosen length is the one with the highest Sharpe ratio in the training window (the shortest on a tie). Data from Feb 1989, 452 months. The last test window runs to the end of the data. Feb 2014 to Sep 2026.

How MarketHeist computes it

Every choice an implementation has to make, as the code makes it today. The page is checked against the code each time it is built.

ChoiceHow it is computedIn the code
DataMonthly total returns (dividends reinvested). Funds younger than the period shown are extended backward with older funds, spliced on returns, not prices.
WindowThe test months of the worked example and the lab are the stitched test windows, Feb 2014 to Sep 2026 (152 months). The folds are cut from the months after the longest filter length the lab offers (24 months) has its first signal; the last test window runs to the end of the data: months left after the last full window form a fold of their own when there are at least 12 of them, and otherwise are added to the previous test window, so no month is dropped.
Portfolio streamsA fund is its proxy-extended monthly return stream. A built-in portfolio is read as the MarketHeist MCP reads it: its composed stream with its own rebalancing rule and overlay. A custom mix is rebalanced yearly.
Walk-forwardIn each fold, every filter length in the lab's range is scored by its Sharpe ratio over the training window and the highest-scoring length (the shortest on a tie) trades the next test window; the test windows are stitched into one series. A rolling fold slides the training window forward by one test window; an anchored fold keeps its start and grows. The comparison filter has a fixed length of 10 months, and buy and hold is measured over the same test months. Decay is 1 minus the stitched test Sharpe ratio over the average training Sharpe ratio of the chosen lengths.
Rule testedThe whole-portfolio trend filter of the trend-filter guide: hold the portfolio while its value is above its N-month moving average, else cash at 0, decided at month-end and applied the next month. The portfolio is first composed with its own rebalancing rule.applyPortfolioOverlaypackages/portfolio/engine/overlay.ts
WindowEvery length is measured over one window that starts after the longest length the lab offers has warmed up, so the training and test months are the same for every candidate and do not move with the range.SWEEP_ANCHOR_MONTHScontent-engine/lab/compute4.ts
Choice on the training monthsFor each candidate length in the range, the Sharpe ratio (mean over sample standard deviation of monthly returns, times √12, risk-free rate 0) over the training months; the length with the highest wins, the shortest on a tie. Nothing from the test months is read.walkforwardDatacontent-engine/lab/compute4.ts
FoldsTraining months [a, b) and test months [b, c), with only complete test windows counted. One training window followed by one test window is a train/test split; more folds slide (or, anchored, grow) the training window by one test window each time, which is walk-forward analysis. Months after the last complete test window are left out.walkFoldscontent-engine/lab/compute4.ts
Reference linesThe fixed 10-month length and buy and hold are measured over exactly the same test months.WALK_FIXED_Ncontent-engine/lab/compute4.ts
Backtester's train/test panelSplits the loaded price history at a chosen percentage (70% training by default, at least 10 bars on each side), runs the same strategy configuration independently on each part, and compares Sharpe, CAGR, maximum drawdown, Calmar, Sortino, Omega and time invested. It does not choose settings for you: the settings in the configuration are tested as they are, and indicators warm up again at the start of the test part. Built-in indicators only.TrainTestSplitModalapps/backtest/src/ui/TrainTestSplitModal.tsx
Engine functionsThe functions the portfolio app and the MarketHeist MCP tools run, called here on the same monthly returns.applyPortfolioOverlaypackages/portfolio/engine/overlay.tssharpepackages/shared/engine/metrics.tsmetricsFromReturnspackages/portfolio/engine/portfolio.ts

Train/test split (out-of-sample testing) compared with its neighbours

Train/test split and other ways to check a backtest
MethodHowTest periodsMain weakness
Full-sample backtestDesign and measure on the same historyNoneMeasures fit, not skill
Train/test splitDesign on the earlier part, test once on the later partOneOne sample of luck; spoiled by repeated peeking
Walk-forward analysisRepeat the split, sliding or growing the training windowSeveral, stitchedFewer training months per fold; still one history
Cross-validation (combinatorial)Many train/test combinations of blocks of timeManyTest blocks can precede training blocks, so leakage needs care
Parameter plateauCheck that neighbouring settings perform similarlyNone, a stability checkA stable plateau can still be fitted to the period
Deflated Sharpe ratioDiscount the Sharpe ratio for the number of trialsNone, a statistical correctionNeeds an honest count of trials

Limitations

  • One split is a single experiment. A test period that happens to suit the rule, or not, moves the verdict, and the result depends on where the split falls.
  • The test is only clean once. Looking at the test result and then revising the strategy, even slightly, makes the test period part of the training data.
  • Markets change. A drop from training to test can be overfitting, or a genuine change in how markets behaved; a split alone cannot tell the two apart.
  • In the lab only one setting, the filter length, is chosen. A real design process makes many more choices (the rule itself, the assets, the period), each of which adds to the luck in the training result.
  • The simulation ignores trading costs and taxes and earns nothing on cash; long histories are extended backward with older funds or indexes.

Frequently asked questions

What is out-of-sample testing?
Measuring a strategy on data that played no part in designing it. In a train/test split, the out-of-sample data is the later test period that was set aside before any design choices were made.
How much does a backtest usually degrade out of sample?
Some degradation is normal, because the training figure is the winner of a search and so flattered by luck. The backtester's colour bands treat a drop under 15% as small, 15% to 35% as moderate and above 35% as severe; these are rules of thumb, not statistical tests.
What ratio of training to test data is common?
Splits of about 70/30 or 80/20 are common, and the backtester starts at 70% training. A longer test period gives a more reliable verdict but leaves less data for design; with monthly data, a test period of only a few years is a small sample.
Can the test period be better than the training period?
Yes. If the test years happen to suit the rule, the test result can beat the training result. That is not proof of robustness either: it is one period's luck in the other direction.
What is the difference between a train/test split and walk-forward analysis?
A train/test split has one training period and one test period. Walk-forward analysis repeats the process through time, re-choosing the settings on each new training window and testing each choice on the period that follows, then stitches the test periods into one out-of-sample record.

References

  1. Lo, A. W. and MacKinlay, A. C. (1990) Data-Snooping Biases in Tests of Financial Asset Pricing Models. Review of Financial Studies 3(3).
  2. White, H. (2000) A Reality Check for Data Snooping. Econometrica 68(5).
  3. Bailey, D. H., Borwein, J. M., López de Prado, M. and Zhu, Q. J. (2014) Pseudo-Mathematics and Financial Charlatanism: The Effects of Backtest Overfitting on Out-of-Sample Performance. Notices of the American Mathematical Society 61(5).
  4. Bailey, D. H. and López de Prado, M. (2014) The Deflated Sharpe Ratio: Correcting for Selection Bias, Backtest Overfitting and Non-Normality. Journal of Portfolio Management 40(5).
  5. Harvey, C. R., Liu, Y. and Zhu, H. (2016) ... and the Cross-Section of Expected Returns. Review of Financial Studies 29(1).

Try it on your own holdings

Run a backtest and open its train/test split panel →

Or ask an assistant connected to the MarketHeist MCP server (https://marketheist.io/mcp), which computes the same figures:

Using MarketHeist.io, compare a 60/40 portfolio with 6-, 10- and 12-month trend filters against plain 60/40 (CAGR, max drawdown, Sharpe), then explain how a train/test split would show whether a filter length chosen on earlier years holds up on later years it was not chosen on

Ask an AI assistant

The prompt opens in a new tab, already filled in. An assistant without the MarketHeist connector answers from its own knowledge, not from the figures on this page.

Historical simulation, not advice. Computed by the MarketHeist portfolio engine from monthly total returns, data through Sep 2026. Markdown version of this page: train-test-split.md.