Parameter robustness and overfitting: the plateau test
On this page
Key takeaways
- Every backtest with an adjustable setting is one point on a curve. The plateau test reads the neighbours: if settings one and two steps away keep most of the result, the point sits on a plateau; if they collapse, it sits on a spike.
- MarketHeist compares the chosen setting with its four nearest neighbours: if they keep 70% or more of its Sharpe ratio it is robust, 50% to 70% sensitive, below 50% fragile. The robustness pages and the lab give no verdict when the chosen Sharpe ratio is below 0.2.
- Picking the highest result of a sweep is a selection: the top of many noisy estimates is biased upward even when no setting has any real skill. Bailey, Borwein, López de Prado and Zhu (2014) showed how few trials it takes to produce an impressive Sharpe ratio by chance.
- The deflated Sharpe ratio corrects a Sharpe ratio for the number of trials behind it; walk-forward analysis tests a selection rule on data it did not see.
- A plateau is necessary, not sufficient: a whole region of settings can share the same luck if the history contains one decisive episode.
Definition
Parameter robustness is how little a strategy's result changes when its settings change slightly. A rule whose performance holds across a broad plateau of nearby settings is more likely to reflect a real effect; one that only works at a sharp peak is likely to have been fitted to the noise of the particular history tested, which is called overfitting.
Also known as: plateau test, parameter sensitivity, backtest overfitting, curve fitting, parameter sweep, data snooping, parameter stability, overfit badge.
How it works
A trading rule usually has a number in it: the length of a moving average, a threshold, a lookback. Run the rule at many values of that number and plot the result, typically the Sharpe ratio, against the value. That is a parameter sweep. A rule that captures a persistent pattern in markets tends to produce a broad, gently sloping hill; a rule that happens to match the twists of one particular history produces a jagged line where the top is a lone spike.
The plateau test turns that picture into a number. Take the setting chosen in advance, the famous one from the literature where there is one, and read the settings one and two steps on either side. If their median Sharpe ratio keeps most of the chosen setting's, the result does not hinge on the exact value. MarketHeist's robustness pages and the backtester's overfit badge use the same cutoffs: robust when the neighbours keep at least 70%, sensitive between 50% and 70%, fragile below. When the chosen setting's Sharpe ratio is below 0.2, a ratio of two small numbers is noise, and no verdict is given.
The other half of the problem is selection. Each setting's Sharpe ratio is an estimate with sampling error, so the highest of many estimates is higher than the true value of the setting that produced it, the way the tallest of many randomly chosen people is taller than average. Bailey, Borwein, López de Prado and Zhu (2014) quantified this for backtests: with enough independent trials, an in-sample Sharpe ratio that looks excellent becomes the expected maximum of pure noise, and the minimum length of history needed to rule that out grows with the logarithm of the number of trials.
A sweep is like tuning a radio: a real station comes in clearly across a small band of the dial, while static can sound like a voice for an instant at one exact spot. The plateau test turns the dial a notch each way to see whether the voice is still there.
The defences fit together. Fix the setting before testing, from theory or from published work, and report it whatever it shows. Show the neighbours, so a reader can see whether the result is a peak or part of a ridge. If a setting was chosen by search, deflate its Sharpe ratio for the number of trials, or test the search itself with walk-forward analysis. MarketHeist's 200-day moving average robustness page applies these checks to a daily rule on SPY with trading costs; the lab here runs the same idea on monthly portfolio values.
Formula
- N₀
- the setting chosen in advance (10 months in the lab)
- S_N
- Sharpe ratio of the rule at setting N, over the same window for every N
- ρ
- plateau ratio: the share of the chosen setting's Sharpe ratio its neighbours keep (above 1 when they did better)
Defined only when S_N₀ is at least 0.2. The backtester's overfit badge perturbs a rule's own parameters by one and two steps in the same way; the robustness pages use the neighbouring cells of their grid.
- K
- number of independent trials (settings or strategies) tried
- SR̂ₖ
- the Sharpe ratio estimated for trial k
- σ_SR
- standard deviation of the Sharpe ratio estimates across the trials
- γ
- the Euler–Mascheroni constant, about 0.5772
- Φ⁻¹
- inverse of the standard normal cumulative distribution function
- e
- Euler's number, about 2.718
The deflated Sharpe ratio is the probabilistic Sharpe ratio tested against this expected maximum instead of zero. Neighbouring settings of one rule are strongly correlated, so they count as fewer than K independent trials; estimating the effective number is part of the method.
Worked example
The S&P 500 (SPY, extended back with its index mutual fund) with the whole-portfolio trend filter at every length from 2 to 18 months, all measured over the same months, Mar 1987 to Sep 2026.
At the length chosen in advance, the 10-month filter had a Sharpe ratio of 0.80, against 0.77 for holding the S&P 500 throughout. It cut the worst decline to -26.4% from -50.8% and grew more slowly, 8.7% a year against 10.8%. Across all 17 lengths the Sharpe ratio ran from 0.64, at 4 months, to 0.89, at 17 months, with a median of 0.84.
The plateau test reads the four neighbours of 10: 8, 9, 11 and 12 months. Their median Sharpe ratio was 107.1% of the 10-month figure, above the 70% that MarketHeist calls robust. The result does not hinge on choosing exactly 10; nearby lengths did about as well.
The weak region was the short end, where a filter of a few months reacts to every dip and pays for it in whipsaws; the lowest Sharpe ratio came at 4 months. In all, 13 of the 17 lengths beat buy-and-hold on Sharpe ratio, so the filter's appeal here rests on a region of settings, not on one.
Quoting the top of the sweep would overstate it. The figure of 0.89 at 17 months is the highest of 17 estimates from one history, and the highest of many noisy estimates is biased upward; the honest figure for this rule is the one at the length fixed before looking. Whether picking the top length each time would have worked going forward is a different test, the subject of walk-forward analysis.
Data through Sep 2026. Reproduce id b0afae80204c20c9 (this guide, its lab setup and the data month).
Interactive lab
Sharpe ratio of the trend filter at every length from 2 to 18 months over the same window, with buy-and-hold as a dashed line, the chosen length marked and its plateau neighbours shaded. Switch portfolios or move the chosen length.
| Measure | S&P 500 (100%) |
|---|---|
| Famous length (10 months): Sharpe ratio | 0.80 |
| Highest-scoring length (17 months): Sharpe ratio | 0.89 |
| Worst length (4 months): Sharpe ratio | 0.64 |
| Median Sharpe ratio of all lengths | 0.84 |
| Plateau (median of lengths 8, 9, 11, 12 over the famous one) | 107% |
| Buy-and-hold Sharpe ratio | 0.77 |
| Lengths beating buy and hold | 13 of 17 |
Every length is measured over the same months. The plateau is defined when the famous length's Sharpe ratio is at least 0.2. Risk-free rate 0; cash earns 0. Mar 1987 to Sep 2026.
| Length (months) | Sharpe ratio | CAGR | Maximum drawdown | Months in cash |
|---|---|---|---|---|
| Buy and hold | 0.77 | 10.8% | -50.8% | 0 |
| 2 | 0.70 | 6.7% | -30.6% | 165 |
| 3 | 0.65 | 6.3% | -31.7% | 151 |
| 4 | 0.64 | 6.5% | -32.6% | 133 |
| 5 | 0.72 | 7.3% | -29.9% | 129 |
| 6 | 0.78 | 8.2% | -25.8% | 121 |
| 7 | 0.80 | 8.6% | -23.5% | 113 |
| 8 | 0.87 | 9.3% | -23.5% | 111 |
| 9 | 0.85 | 9.3% | -23.5% | 104 |
| 10 | 0.80 | 8.7% | -26.4% | 104 |
| 11 | 0.84 | 9.3% | -26.4% | 97 |
| 12 | 0.88 | 9.9% | -26.4% | 98 |
| 13 | 0.81 | 9.2% | -26.4% | 93 |
| 14 | 0.84 | 9.7% | -26.4% | 91 |
| 15 | 0.84 | 9.7% | -26.4% | 87 |
| 16 | 0.87 | 10.3% | -26.4% | 84 |
| 17 | 0.89 | 10.4% | -23.8% | 83 |
| 18 | 0.88 | 10.3% | -23.8% | 85 |
The portfolio is held while its value is above its N-month average and sits in cash otherwise. Mar 1987 to Sep 2026.
How MarketHeist computes it
Every choice an implementation has to make, as the code makes it today. The page is checked against the code each time it is built.
| Choice | How it is computed | In the code |
|---|---|---|
| Data | Monthly total returns (dividends reinvested). Funds younger than the period shown are extended backward with older funds, spliced on returns, not prices. | |
| Window | The worked example and the lab measure every filter length over the same Mar 1987 to Sep 2026 (475 months), which starts once the longest length the lab offers (24 months) has its first signal, so the lengths are comparable and the window does not move when the range does. It starts later than the window of a single trend-filter page, so a length's figures here differ from that page's. | |
| Portfolio streams | A fund is its proxy-extended monthly return stream. A built-in portfolio is read as the MarketHeist MCP reads it: its composed stream with its own rebalancing rule and overlay. A custom mix is rebalanced yearly. | |
| Trend-filter sweep | The portfolio-level trend filter of the trend-filter pages and the overlay lab, run for every length from the lab's shortest to its longest: invested while the portfolio's value is above its average over the last N month-ends, in cash otherwise, the decision applying to the following month, cash earning 0. Each length's Sharpe ratio is over the same months. The plateau is the median Sharpe ratio of the four lengths within two months of the famous one, divided by the famous length's; it is left undefined when the famous Sharpe ratio is under 0.2, as on the robustness pages. | |
| Rule | The whole-portfolio trend filter of the trend-filter guide: hold the portfolio while its month-end value is above its N-month simple moving average, otherwise cash at 0, decided at month-end and applied the next month, no trading costs. | applyPortfolioOverlaypackages/portfolio/engine/overlay.ts |
| One window for every length | Every length is measured over the same months, starting after a 24-month warm-up (the longest length the lab offers), so their Sharpe ratios are comparable and the window does not move when the range changes. Because the trend-filter guide warms up for 12 months, its 10-month figures differ slightly from this page's. | SWEEP_ANCHOR_MONTHScontent-engine/lab/compute4.ts |
| Sharpe ratio | Monthly returns, risk-free rate 0, sample standard deviation, ×√12, for the filtered portfolio and for buy-and-hold over the same months. | sharpepackages/shared/engine/metrics.ts |
| Plateau | Median Sharpe ratio of the lengths N₀ − 2, N₀ − 1, N₀ + 1 and N₀ + 2 divided by the Sharpe ratio at N₀; not computed when the Sharpe ratio at N₀ is below 0.2. | MIN_FAMOUS_SHARPEpackages/shared/engine/robustness.ts |
| Verdict thresholds | Robust at 0.7 or more, sensitive from 0.5 to 0.7, fragile below 0.5: the function shared by the backtester's overfit badge, the portfolio overlay check and the robustness pages. | stabilityVerdictpackages/shared/engine/overfit.ts |
| Backtester's overfit badge | For a single-indicator backtest with a positive Sharpe ratio, the backtester moves one numeric parameter at a time one and two steps each way (the parameter's declared step, else 1 for whole numbers, else 10% of its value), runs at most 8 such neighbours in total on the same data, and reports the median neighbour Sharpe ratio over the base Sharpe ratio with the same verdict; it needs at least 2 neighbours. The robustness pages run full grids of daily rules with trading costs and a Treasury-bill cash leg. | assessParameterStabilitypackages/shared/engine/overfit.ts |
| No deflation | Neither the lab nor the app computes the deflated Sharpe ratio: it needs the number of independent trials, which a sweep of correlated settings does not give directly. | |
| Engine functions | The functions the portfolio app and the MarketHeist MCP tools run, called here on the same monthly returns. | applyPortfolioOverlaypackages/portfolio/engine/overlay.tstrendSignalpackages/portfolio/engine/overlay-activity.tsmetricsFromReturnspackages/portfolio/engine/portfolio.tsMIN_FAMOUS_SHARPEpackages/shared/engine/robustness.ts |
Parameter robustness and overfitting (the plateau test) compared with its neighbours
| Check | Question it answers | What it needs | What it misses |
|---|---|---|---|
| Fixed setting chosen in advance | Does the rule work at a value nobody tuned? | A setting from theory or the literature | Whether that value was itself selected by others' searches |
| Plateau test | Do nearby settings keep the result? | A sweep over the same window | Luck shared by a whole region of settings |
| Deflated Sharpe ratio | Is the top result better than the expected maximum of noise? | The number of independent trials | Trials nobody recorded |
| Walk-forward analysis | Does the selection rule work on data it has not seen? | Enough history for several test windows | Only one path through history; few independent test windows |
| Train/test split | Does the chosen setting hold on a later period? | One held-out period | A single split can be lucky or unlucky |
| Bootstrap or Monte Carlo | How much could the result vary with other draws of the same months? | A model of how returns are resampled | Effects absent from the history |
Limitations
- The plateau is measured on one history. If a single episode, such as one long bear market, drives the result for every nearby setting, the whole plateau shares that luck.
- The neighbours are fixed at one and two steps. For a rule whose natural step is coarser or finer, the same test can call a spike a plateau or the reverse.
- The lab's rule earns 0 in cash and pays no trading costs; shorter lengths trade more, so costs would lower the short end further.
- A robust verdict says nothing about whether the rule beats a simpler alternative; read the plateau beside buy-and-hold.
- Long histories are extended backward with older funds or indexes where a fund is younger than the window, so early months approximate the fund rather than reproduce it.
Frequently asked questions
- What is overfitting in backtesting?
- Choosing a rule or its settings because they performed well on the history tested, when part of that performance came from noise specific to that history. An overfitted strategy looks strong in the backtest and weaker on new data, because the noise does not repeat.
- How do I know if a moving-average length is overfitted?
- Run the rule at nearby lengths over the same period. If the result holds up at lengths one and two steps away, the chosen length sits on a plateau. If only one exact length works, treat the result as noise, and if the length was picked from many by search, test the search itself on data it did not see.
- Is the highest Sharpe ratio in a parameter sweep the right setting to use?
- Not on that evidence alone. It is the maximum of many estimates from one history, which overstates its true value; the more settings tried, the larger the overstatement. A setting in the middle of a broad plateau is the more defensible choice.
- What is the deflated Sharpe ratio?
- The probability that a strategy's true Sharpe ratio exceeds the Sharpe ratio expected from the top result of the trials run if none of them had skill. Bailey and López de Prado introduced it in 2014; it needs the number of independent trials and the spread of their Sharpe ratios.
References
- Bailey, D. H., Borwein, J. M., López de Prado, M. and Zhu, Q. J. (2014) Pseudo-Mathematics and Financial Charlatanism: The Effects of Backtest Overfitting on Out-of-Sample Performance. Notices of the American Mathematical Society 61(5). Minimum backtest length and the expected maximum Sharpe ratio of many trials.
- Bailey, D. H. and López de Prado, M. (2014) The Deflated Sharpe Ratio: Correcting for Selection Bias, Backtest Overfitting and Non-Normality. Journal of Portfolio Management 40(5).
- White, H. (2000) A Reality Check for Data Snooping. Econometrica 68(5). Testing the best of many rules against the multiplicity of trials.
- Sullivan, R., Timmermann, A. and White, H. (1999) Data-Snooping, Technical Trading Rule Performance, and the Bootstrap. Journal of Finance 54(5). Moving-average and other rules after correcting for the search.
- Harvey, C. R. and Liu, Y. (2015) Backtesting. Journal of Portfolio Management 42(1). Haircuts for Sharpe ratios found by search.
Try it on your own holdings
See the plateau test on the 200-day moving average for SPY →
Or ask an assistant connected to the MarketHeist MCP server (https://marketheist.io/mcp), which computes the same figures:
Using MarketHeist.io, run a 10-month moving-average trend filter on the S&P 500 and on the same portfolio with 8, 9, 11 and 12 months: how much of the 10-month Sharpe ratio do the neighbouring lengths keep, and is the result a plateau or a spike?
Ask an AI assistant
The prompt opens in a new tab, already filled in. An assistant without the MarketHeist connector answers from its own knowledge, not from the figures on this page.
Historical simulation, not advice. Computed by the MarketHeist portfolio engine from monthly total returns, data through Sep 2026. Markdown version of this page: parameter-robustness.md.