Robustness Testing: Why Most Traders Fail, and What 36,252 Backtests Say Actually Works

If you have ever built a strategy that printed money on historical data and then quietly bled out in live trading, this article is for you. Not because your code was wrong. Because nobody ever told you what a lucky backtest looks like, so you had nothing to compare yours to.

Everyone in this corner of the internet tells you to run robustness tests. Split your data. Walk it forward. Run a Monte Carlo. Check your parameter stability. What almost nobody does is measure whether those tests work — whether a strategy that passes them actually does better afterwards than one that does not.

So I measured it. 5,472 parameter backtests across 8 markets and 3 strategy families, plus 5,000 randomly generated rules mined on SPY and re-mined on data engineered to contain no edge at all. Then I scored 11 robustness checks by how well each one predicted what happened in a slice of history the checks never saw.

Some of them are worth every minute. One of them is actively misleading. And the whole exercise buys you something different from what you think you are buying.

Then I did the thing I should have done first, and it cost me some of my own conclusions. Eight markets sounds like breadth. Measured properly, those eight were worth 2.87 independent bets — SPY and ES are one index in two wrappers — so I reran everything on 53 markets and 36,252 backtests. The main finding got stronger. Several of the individual numbers below were overstated by a factor of three, and two of the checks stop working altogether once you leave US equities. That section is near the end, and it is the most useful part of this article.

TL;DR — the answer box

• The noise floor is high, and almost nobody accounts for it. Optimising 384 ordinary RSI variants on three years of shuffled, drift-free data — where an edge cannot exist — produced a median best backtest of 9.2% a year at Sharpe 1.17. On twenty years of the same data it produced 3.1% at Sharpe 0.46. Your backtest is only as impressive as its margin over that number.
• Mining 5,000 rules on real SPY found an 11.2% CAGR strategy. Mining the same 5,000 rules on zero-edge data found a 6.4% one. The real effect is the difference — 4.8% a year — not the headline.
• Keeping the in-sample winner rarely blows up. It just rarely wins. Across 24 market-strategy grids the in-sample champion stayed profitable out-of-sample in 19 of 24 cases, but beat plain buy-and-hold in only 9 of 24.
• Walk-forward delivers about four-tenths of its promise. Median walk-forward efficiency across 471 rolling windows: 0.42. It beat buy-and-hold in 6 of 24 market-strategy combinations.
• Robustness testing buys survival, not outperformance. Variants passing at least 10 of 11 checks were profitable in the untouched final quarter 98.0% of the time versus a 63.7% base rate — while their odds of beating buy-and-hold fell to 20.7% from a 27.8% base rate.
• The stack holds on 53 markets. The individual checks do not. Retested on 36,252 backtests, passing all 11 checks still delivered 97.2% survival — but the parameter-neighbour check drops from +0.598 lift to +0.166, and a development Sharpe above 0.5 collapses from +0.370 to +0.041.
• Where you test decides whether the checks work at all. The neighbour check is worth +0.366 on equity index markets and −0.210 on FX, where variants that pass it survive 29.5% of the time against a 45.6% base rate. Passing it there makes you less likely to survive.

How I tested

Eight markets, all daily bars, all frozen local files: SPY, QQQ, IWM and DIA on the equity side, and ES, NQ, NG and JY as continuous back-adjusted futures. SPY runs from 1993-02-02 to 2026-06-12; the shortest series, the futures, start in January 2007. The last data point anywhere in the study is 2026-07-02.

Three strategy families, chosen because they are what retail traders actually optimise, not because they are clever:

684 variants per market × 8 markets = 5,472 backtests. Signals are read at the close and entered at the next open, so nothing sees a price it could not have traded on. Long only, one unit, no leverage, frictionless unless a cost is explicitly charged. Every variant is measured in-sample (the first 70% of each market's bars), out-of-sample (the last 30%), and over the full sample.

Two things make this more than another grid sweep.

A control group. Every experiment that measures luck is re-run on bootstrapped versions of SPY's own bars, with the order shuffled and the drift removed. Same fat tails, same daily return distribution, no sequence and no upward pull. Buy-and-hold expects zero there, and no rule can have an edge by construction. Whatever a backtest earns on that data is what luck alone produces.

A held-back final quarter. For the scorecard, every check is computed on the first 75% of each market's history and scored on the last 25%, which the checks never touched.

Every number below is computed by research/study.py and written into the study's facts sheet; the full 5,472-row grid is downloadable as grid_master.csv.

Does keeping the in-sample winner actually ruin the strategy?

No — and this is the first place the folklore is wrong.

I optimised each of the 24 market-strategy grids on the in-sample period, kept the single best variant by return, and carried it unchanged into the out-of-sample period.

Read the third row and the fourth row together, because that is the whole story of this article. The optimised strategy usually keeps making money. It usually loses to owning the index.

The winners that did break, broke where you would expect: on markets with no underlying tailwind. Japanese yen futures, RSI mean reversion: 1.7% in-sample became -7.5% out-of-sample, landing at the 10th percentile of its own grid. Natural gas trend and breakout picks lost money in both halves.

There is a catch buried in the method, though. The 70/30 split is itself a choice, and it is not a stable one. Moving the in-sample share from 50% to 70% changes which variant you would have kept in 62.5% of the 24 cells. At 90/10 it changes 91.7% of them. You did not discover the best parameters. You discovered the best parameters for the split you happened to pick.

What does a great backtest look like when there is nothing there?

This is the test almost nobody runs, and it reframes everything else.

Take a bathroom scale that swings five pounds either way. Step on it, read three pounds down, and celebrate — that is a backtest without a noise floor. You are not measuring weight loss. You are measuring the scale.

So I built the scale. Shuffled SPY's own bars, removed the drift, and ran the identical parameter grids on the result. No sequence, no trend, nothing to find. Then I recorded the best variant each search produced.

A Sharpe ratio of 1.17 over three years, from a strategy with zero edge, found by testing a completely ordinary 384-variant grid. That is the median result — half the searches did better, and one in twenty produced a Sharpe of 1.78.

The mechanism is just arithmetic. On the same zero-edge data, testing one RSI variant returns a median Sharpe of 0.00. Testing all 384 returns 0.36. Every extra combination is another lottery ticket, and you keep only the winning one.

This is not a StatOasis invention. It is the same problem Bailey, Borwein, López de Prado and Zhu formalised in The Probability of Backtest Overfitting, and the reason Harvey, Liu and Zhu argued that a newly discovered factor should clear a t-statistic of 3.0 rather than the usual 2.0. The numbers above are what that abstraction looks like on a chart of SPY.

What happens when you search 5,000 rules instead of 384?

A parameter grid is the mild version of searching. The real version — what a strategy generator, a spreadsheet marathon or an AI assistant does — is searching over rules.

So I built a pool of 221 ordinary technical conditions: oscillator levels, moving-average positions, breakouts, consecutive up and down days, volatility regimes, and seven deliberately meaningless calendar conditions like "it is a Tuesday". Then I combined them at random into 5,000 rules of one to three conditions each, gave each a fixed holding period, and backtested every one on SPY.

The winner was RSI(2) < 15, hold 5 days: 11.2% a year at a Sharpe of 0.79 on 339 trades. Out-of-sample it did 9.6%. That is a genuinely good result, and it survived.

Now the same 5,000 rules, mined on the zero-edge data:

Search hard enough on data with no edge and you will find a 23-year backtest returning 6.4% a year at a Sharpe of 0.65. One of the ten runs found 10.4%. If someone showed you that equity curve without telling you where it came from, you would fund it.

The honest read of the SPY result, then, is not "11.2%". It is 11.2% minus 6.4% — the real effect is worth about 4.8% a year of CAGR and 0.13 of Sharpe over what searching that hard produces from nothing.

Two more findings from the mining run, one reassuring and one brutal:

• The junk conditions did not win on real data. Calendar conditions made up 7.2% of the rule pool and 0.0% of the top 25 rules on SPY. Every one of the top 25 was a mean-reversion rule. When there is a real effect in the data, a wide search tends to find it. On the zero-edge data, junk conditions appeared in a median 4.0% of the top 25 — the search had nothing better to grab.
• Beating the market is the rare part. Of the 5,000 rules, 41.5% were profitable in-sample, and 84.5% of those stayed profitable out-of-sample. But only 3.9% beat buy-and-hold in-sample, and only 0.2% — 10 rules out of 5,000 — beat it in both periods. SPY itself returned 6.9% in the in-sample window and 13.4% out-of-sample, price only, no dividends.

Is your in-sample champion just the luckiest one?

There is a formal way to ask that question, and it does not need you to guess. Combinatorially symmetric cross-validation, from the Bailey et al. paper above, chops history into 10 chunks, forms every one of the 252 ways to split them into an in-sample half and an out-of-sample half, and asks how often the in-sample champion lands below the out-of-sample median. That share is the probability of backtest overfitting, or PBO. 50% means your selection carries no information whatsoever.

The median across all 24 grids was 43.7%, and 7 of 24 scored above 50% — selection that was actively worse than picking at random.

Look at the spread between families, because this is the finding I did not expect. The identical test, run with identical code on identical markets, says RSI mean reversion selection is broadly informative and moving-average trend selection is close to noise. There is no such thing as "my process is robust". There is only "this effect held up, and that one did not".

What does walk-forward actually deliver?

Walk-forward analysis is the method every guide calls the gold standard: optimise on a window, trade the next window unchanged, roll forward, repeat. I ran it properly — 4 years in-sample, 1 year traded, stepped annually, 471 windows across all 24 market-strategy combinations.

Walk-forward efficiency is the traded year's return divided by the optimised return that justified choosing those parameters. The median is 0.42. You keep about four-tenths of what the optimiser promised — which is exactly what walk-forward is for, since it tells you that in advance instead of after you have funded it.

The parameter churn is the underrated number. On SPY, the optimiser picked 17 different "best" parameter sets across 29 annual windows. Whatever the optimiser found each year, it did not stay found.

And walk-forward is not a way to beat the market. It beat buy-and-hold in 6 of 24 cells — and those wins are concentrated in the two markets that fell over the period, natural gas and the yen, where being out of the market most of the time was the entire edge.

Should you pick the peak or the middle of the plateau?

Every robustness guide tells you not to pick the spike on the parameter surface — pick the middle of the broad flat region, because a peak surrounded by cliffs is an accident. I have written that advice myself. So I tested it: for each grid, the outright in-sample best against the variant with the best average result across its immediate parameter neighbourhood.

It made no difference.

I expected the plateau pick to win. The data said it does not, on this grid, on these markets, at this resolution. Report it and move on.

But the related check does work, and the distinction matters. Asking "is the whole neighbourhood profitable?" — not "which point in it is highest?" — was one of the strongest predictors in the entire study. More on that below.

The one hard threshold anybody quotes, tested

There is a rule that circulates in exactly one form: vary a parameter by 25%, and the metric should move by less than 25%. It is the only concrete robustness number most traders can name. So I ran it.

I went in expecting it to be folklore. It is not.

Two things to read carefully here. First, that base rate is 90.5%, not 63.7% — the rule can only be applied to a strategy whose metric is far enough from zero for a percentage change to mean anything, which already filters to variants making at least 1% a year. 2,862 of the 5,472 qualify, and inside that group most things survive anyway.

Second, and more interesting: the graded rule beats the yes/no version of the same idea. Asking how much the neighbourhood moves is worth more than twice the lift of asking whether the neighbours are profitable, on identical data.

Is 25% the right number? Close enough that it does not matter:

Anywhere from 15% to 25% behaves the same. What breaks it is tightening: at a 5% tolerance the lift is -0.001 — you are no longer measuring stability, you are selecting on noise.

And most strategies fail the test. The median variant's worst metric move under a ~25% parameter shift is 35.4%, and 61.3% of them move more, proportionally, than the parameter you shifted. The typical strategy is more sensitive than the rule allows, which is the point of having the rule.

One caveat, and it is the same one PBO produced. By family, the lift is +0.105 for RSI mean reversion, +0.041 for MA trend, and -0.002 for Donchian breakout. On breakout the rule does nothing at all. A robustness test only reports something where there is an effect to be robust about.

How bad can the drawdown really get?

Your backtest's worst drawdown is not the strategy's worst drawdown. It is one sample from a distribution, and you got to see exactly one draw.

I took SPY's in-sample RSI winner (RSI(2) < 15, hold 5 days, 470 trades, 70% win rate, profit factor 2.13) and resampled its own trades 10,000 times.

The backtested drawdown sat at the 29th percentile of resampled histories — meaning 29.5% of alternate orderings of the same trades were worse. 2.4% of runs exceeded one and a half times the backtested drawdown; 0.1% exceeded twice it.

Plan your position sizing off the 1-in-20 number, not the backtest number. If -37.6% is more than you can sit through, you do not have a strategy problem. You have a size problem.

How many trades, how many markets, and how much cost?

Three practical thresholds, all measured rather than asserted.

Trades. Every in-sample-profitable variant, bucketed by how many trades produced the result:

The relationship is monotone and it is large: 70.3% at the thin end, 95.6% at the thick end.

Markets. Take an RSI parameter set, count how many of the 8 markets it was profitable on in-sample, then look at what it did out-of-sample:

One market proves nothing. Seven prove something. This is the cheapest robustness test in existence — you already have the code — and it is worth 53 percentage points of survival odds in the scorecard below.

Costs. Frictionless is the repo default, so I charged costs explicitly to see what breaks:

Across the whole RSI grid, going from zero to 20 basis points round trip cut the profitable share from 65.8% to 53.9%. For daily strategies holding several days, costs are a haircut, not an executioner. The faster you trade, the less that sentence applies.

Which robustness checks actually predict anything?

Here is the experiment the rest of the internet has not run.

I defined 11 checks, computed every one of them using only the first 75% of each market's history, and then scored each variant on the untouched final 25%. "Lift" below is the difference in outcome between variants that passed a check and variants that failed it. The base rate across all 5,472 variants: 63.7% were profitable in that final quarter, and 27.8% beat buy-and-hold there.

Nine of the eleven checks earn their keep, and the two best are the two cheapest: hold back a slice and look at it, and check whether the parameters next to yours also work. Neighbours being profitable took survival from 31.9% to 91.7%.

Then there is the last row, which is the most interesting number in the study. Variants that beat buy-and-hold during development were profitable afterwards only 41.2% of the time — well below the 72.5% of variants that never beat the market at all. The same check is the only one that predicts beating the market: those variants beat buy-and-hold again 76.9% of the time versus 8.5% for the rest.

Aggressive strategies win big or die. Boring strategies survive and underperform. The check does not lie; it just answers a different question than the one you thought you asked.

Stack the checks and the pattern is unmistakable:

Robustness testing takes the odds of your strategy still working from 63.7% to 98.0%, and takes the median forward return from 1.7% to 7.8% a year. It takes your odds of beating buy-and-hold from 27.8% to 20.7%.

That trade is worth making. It is just not the trade most people think they are making.

Do any of these checks work outside US equities?

Everything above rests on eight markets, and eight markets is not what it sounds like.

SPY and ES are the same index in two wrappers; they move together 0.98 of the time. Run the maths on the whole set and the eight are worth 2.87 independent bets, at a mean pairwise correlation of 0.533. Their common history starts in 2007, so every number above was fitted inside one macro era. That is a poor foundation for a claim about generality, and "these checks predict survival" is exactly that kind of claim.

So I rebuilt the universe: 53 markets spanning equity index, equity sector, FX, metals, energy, rates, grains, softs and livestock. 17.28 effective bets at mean correlation 0.231 — 6.0× the independent information — and 36,252 backtests under the identical protocol.

The thesis survived. Several of my numbers did not.

The ordering holds. The magnitudes do not. A Sharpe filter that looked worth 37 points of survival is worth 4. Every one of these was measured honestly the first time; they were just measured on a universe that was mostly one market.

What did hold is the part the whole article is built on. Passing all 11 checks still produced 97.2% survival — from 325 variants, 0.9% of the grid. The base rate itself falls from 63.7% to 59.1%, because a random variant does worse once the universe is not mostly US equities. The stack is the finding. The individual checks were never meant to carry it alone, and now there is evidence they cannot.

The check that inverts

Here is the part eight markets could not have told me:

Read the bottom two rows slowly. On FX, a variant whose parameter neighbours are also profitable survives 29.5% of the time against a 45.6% base rate. The check does not merely stop helping. It points the wrong way. Same on rates: 18.1% against 25.3%.

The strongest single filter in this study is a US-equity filter.

I do not have a clean mechanism for it, and I would rather say that than invent one. The pattern is consistent with these markets being genuinely harder — rates has a 25.3% base rate against equity index's 77.4% — so a smooth, profitable-looking parameter neighbourhood in a market that mostly does not reward the strategy may be the signature of a fit rather than an edge. That is a hypothesis, not a result.

Mean reversion's advantage was the universe

The same correction hits the family comparison. PBO — the probability the in-sample champion lands below the out-of-sample median, where higher is worse:

Mean reversion looked like the safe family at roughly a third of trend's overfitting. On a proper universe it is barely distinguishable from breakout. And trend improves — it was worst on a set that was six-eighths US equity, which is precisely where a slow trend filter has least to work with.

The original eight are measurably the flattering subset: median PBO 0.391 against 0.492 for the 45 markets added. And the spread across asset classes — livestock 0.052 to softs 0.837 — is wider than the spread across strategy families.

Where you test matters more than what you test. That is not a caveat on the study. It is the largest single effect in it.

How many strategies survive everything?

Finally, the whole population through a seven-stage gauntlet, in order:

262 of 5,472 — 4.8% — survived all seven. Judge them on Sharpe instead of raw return and 320 survive (5.8%), so the choice of benchmark metric moves the answer by about a fifth.

The single biggest cut is not out-of-sample degradation. It is the buy-and-hold comparison: 2,626 variants were profitable in-sample and only 473 beat simply owning the thing, an 18.0% pass rate. Everything after that stage is comparatively gentle.

And look at what is left: 253 mean-reversion variants, 7 trend, 2 breakout. Long-only trend and breakout systems on these markets over these periods were, overwhelmingly, expensive ways to own less of an asset that went up.

The verdict — and the honest limits

Robustness testing works. It does not work the way it is sold.

Where the standard advice is right. Holding back data and looking at it is the single most predictive thing you can do — +67 points on eight markets, and still +0.293 on 53, the strongest check in both runs. Checking that neighbouring parameters also work takes one line of code and is worth +0.166 across the wide universe. Trade count matters, and the threshold is higher than the usual "30 trades" folklore — the jump from under 30 trades to 300-plus moves survival from 70.3% to 95.6%.

Where it is wrong. Picking the middle of the plateau instead of the peak did nothing measurable here (9 of 24, median difference 0.00%). Costs are a haircut, not a killer, for multi-day holds. Walk-forward measures an edge; it does not create one: median efficiency 0.42, and 6 of 24 wins against buy-and-hold.

Where I was wrong. "My process is robust" is not a thing, and I proved it on myself. The first version of this study reported PBO of 19.2% on mean reversion against 71.4% on trend, and concluded mean reversion was the sturdier family. On 53 markets those become 0.444 and 0.603 — the gap largely closes and trend improves. The eight markets I started with were the flattering subset (median PBO 0.391 versus 0.492 for the 45 added). Every individual check lift above is smaller on a wide universe, several by a factor of three, and on FX and rates the neighbour check inverts outright. The stack still works at 97.2%. The pieces were oversold, by me, on this page.

The part nobody says out loud. A fully robustness-tested strategy is a strategy that will probably still be making money in five years and will probably still be losing to the index. If your goal is to beat buy-and-hold, robustness testing is not the tool — it selects against exactly the aggressive variants that do it. If your goal is to build something you can trade with real money and sleep at night, it is the whole game.

Limits. This study is long only, one unit, no leverage, no shorting, no stops, no position sizing; adding any of those changes the numbers. The ETF series are price-only, so the buy-and-hold benchmark excludes dividends and is understated — a dividend-adjusted benchmark would make these strategies look worse, not better. Three strategy families and 221 mining conditions are a sample of retail practice, not a census of it. The zero-edge control removes serial structure and drift by construction; it is a floor for how good a backtest can look by luck alone, not a claim that markets are random. "Out-of-sample" still means history I happened to hold back, not the future.

The wide rerun has limits of its own, and they run the other way. Most of the energy complex is missing: CL, NG, HO and BZ are additively back-adjusted into negative prices, where percentage returns stop meaning anything, so energy rests on three markets. The bond and multi-asset ETFs are excluded because they arrived dividend-adjusted, and mixing total-return with price series would compare two different quantities. Three asset classes — energy, rates, softs, livestock — carry only three markets each, so their numbers are directional, not precise. And the breadth check could not be carried across literally: "profitable in 5 of 8 markets" is trivially easy across 53, so it is applied as the same fraction instead.

Two independent facts worth putting beside all of this: McLean and Pontiff found that published anomaly returns are 26% lower out-of-sample and 58% lower after publication, in the Journal of Finance. Decay is the normal case even for effects that cleared academic review. Assume yours decays too.

What this means for you

1. Compute your noise floor before you celebrate. Shuffle your own data, remove the drift, re-run your identical optimisation, and record the best result. On three years and a few hundred variants, expect a Sharpe near 1.17 from nothing. Your edge is your result minus that number.
2. Count your trials and say the number out loud. Testing 384 variants on zero-edge data lifts the best Sharpe from 0.00 to 0.36. If you tested 5,000 rules, you are entitled to roughly none of the headline.
3. Hold back a slice and actually look at it. The strongest single check in both runs — +67 points on eight markets, +0.293 on 53. And accept that the split is arbitrary: at 70/30 you keep a different variant than at 50/50 in 62.5% of cases, so treat the choice as a sensitivity, not a truth.
4. Check the neighbours, not the peak — and know what you are trading. Ask whether every parameter set adjacent to yours is profitable. It is worth +0.366 on equity index markets and −0.210 on FX, so on currencies and rates this check is worse than useless. Do not bother relocating to the centre of the plateau — that part did nothing. Better still, grade it: hold the metric's move under 25% for a 25% parameter shift, which more than doubles the lift of the yes/no version on the same data.
5. Demand trades and markets. Under 30 trades, survival is 70.3%. Over 300, it is 95.6%. Profitable on 7 of 8 markets, 74.5% of forward results held up; on zero other markets, 0.0%.
6. Count your markets the way you count your trials. Eight markets bought me 2.87 independent bets and a conclusion about strategy families that did not survive contact with 53. Before you trust a cross-market result, check the correlation matrix — six US equity proxies is one test run six times.
7. Size off the Monte Carlo, not the backtest. The backtested drawdown sat at the 29th percentile of resampled histories. Plan for the 1-in-20 figure — here, -37.6% against a backtested -27.4%.
8. Benchmark against buy-and-hold every single time, and know what you are choosing. Only 473 of 5,472 variants beat it in-sample. If a variant does beat it, that is the one check that predicts beating it again (76.9%) — and the one that predicts a worse chance of just staying profitable (41.2%).

Four of those steps have a study of their own, each one built on this same population of strategies: how to configure a walk-forward (fit length is the only setting that matters), the shortest validation checklist that works (it is two items long), which metric to rank the survivors on (the plainest one), and what Monte Carlo gets wrong (the drawdown, badly).

If you want the worked examples, they are all on this site and they all went through some version of this treatment: the Better-RSI showdown as a test of a widely believed claim, 33,792 backtests of RSI against Stochastic and Williams %R, what survives when you backtest ICT and smart-money concepts, the Larry Williams COT filter, a hedge-fund trend rule that did survive, what happens when an AI generates the strategies, and measuring a market effect before trading it.