Every trader who has been burned by a backtest gets the same advice: test it out of sample. It is good advice. It is also a test you can only take once.
Look at the sealed data, go back and change a rule, look again, and it is not out of sample any more. You have fitted to it. I learned what overfitting costs the expensive way, early on, which is why I treat that one look as the most valuable thing a dataset gives you.
So the useful question is not "should I test out of sample?" It is: which check can I run before the exam that tells me how I will do in it?
The validation checklists I have read don't answer that. Split your data. Walk it forward. Run a Monte Carlo. Check parameter stability. Charge costs. Count the trades. All sensible. All unranked. Not one of them came with a test of whether following it helps.
Here is the version with the ranking attached. One check does almost all the work: 95.4% of the precision gain reached by the four-check list picked on a separate selection panel. Judged by the lists of one to four checks picked there under a 30% recall floor, the other ten are optional, and one of them is worse than not looking.
TL;DR: the answer box
- Out of sample, the base rate is 63.7%. Of 5,472 strategies built and checked on the first 75% of history, 63.7% were still profitable in the sealed final 25%. Every check has to beat that.
- One check lifts it to 91.7%. Every parameter neighbour also profitable. It passes 53.1% of candidates, catches 87.9% of failures and rejects only 23.5% of the strategies that would have worked.
- Each check the selection panel added after that adds under a point. Its two-check list reaches 92.4%, three reach 93.0%, four reach 93.1%, while the pass rate falls from 53.1% to 39.0%.
- The most precise check is also the most wasteful one that beats the base rate. A development Sharpe of at least 0.5 gets 94.5% precision but rejects 75.2% of the strategies that would have survived.
- "Beats buy-and-hold" fails as a standalone check. Precision 41.2%, below the base rate, and none of the short lists picked on the selection panel includes it.
What is out-of-sample testing?
Out-of-sample testing judges a strategy on price history it was never built, tuned or checked on. Here, each market's history splits at 75%: the first 75% is in-sample, where every rule and every check lives, and the final 25% is out of sample, sealed until the end. A strategy "survived" if it was profitable in that sealed quarter.
The catch is the word sealed. The moment a result from the final 25% changes a decision, that history has been used to fit, and the next test on it is in-sample under a different label. Bailey, Borwein, Lopez de Prado and Zhu (2014) put a number on how far that goes wrong: with enough configurations tried, the best in-sample Sharpe carries essentially no information about out-of-sample performance.
That is why the in-sample checks matter. Think of them as mock exams. You can sit a mock as often as you like, because nothing rides on it. The real exam you sit once. This study asks which mock predicts the real grade.
How I tested
Eight markets on daily bars (SPY, QQQ, IWM, DIA and ES, NQ, NG, JY) and three strategy families that produce 684 parameter variants per market between them: RSI mean reversion, moving-average trend, Donchian breakout. 5,472 variants in total. Every trade follows Execution rule v2: the signal is read at the close, the trade buys the next open, and every exit sells at an open. Long only, one unit.
Everything a check can see lives in the first 75%, including a validation slice held back inside it, the last third of the development window. That held-back-slice design is the simplest member of the family Arlot and Celisse's survey of cross-validation procedures reviews, and the survey's warning applies here too: reusing one slice to both choose and score a procedure biases the score upward.
One more split decides which checks go on a list, and in what order. That choice is made on a separate selection panel: every check recomputed on each market's first half of history, with success measured only up to 2018-02-05, the first date any market's test window begins. The chosen lists are then scored on the sealed final 25%, and every list score below is that score. Picking a list on the same quarter that grades it would be the very overfitting you are trying to catch, which is why the selection panel is separated from the grading slice rather than merely sampled differently.
How is a check scored?
Base rate with no checks at all: 63.7% survived. That is the bar. A check meant to pick strategies that stay profitable and cannot beat 63.7% is not a check, it is a superstition.
Two words are used in their statistical sense throughout, because the whole article turns on the difference:
- Precision: of the strategies a check passes, the share that survived. How much you can trust a pass.
- Recall: of the strategies that survived, the share the check let through. How much good stuff you keep.
Any single number can be gamed by ignoring one of them. A check that passes three strategies and gets all three right has 100% precision and is useless.
Which in-sample check best predicts out-of-sample survival?
| Check | Passes | Precision | Recall | Failures caught | Survivors thrown away |
|---|---|---|---|---|---|
| Development Sharpe ≥ 0.5 | 16.7% | 94.5% | 24.8% | 97.5% | 75.2% |
| Every parameter neighbour also profitable | 53.1% | 91.7% | 76.5% | 87.9% | 23.5% |
| Profitable in all three thirds | 41.0% | 91.3% | 58.8% | 90.2% | 41.2% |
| Profitable in the held-back validation slice | 65.5% | 84.2% | 86.6% | 71.6% | 13.4% |
| Still profitable at 5 bps per side | 66.5% | 83.9% | 87.6% | 70.6% | 12.4% |
| Profitable over the whole development sample | 71.1% | 82.2% | 91.8% | 65.1% | 8.2% |
| At least 100 trades | 28.7% | 81.2% | 36.5% | 85.2% | 63.5% |
| Validation CAGR ≥ half the fitted CAGR | 69.5% | 71.9% | 78.5% | 46.3% | 21.5% |
| Same parameters work on 5+ of 8 markets | 87.4% | 70.5% | 96.8% | 28.9% | 3.2% |
| No single trade worth a quarter of the profit | 71.0% | 68.8% | 76.6% | 39.0% | 23.4% |
| Beats buy-and-hold in development | 28.2% | 41.2% | 18.2% | 54.4% | 81.8% |
Every check computed on the first 75% of history, scored on the sealed final 25%. 5,472 strategies, 63.7% base rate.
Read the top two rows together, because they are the same trade in opposite directions.
Sharpe ≥ 0.5 is the most precise check in the study at 94.5%. It is also the most destructive of the checks that beat the base rate: it passes 16.7% of candidates and throws away 75.2% of the strategies that would have worked. Screen these 5,472 variants on it and three of every four that went on to survive die in the screen.
The parameter-neighbour check (is every setting immediately either side of yours also profitable?) gets 91.7% precision while letting through 53.1% of candidates and discarding only 23.5% of eventual survivors. Under three points less precise, and it costs you a third as much.
What does each check cost you?
That is the column most checklists never mention. Every check has a price, paid in strategies you would have been happy to trade.
And at the bottom of the table: beats buy-and-hold in development has a precision of 41.2% against a 63.7% base rate. Strategies that beat the index during development were less likely to still be making money afterwards.
How many checks do you need?
Now the actual question. Not "which check is best" but "which set do I run."
I searched every combination of every size on the selection panel, with one constraint: the combination has to keep at least 30% of the strategies that survived there. Without that floor the search collapses into a veto, and a checklist that approves 2% of candidates is not a filter.
| Checklist | Precision | Passes |
|---|---|---|
| No checks | 63.7% | 100% |
| 1 check: parameter neighbours also profitable | 91.7% | 53.1% |
| 2 checks: + no single trade worth a quarter of the profit | 92.4% | 41.6% |
| 3 checks: + still profitable at 5 bps per side | 93.0% | 40.5% |
| 4 checks: + profitable in the held-back validation slice | 93.1% | 39.0% |
Each list picked on the selection panel, then scored out of sample on the final 25%.
One check gets you from 63.7% to 91.7%. The second adds 0.7 percentage points. The third adds 0.5, the fourth 0.1, and the list that started by passing 53.1% of candidates ends by passing 39.0%.
Did the longer checklists hold up out of sample?
The longer lists looked far better where they were picked. On the selection panel, where the base rate was 65.0%, the two-check list scored 97.4% and the one-check list 91.6%. Three and four checks both scored 97.9% there. On the sealed final quarter the one-check list held at 91.7%, and the two-, three- and four-check lists fell to 92.4%, 93.0% and 93.1%.
The single check kept its score on history it had not seen. Most of the extra checks' lead did not survive the move. Stopping at one is my reading of this final quarter, the same quarter that scores it. On the selection panel alone, every longer list scored higher than the single check.
That is the checklist. One item, and an optional second that buys 0.7 points:
- Are the parameter settings either side of yours also profitable? Not "is there a plateau", just: do the neighbours work? Move every parameter one grid step either side, diagonal combinations included, so your lookback and your threshold change together as well as alone. If any of those settings, or yours, was not profitable over the development window, the strategy fails the check.
- Optional: does any single trade carry more than a quarter of the profit? If one does, drop the strategy. On its own this check is weak, 68.8% precision, tenth of the eleven. Added to the neighbours it lifts precision from 91.7% to 92.4% and cuts the pass rate from 53.1% to 41.6%. Run it only if you can spare the pipeline.
The neighbour check is not free. The study scores it 3 on a 1-to-3 effort scale (my judgement, not a measurement), because it needs the runs either side of your settings. If you optimised on a parameter grid, those runs already exist. The single-trade check scores 1: it reads the trade list you already have.
Why do longer checklists stop helping?
There is a failure mode worth seeing, because it is what happens if you do the obvious thing and keep adding whatever check most improves your hit rate. Greedily selecting the next-best check against the same outcome you are trying to predict is data dredging with extra steps, which is why the ordering below is fixed on the selection panel before anything is graded.
Greedy selection with no recall floor, picked on the selection panel, takes Sharpe ≥ 0.5 first: precision jumps to 94.5%, recall drops to 24.8%, and 16.7% of candidates remain. It takes "at least 100 trades" second: precision 97.5%, and now 6.5% of candidates remain, with recall at 10.0%.
Then the search runs out of road. After two checks the selection panel has no failures left among the strategies still standing, so the other nine cannot be put in any order. Apply them all together and you end at 97.4% precision on 2.1% of candidates. That is the same precision as two checks gave: 97.47% on 356 variants against 97.39% on 115, a gap smaller than one strategy in the smaller pool. The pool is 0.32 times the size. You have not built a validation process. You have built a machine for saying no.
Is a check's rank its value?
No. The second greedy step shows it. "At least 100 trades" is a middling check on its own: 81.2% precision, seventh of the eleven, and it rejects 63.5% of the strategies that would have survived. But conditional on already having a Sharpe of 0.5, it lifts precision from 94.5% to 97.5%.
The short lists show the same thing. "No single trade carries it" is tenth of eleven on its own, and it is still the check paired with the neighbours in the two-check list picked on the selection panel. A check's rank alone is not its value in company. Which is why "here are eleven checks, run the good ones" is not advice, and why the combination search matters more than the individual ranking.
Which checks overlap?
Some of these checks are near-duplicates.
| Pair | Correlation | Of those passing the first, also pass the second |
|---|---|---|
| Profitable overall + survives costs | 0.8977 | 93.5% |
| Survives costs + parameter neighbours | 0.7067 | 78.2% |
| Profitable overall + parameter neighbours | 0.6783 | 74.7% |
| Profitable overall + validation slice profitable | 0.6770 | 86.0% |
Across these 5,472 daily variants, charging 5 bps per side and asking "is it profitable" are nearly the same question: 93.5% of the strategies profitable over the development sample also stay profitable at 5 bps per side. That matches the robustness study's finding that costs at these levels are a haircut, not an executioner. Run both if you like, but do not feel more validated for having done so.
The verdict
Run one check before you look out of sample: parameter neighbours. 63.7% to 91.7%, which is 95.4% of the precision gain the selection panel's four-check list reaches. Every check that panel added was worth under a point on the final quarter.
Use Sharpe to rank, not to screen. It is the most precise single check at 94.5% and it discards 75.2% of your good ideas. Whether that trade is worth taking depends on your own pipeline and is not something this study measured: with a hundred candidates and time for one it looks cheap, with three it looks ruinous. What this study measured is the screen: ranking by Sharpe is my practice, not a result tested here.
Do not screen on beating buy-and-hold. Precision 41.2%, below the base rate. Benchmark against it, always, but as a separate question about whether the strategy is worth trading at all, not as a validation gate.
Stop adding checks when the next one buys under a point of precision. That is my stopping rule for the one-to-four-check lists picked here under the 30% recall floor, and on them it stops at one. The optional second check bought 0.7 points. The study did not test one point as a general threshold.
One check is a filter, not a process. Telling an overfit backtest from a real edge is one of the five things the Algo Trading Masterclass teaches, built on the same testing workflow these studies run on.
The honest limits
Long only, one unit, no leverage, no shorting, no stops, no position sizing, and no commissions or slippage when survival is scored on the final quarter. "Survived" means profitable in the final quarter. It does not mean the strategy beat buy-and-hold, and the robustness guide shows these checks actively select against that.
The effort scores in the underlying data are my judgement about how much work each check is, not a measurement. The 30% recall floor is a choice: a different floor produces a different short list.
The lists and the order are picked on a selection panel whose success window ends before 2018-02-05, with a 65.0% base rate against the final quarter's 63.7%, so a list's score there and its score here come from two different tests. The choice of one check over two is read off the final quarter, so it has no untouched history of its own.
Three strategy families are a sample of retail practice, not a census. And every result is a property of these eight markets over these date ranges.
What this means for you
- Check the neighbours first. 91.7% precision out of sample, and it passed 53.1% of the 5,472 candidates here.
- Add a second check only if you can spare the pipeline. No single trade worth more than a quarter of the profit takes you to 92.4% and cuts the pass rate to 41.6%.
- Stop there. The second check bought 0.7 points, the third 0.5 and the fourth 0.1.
- Rank the survivors on Sharpe rather than screening on it (ranking is my practice, the screen is what was tested), unless you can afford to lose three eventual survivors in four, as it did here.
- Do not screen on beating the index (41.2% precision), but do measure it, because it is the question your checklist cannot answer.
- Skip the duplicate. 93.5% of the strategies profitable over the development sample also survived 5 bps per side here.
- Then sit the real exam once. Out of sample is the grade. The checks are how you show up ready for it.
For what the out-of-sample result can and cannot tell you about the money you need to trade it, see what Monte Carlo gets wrong and the robustness guide that ties the validation steps together.
Mock exams are cheap. The real one you only get once.
Want the next study when it is finished, including the ones that overturn something I believed? That is what the newsletter is for: StatOasis.com/Overfit
Methodology and honesty footer: 5,472 strategy variants (three long-only families, 684 per market) on daily bars for SPY, QQQ, IWM, DIA, ES, NQ, NG and JY. Every check computed on the first 75% of each market's history, survival scored on the sealed final 25%. Short lists and check order picked on a separate selection panel whose success window ends before 2018-02-05. Execution rule v2 (signal at the close, fill at the next open, exits at an open). One unit, no leverage, frictionless survival scoring. Buy-and-hold and seeded, frequency-matched random controls computed on the same bars. Every figure here comes from the study's own results tables. This is research, not investment advice.
