Can AI Build a Profitable Trading Strategy? I Backtested 5 LLM-Generated Rules on SPY to Find Out

Methodology & risk note: Backtested event study on SPY daily OHLCV data, 1993-02-02 to 2026-06-12, totalling 8,398 bars. Layer A: 2,304 strategy variants (RSI, Stochastic, Williams %R sweep); 2,068 eligible (at least 30 IS trades). Layer B: 5 LLM-generated rule-sets implemented verbatim. In-sample: 1993-02-02 to 2016-06-03 (5,878 bars). Out-of-sample: 2016-06-06 to 2026-06-12 (2,520 bars). $35,000 starting capital. Flat-only sequential backtest (no compounding). Frictionless (no commissions or slippage). Next-open entry fills. All results are historical and for educational purposes only. Past performance does not guarantee future results. Not investment advice.

TL;DR — the answer box

• I ran two experiments on SPY. First: a 2,304-variant optimizer sweep to simulate what AI does when it searches for "the best strategy." Second: 5 rules generated verbatim from an AI prompt, tested blind.
• The optimizer found 25 top in-sample picks with a median risk-adjusted score (CAR/MaxDD) of 0.80. Out-of-sample, that score collapsed 85% to 0.12. All 25 stayed profitable — but 0 of 2,068 eligible strategies beat simply buying and holding SPY, which returned +$87,835 out-of-sample.
• Of 5 AI-generated rules: 1 was a clear loser (lost money both periods, 0th Monte Carlo percentile), 1 was statistically indistinguishable from random (14th percentile), 2 were noisy, and 1 was a genuine survivor (+$35,346 out-of-sample, 99th percentile).
• Even the survivor lost to buy-and-hold by more than $52,000.
• The verdict: AI produces impressive backtests, not guaranteed edges. Out-of-sample testing and Monte Carlo validation separate a curve-fit from a real edge — this study shows exactly why that distinction matters.

How I tested this

If you have ever typed "build me a profitable trading strategy for SPY" into an AI and stared at the output wondering whether it actually works — this study is for you.

I ran two separate experiments, each designed to answer a different version of the same question.

Experiment A — the optimizer sweep:

I ran a systematic sweep across 2,304 SPY daily strategy variants. Think of this as what any AI strategy builder does under the hood: try a massive number of combinations, pick the best-looking one in the backtest window, and hand it to you as the answer. I used three momentum oscillators — RSI, Stochastic %K, and Williams %R — swept across multiple lookback lengths, entry conditions, trade directions (long and short), entry styles (mean-reversion and breakout), and time exits.

The data: SPY daily price history from 1993-02-02 to 2026-06-12, totalling 8,398 trading bars. I split this 70/30. In-sample (IS), used for building and selecting strategies: 1993-02-02 to 2016-06-03, covering 5,878 bars. Out-of-sample (OOS), the blind test window the strategies have never seen: 2016-06-06 to 2026-06-12, covering 2,520 bars. The OOS window was locked from the start and never touched during the sweep.

All results are frictionless — no commissions, no slippage, next-open fills, $35,000 starting capital, flat-only (no compounding). Of 2,304 total variants, 2,068 (89.8%) qualified with at least 30 in-sample trades. Anything below 30 trades is too thin to interpret; those 236 variants are excluded from the eligible population. The eligible pool splits across the three indicator families as 532 RSI variants (25.7%), 768 Stochastic (37.1%), and 768 Williams %R (37.1%) — so no single oscillator dominates the population, and the findings below are not an artifact of one indicator family.

Experiment B — 5 AI-generated rules, verbatim:

I gave an AI a single prompt and asked it to generate 5 fully mechanical daily trading strategies for SPY using a fixed indicator set. I then implemented each rule exactly as written — no tweaking, no parameter adjustment — and ran it through the same engine and the same IS/OOS split.

Here is the exact prompt I used:

"You are a systematic trading strategy developer. Build me 5 profitable daily trading strategies for SPY (the S&P 500 ETF). Each strategy must use ONLY these indicators: RSI(14), Stochastic %K(14, smooth=3), Williams %R(14), SMA(100), ATR(20), 1-day prior close-to-close return, or 5-day prior return. For each strategy provide: (1) the exact entry condition with the indicator name and the precise numeric threshold, (2) the entry style — mean-reversion or momentum/breakout, (3) the direction — long or short, (4) a time-based exit in trading days — choose from 5, 10, or 15 bars. Every strategy must be fully mechanical with zero discretion so it can be coded and backtested immediately."

The validation layer — Monte Carlo:

For the 5 AI rules, I also ran a Monte Carlo test. Monte Carlo percentile answers one specific question: if a strategy entered trades on random bars with the same frequency as this rule, what share of those random simulations would do worse? A 99th percentile means only 1 in 100 random-entry runs would beat it — that is a real signal in the data. A 14th percentile means you could have picked trade entries out of a hat and done at least as well 86% of the time. That is noise.

Finding 1: Does the optimizer's single best in-sample pick survive out-of-sample?

The optimizer's top pick, the one any AI strategy builder would hand you as its best result, was a Williams %R(2) long mean-reversion strategy with a 10-bar time exit. In-sample, the numbers looked exceptional.

CAR/MaxDD stands for Compound Annual Return divided by Maximum Drawdown — it measures how much return you earned per dollar of worst-case loss. A score of 1.0 means you gained roughly one dollar of annualized return for every dollar of your worst drawdown. The in-sample score of 1.46 is genuinely strong. The out-of-sample score of 0.10 is a fraction of that.

The win rate held (73.6% IS, 70.6% OOS). The strategy stayed profitable. But the risk-adjusted quality dropped sharply from 1.46 to 0.10 — and the strategy made $23,365 over the OOS window while simply buying and holding SPY made $87,835 over the exact same period. The optimizer's top pick trailed buy-and-hold by more than $64,000.

Finding 2: What happens to the top 25 "AI-selected" strategies when tested on new data?

One bad pick might be a fluke. What if you take the top 25 — the cream of 2,068 eligible strategies, ranked by in-sample risk-adjusted score? This is the shortlist any AI strategy tool would generate.

Median CAR/MaxDD dropped from 0.80 to 0.12 — an 85% collapse in the risk-adjusted score.

All 25 stayed profitable out-of-sample. They are not blown-up, money-losing strategies. But not one of them beat the baseline: buy-and-hold SPY returned $87,835 over the OOS window. The best of the best from a 2,304-variant sweep could not match a strategy with zero intelligence — just hold the index.

An in-sample backtest is like a job interview where the candidate designed the questions themselves. They score close to perfect every time. The out-of-sample test is the actual job. Acing the custom interview and excelling at the real work are two very different things.

Finding 3: Across all 2,068 eligible strategies, does any optimized variant beat buy-and-hold out-of-sample?

Not one.

Exactly half of the 2,068 strategies made money out-of-sample. The other half did not. The median out-of-sample net profit was $0.00. That distribution is centered on zero — a coin flip. And none of them, across the full eligible population, crossed the $87,835 buy-and-hold line.

The optimizer is extremely good at one thing: finding parameters that fit the past data it was shown. It is not finding real edges. It is finding shapes in historical noise that look like edges in the training window and do not persist in the test window.

This does not mean systematic trading is hopeless. The buy-and-hold result itself is a systematic strategy — probably the most robust one in the study. What it shows is that adding oscillator conditions to a basic long-index position is not automatically adding edge. The OOS window (2016–2026) was a strong bull market for SPY, so the structural long drift dominated. Strategies that were in the market some of the time benefited from that drift, but none captured it as efficiently as simply staying in.

The benchmark nobody optimized: a 20-year-old public rule

There is one more comparison in the data, and it might be the most deflating one for the optimizer. Alongside the sweep, I ran a fixed benchmark rule that required zero optimization: the classic RSI(2) long mean-reversion strategy — published by Larry Connors in the mid-2000s, freely available in every trading book since. No parameter search. No AI. One rule, decades old.

Read that first pair of rows twice. The single best pick from a 2,304-variant optimization — the output of the entire search apparatus — made $23,366 out-of-sample. The textbook rule anyone could have copied out of a 2004 book made $22,569 over the same window. The whole optimization enterprise bought roughly $800 of extra out-of-sample profit over the free, public, unoptimized alternative.

That is the cleanest summary of what optimization actually purchased in this study: almost nothing that wasn't already public knowledge. The AI's one validated rule did meaningfully better than both — and still trailed doing nothing by $52,000.

Finding 4: Do the 5 AI-generated rules produce profitable results when tested on new data?

Experiment B was more direct. I implemented each of the five AI-generated rules exactly as specified and ran them through the same engine and the same OOS window.

Four of five were profitable out-of-sample. That sounds solid. Look at what those profits actually mean.

Rule 3 was the clearest failure. It lost money in-sample (-$15,993) and continued losing out-of-sample (-$12,861). Its 42.6% win rate OOS means it was wrong more than half the time. Monte Carlo percentile: 0.0 — the very worst category, where every single randomized simulation outperformed it. The AI's rationale ("Williams %R below -85 combined with a prior-day down move identifies capitulation where a snap-back is probable") was a coherent story. The data said no.

Rule 2 made $586 out-of-sample. That is not a typo. The Monte Carlo percentile was 14.1, meaning 86% of random-entry strategies would have matched or beaten it. This is not a rule. This is luck indistinguishable from noise.

Rule 1 came through OOS with $5,311 and a respectable 70.6% win rate. Its 85.5th Monte Carlo percentile is suggestive — better than most random strategies — but it does not cross the 95th percentile threshold that marks "likely a real edge."

Rule 5 flipped from a small IS loss (-$1,543) to a small OOS profit ($1,807). Its 91.8th Monte Carlo percentile is interesting, but this is a short-side rule on SPY in the middle of a bull market decade. Small OOS profit, structurally unfavorable regime. Thin confidence.

Rule 4 was the only rule that cleared the bar.

Finding 5: How do I know if an AI rule has a real edge — or just got lucky?

The Monte Carlo percentile is the answer. Of 5 AI-generated rules, only one crossed the 95th percentile threshold.

Rule 4 — RSI(14) > 55 AND Stochastic %K(14) > 50 AND Williams %R(14) > -50 AND 5-day prior return > 1% — is the survivor. It is the AI's most complex rule, requiring all three oscillators to confirm momentum plus a recent price move. Out-of-sample net profit: $35,346.17. Monte Carlo percentile: 99.1. This result is not a curve-fit artifact — 99% of random-entry strategies using the same trade frequency performed worse. That is a real edge in the data.

Even Rule 4, the one validated edge out of five AI-generated rules, did not beat buying and holding SPY. Buy-and-hold returned $87,835 in the OOS window. Rule 4 returned $35,346. The gap is more than $52,000.

The OOS window (2016–2026) was one of the strongest SPY bull markets in the last three decades. Any strategy that enters conditionally will spend some fraction of time on the sidelines, and in a straight-line bull market, time on the sidelines costs you. This is a regime caveat: in a sideways or bear market, the picture could reverse. The structural finding — that in-sample performance collapses 85% out-of-sample, and that validation requires both an OOS split and a Monte Carlo check — holds regardless of regime.

You could not tell Rule 4 from the other four by looking at in-sample numbers alone. Rule 4 had the highest in-sample profit, which might make it the first choice — but the AI presented all five rules with equal confidence. Only the out-of-sample result and the Monte Carlo percentile revealed which one was real.

The verdict — and the honest limits

AI generates impressive backtests. It does not generate guaranteed edges.

This is not a condemnation of AI. Rule 4 is a real, validated edge — the AI did find something that works out-of-sample. But that is 1 out of 5 rules, and only after running an OOS blind test and a Monte Carlo screen. The other four rules ranged from clear losers to statistical noise to "probably something but not proven." And none of the five beat buy-and-hold.

The optimization result is starker: across 2,068 eligible strategies, the in-sample ranking told you almost nothing about who would win out-of-sample. The top 25 all stayed profitable, but their risk-adjusted edge collapsed 85% and not one beat the benchmark. Half of the full 2,068 made money OOS — a coin flip.

The trap is that in-sample backtests always look good. That is what they are designed to do. An AI tool that searches through thousands of parameter combinations and surfaces the best-looking one is not finding your future edge. It is finding the best-fitting shape in past data. Your job — the actual work — is validating whether that shape is real.

Honest limits of this study:

The OOS window (2016-06-06 to 2026-06-12) was a historic bull market for SPY. Buy-and-hold being unbeatable is partly a regime result. The 85% edge-collapse finding is the regime-robust result — it captures the gap between in-sample fit and out-of-sample reality regardless of whether the market is bullish, bearish, or sideways.

This study used a single IS/OOS split, not walk-forward or multiple-window validation. A single split can be unlucky at the boundary. Results are fully frictionless — no commissions, no slippage, next-open fills. Real trading costs reduce all OOS profits shown. The study covers SPY daily data only; results may not generalize to other instruments, time frames, or asset classes. Testing on a second index such as QQQ is the logical next step before treating these results as universal. Monte Carlo tests used a fixed-frequency random-entry benchmark, which is a valid but not exhaustive robustness check.

What does this mean for you?

1. Never trust an AI strategy on in-sample data alone. The in-sample result is the application. The out-of-sample result is the job. Always lock away a test window — at least 30% of your data, untouched — before you optimize or evaluate anything.
2. OOS profitability is not enough — run the Monte Carlo. Rule 2 in this study made $586 out-of-sample and was technically profitable. Its 14th Monte Carlo percentile means it was noise. Without that check, you would not know.
3. Beat the benchmark, not just breakeven. The minimum bar for an active strategy is beating buy-and-hold in the OOS window. If the index made $87,835 and your strategy made $5,311, you are taking on real risk and effort for no reward. Set the hurdle before you start.
4. Treat AI-generated rules as hypotheses, not answers. They are useful starting points — well-defined, fully mechanical, immediately testable. But the validation stack (IS/OOS split, Monte Carlo, benchmark comparison) is the work that actually tells you whether the hypothesis is true.
5. Simple rules with large samples beat complex rules with thin samples. Rule 3 — Williams %R below -85 AND prior-day return below -1% — fired so rarely that the OOS sample was too small to measure reliably. Complex multi-condition rules are the textbook overfitting risk. Demand large trade counts before trusting any result.