Random Walk Theory, Tested: Are Markets Random? 5 Tests on 8 Markets Against Their Own Shuffled Returns

TL;DR: the answer box

• Random walk theory fails in all 8 markets on volatility clustering. Clustering ran from +0.137 (10-year note) to +0.289 (QQQ). The widest random band reached only +0.025.
• Direction carries a little memory, and it is small. SPY's day-to-day memory was -0.082, ES -0.075, QQQ -0.057, all outside the random band. Yesterday's move explains 0.7% of the variation in today's at most.
• SPY, ES and QQQ leaned up. SPY closed up on 54.0% of days that moved, against a random band of 49.0% to 51.1%.
• Natural gas showed no short bias in its price. Measured inside each contract, it closed up on 49.1% of days, inside the random band.
• The Canadian dollar is the closest thing to a random walk here. It failed 1 of 5 tests, and once volatility is adjusted for, 0 of 5.

What is random walk theory, and how do you test it?

Random walk theory says each day's price change has nothing to do with the days before it. The formula is one line: price today = price yesterday + a random step. If that holds, a chart's past tells you nothing about its next move. The idea goes back to Louis Bachelier's thesis, and Burton Malkiel's book made the phrase famous.

Traders hate it. "Markets are never random" is one of the most repeated lines in trading. So are its cousins: the S&P 500 has a long bias, natural gas a short one, and USD/CAD has none.

Here is the problem with that debate. Nobody defines "random." So I did, with a ruler built from each market's own data.

The ruler: the market, shuffled. Take every daily return a market produced. Keep them all, and deal them back out in a random order, like a deck of cards. The shuffle has the same crash days, the same wild days and the same average. Only the order is gone.

Do that 1,000 times. If the real market sits inside what most shuffles produce, it behaves like a random walk on that test. I call that the random band: the middle 95% of the shuffles. If the market sits outside the band, the order of its days matters.

For drift the ruler changes, because a shuffle cannot change an average. There I kept every day's size and let a coin pick its sign, 1,000 times. Same moves, no direction.

How I tested it

The five tests, in plain words.

1. Drift. Does the market lean one way? The score is the mean daily return. I also show the share of days that moved that closed up, where a coin gives 50%.
2. Memory. Does today's move tell you about tomorrow's? The measure is autocorrelation: how much one day's return lines up with the next. 0 means no memory. Below 0 means a move tends to be partly undone the next day.
3. Streaks. How long do runs of up days or down days last? Shorter than the shuffles means direction flips more often than chance.
4. 20-day ratio. Do daily moves add up over 20 days the way a random walk says? The variance ratio compares the spread of 20-day moves with 20 single days. A random walk gives 1. Below 1, moves get partly undone. Lo and MacKinlay used this test to reject the random walk for US stocks.
5. Clustering. Does the size of today's move tell you about the size of tomorrow's? 0 means a big day says nothing about the next one.

The markets. Eight: SPY and ES (the S&P 500 as an ETF and as a future), QQQ (Nasdaq-100), gold, crude oil, natural gas, the 10-year Treasury note and the Canadian dollar. Each runs over its full history, from 1975-01-03 for gold to 2000-05-24 for the Canadian dollar, all ending in August 2026. Gold alone has 12,988 daily returns.

Futures, measured properly. A continuous futures chart glues contracts together, and the glue shows up as fake moves. For ES, gold, crude, natural gas and the 10-year note, each day's return is measured inside one contract. The Canadian dollar comes from a back-adjusted series, which keeps each return's sign and order.

No trades. This is measurement, not a trading system. There are no entries or exits, so Execution rule v2 does not apply. The shuffles are the baseline.

Are markets random? The scoreboard

No. 22 of the 40 test results landed outside the random band, where luck alone would put about 2.

To keep myself honest, each test has one fixed headline number, set in the code before I read the answer. That gives 40 results. Here they are, market by market.

Three groups fall out. The stock indexes break almost every rule: SPY fails 5 of 5, ES and QQQ 4 of 5. The commodities and the 10-year note break one to three. The Canadian dollar breaks one.

Now look at the tests instead of the markets.

One test failed everywhere. That is the whole answer, so start there.

Is volatility random? The test every market failed

No. In all eight markets tested, big days tended to follow big days.

Here is SPY against its 1,000 shuffles. The blue pile is what random order produces. The orange line is the real market.

The shuffles ranged from -0.031 to +0.039. SPY scored +0.273, about 13 times the top of its random band. Not one shuffle came close.

Every other market looks the same.

Clustering ran from +0.137 to +0.289. The widest random band reached only +0.025. Benoit Mandelbrot described this in the 1960s as large changes followed by large changes, of either sign. Every market here still does it.

It held across time too. Split by decade, clustering sat outside the random band in 31 of 33 market-decades.

Here is what it looks like on a chart.

From 2020-01-02 to 2020-02-21, SPY had 35 trading days and 0 moves of 3% or more. From 2020-02-24 to 2020-04-30 it had 48 trading days and 22 of them, including -10.9% on 2020-03-16. Across SPY's 1,000 shuffles of its full history, clustering ranged from -0.031 to +0.039. The real market scored +0.273, and here it stacked its big days in a few weeks.

A random walk is a well-shuffled deck. These eight markets dealt the face cards in clumps.

Does today's move predict tomorrow's?

A little. In the stock indexes, a move tended to be partly undone the next day.

SPY's memory was -0.082. Not one of the 1,000 shuffles got that far from zero. ES gave -0.075 and QQQ -0.057, both outside their bands too.

So the memory is real. Is it big? Square it, and you get the share of today's move that yesterday explains.

Yesterday's return accounts for 0.7% of the variation in today's. The other 99.3% is not explained by it.

The sign has been steady, though.

All 10 stock index decades were below zero, and 8 of them sat outside their own random band. SPY ran -0.057 in the 1990s, -0.072 in the 2000s, -0.043 in the 2010s and -0.149 in the 2020s.

Outside the stock indexes, memory comes and goes. Gold, crude and the Canadian dollar were inside the band. Natural gas was just outside at -0.024, the 10-year note at +0.019. Across 8 markets and 5 daily lags, 19 of 40 memory readings landed outside the band. With that many readings and effects this small, I would not trade any one of them on this evidence alone.

Do moves add up the way a random walk says?

Not in stock indexes. Their 20-day moves were smaller than twenty independent days would make them.

SPY's 20-day ratio was 0.71, against a random band of 0.89 to 1.12. ES gave 0.72 and QQQ 0.77. In plain terms, a typical 20-day SPY move was 84% of the size a random walk would give. Over 20 days, part of the move got handed back.

The other five stayed near 1, from 0.91 to 1.02.

Are stock indexes the only market with a bias?

No. Five of the eight closed up more often than random signs produce. Natural gas did not lean down.

The S&P 500's long bias shows in SPY's price since 1993. SPY closed up on 54.0% of days that moved, against a random band of 49.0% to 51.1%. Its mean daily return of 0.041% is 10.2% a year when multiplied by 252 trading days, before dividends. That is a plain scaling, not a compounded return.

No short bias showed up in natural gas's traded price. Measured inside each contract, natural gas closed up on 49.1% of days and its mean daily move was -0.013%. Both sit inside the random band. A back-adjusted natural gas chart falls hard, but that fall includes the cost of rolling from one contract to the next. My coin flip study splits the roll from the price.

USD/CAD's neutrality holds. On the headline statistics, the Canadian dollar showed no drift, no day-to-day memory and no streaks, and a 20-day ratio of 0.91, inside the band. It failed 1 of 5: clustering.

Two odd cases. Crude's 51.7% up days is outside the band, but its average return is not. Gold's streaks averaged 1.88 days against 2.00 in the shuffles, so its direction flipped more often than chance.

How much can one day change the answer?

In crude oil, one day flips the memory test.

On 2020-04-20 the expiring crude contract settled below zero. As a percentage that is a -306.0% day. I took it out and ran crude both ways.

With that day left out, crude's memory is -0.014, inside the band. With it in, +0.092, outside. One return out of 10,907 decided the result. When one day dominates a series, report the result with and without it.

Is the memory just volatility clustering in disguise?

No. Adjust for calm and wild stretches, and stock index memory roughly halves. But it stays outside the random band.

A shuffle mixes calm and wild stretches together, so a memory reading could partly reflect clustering. To check, I reran every test with each day's return divided by the volatility of the 20 days before it. Only days that had already closed went in.

SPY's memory moves from -0.082 to -0.036, and its 20-day ratio rises from 0.71 to 0.82. Both stay outside the band. Natural gas's memory disappears. Something new shows up in commodities: adjusted, gold, crude and natural gas have 20-day ratios of 1.10, 1.12 and 1.17, outside the band on the trending side. In those terms, commodity moves built over a month.

The Canadian dollar, adjusted, failed 0 of 5. Of the eight, it looks most like a random walk.

Where is the edge in trading, then?

The most consistent departure from random was in the size of moves, in all eight markets. Direction effects were smaller and differed from market to market.

An edge in trading starts as a measurable way a market departs from random. Whether a rule can trade it after costs is a separate test, and this study did not run one. On this evidence:

• Clustering, 8 of 8 markets. The only departure found everywhere.
• Direction memory, 5 of 8 markets, and small. Stock indexes partly undo daily moves, and yesterday explains 0.7% of the variation in today's return.
• Drift, mainly the stock indexes and the 10-year note. SPY closed up on 54.0% of days that moved.

Random walk theory versus the efficient market hypothesis is a different argument. The efficient market hypothesis says prices already hold what is known. The random walk is the narrower claim that price changes are independent. A market can break the second, as all eight do on clustering, without handing anyone a free directional trade.

I learned what an untested pattern costs the hard way: I lost roughly $270K early on, over years of buying overfit systems and trading with too much leverage. A shuffle test asks the question every pattern should answer first: does it depend on the order of the data at all?

The verdict, and the honest limits

What the common view gets right. None of these eight markets behaved like a random walk. All eight failed at least one test, and 22 of 40 results fell outside the random band. SPY's price series showed the S&P 500's long bias. USD/CAD, through the Canadian dollar future, is the most random market here.

What it gets wrong. "Never random" overstates direction. The largest day-to-day memory explains 0.7% of the variation in the next day's return. Natural gas showed no downward drift in its traded price. And the most consistent departure in these tests, clustering, says nothing about which way the next move goes.

The limits, in plain words.

• Eight markets, daily bars, each over its own history. Intraday bars may behave differently.
• A result outside the band says the order matters. It does not say the effect is big enough to trade after costs.
• With many readings, a few land outside the band by luck. That is why the scoreboard uses one fixed number per test.
• SPY and QQQ are price only, so their drift leaves out dividends.
• Crude has one day removed, 2020-04-20, and the result with it kept is reported above.

What this means for you

1. Shuffle a pattern before you call it an edge. Run your measurement on the same returns in random order, many times. For drift, flip the signs instead, because a shuffle keeps the average. If the real number sits inside the shuffles, it has not shown you an edge yet.
2. Volatility clustered in every market's history. It was the one departure from random in all 8 markets' daily returns. Whether volatility-based sizing helps your strategy is a separate test, and this study did not run it.
3. On stock indexes, one-day moves tended to be partly undone. Memory was below zero in every decade tested for SPY, ES and QQQ.
4. Know your market's lean. Stock indexes leaned up. The Canadian dollar and natural gas's traded price did not lean outside the random band.
5. Check your most extreme day. One crude oil day turned memory from -0.014 into +0.092. Report the result with and without it.

Finding a market's edge before writing a single rule is where the Algo Trading Masterclass starts. The market regimes study is the natural next read.

None of these eight markets behaved like a random walk. What all eight shared was how big the moves were, not which way they went.

Reference tables

Table 1. Drift: share of up days and average daily return

Bold = outside the random band (1,000 random sign flips). p is the share of the 1,000 sign flips, plus the market itself, at least as extreme as the market.

Table 2. Memory and the 20-day ratio, raw and adjusted for volatility

Bold = outside the middle 95% of 1,000 shuffles of the same (raw or adjusted) returns. Adjusted = each return divided by the volatility of the 20 days before it.

Everything here is research, not investment advice. It is a measurement of historical data with the assumptions stated above. It is not a recommendation to buy or sell anything, and past results do not predict future ones.

The next study goes to the Overfit newsletter first. StatOasis.com/Overfit