The school's files hold 300 Nifty sessions, which is 1.19 years of daily data. Suppose a rule is tried in 100 variants on those sessions and the best of them shows an annual Sharpe ratio of 2.0. It looks like skill. It is also what noise alone is expected to produce: the best of 100 rules with no skill at all is expected to show a Sharpe of about 2.53 on a year of data. The backtest has not found an edge so much as the best accident. This lesson sets out how to count the variants, how to deflate the Sharpe ratio for them, how to measure the chance that the winner is a fluke, and how long a history it takes to say anything. It builds on the lesson on biases and walk forward testing. The numbers are a model, and no rule here is a view.
3By the end you can
- Say why the best of N variants is biased upward and how fast the bias grows.
- Deflate a Sharpe ratio for the number of trials and work out the years of data needed.
- Explain what the probability of backtest overfitting counts and how it is computed.
The idea, plainly
Why does trying more variants make the best one look better?
A Sharpe ratio is an estimate, and an estimate has error. For daily returns that are roughly independent and normal, the error in an annual Sharpe ratio is about one divided by the square root of the years of data. With one year it is about 1.0, so a rule with no skill shows a Sharpe between minus one and plus one about two times in three. Now run many rules and keep the best. The best of several noisy numbers is higher than a typical one, and the more rules, the higher it is. This is multiple testing, and it is the main way an honest backtest goes wrong. Nobody has to cheat. Trying 100 parameter settings by hand is enough.
Read one more paragraph, 51 words
The count is easy to understate. A grid of 4 values for each of five parameters is 1,024 variants. Every time you tweaked an exit rule after a poor result, that was another. Every discarded idea counts as well, because the ideas that failed are the reason the survivor looks good.
How high is the best of N pure noise?
Under the assumptions above, the expected maximum of N independent Sharpe ratios with zero true skill is the standard error times a quantile of the normal distribution that rises slowly with N. The chart gives the values for one year of data. They climb quickly at first and then slowly, which is why the first dozen variants raise the bar more than the next few hundred, and why a thousand variants do not make the bar infinite.
With 100 variants the best no skill rule is expected to show 2.53
Maths: expected maximum and the deflated Sharpe ratio
E[max] ≈ SE × ((1 − γ) × Z(1 − 1/N)
+ γ × Z(1 − 1/(N × e)))
Z is the inverse normal distribution function, γ is Euler's constant (about 0.577) and SE is the standard error of one Sharpe estimate. The deflated Sharpe ratio takes the observed Sharpe, subtracts E[max] for the number of trials N, divides by the standard error and reads the result on the normal distribution. It is the probability that the true Sharpe is above zero once the search is allowed for. The full version also widens the standard error for skew and for fat tails, which the simple version here leaves out. Fat tails and negative skew, typical of option selling, make a Sharpe ratio flatter the strategy even more.
What does deflating do?
It asks how likely the observed Sharpe is under the assumption that the true Sharpe is zero, given how many things were tried. With one variant and a Sharpe of 2.0 on one year, the z score is 2.0 and the probability that the true Sharpe is above zero is about 98%. With 100 variants the same Sharpe has to beat 2.53, the z score becomes −0.53 and the probability falls to about 30%. The number of trials moved a result that looked certain into one that is more likely than not to be luck.
What is the probability of backtest overfitting?
A second tool looks at how the winner behaves on data it was not picked on. Cut the history into an even number of blocks, such as 8. Take every way of choosing half of them as in sample: that is 70 combinations for 8 blocks. In each, pick the best variant on the in sample half and see where it ranks among all variants on the other half. The probability of backtest overfitting is the share of combinations in which the in sample winner lands below the median out of sample. A value near one half means the selection is no better than a lottery. A value near zero says the winner carries over.
The overfitting probability asks how often the winner loses on data it did not choose on
How long a history does it take?
The minimum record length is the years needed before a Sharpe ratio is distinguishable from zero. With a normal approximation and a two sided test at the usual level, it is about the square of the test statistic over the Sharpe: a Sharpe of one needs close to four years, and of a half more than fifteen. A rule that trades weekly has few trades a year, and a few years of data are a handful of independent bets. A discovery in finance should clear a higher bar than the usual one, because so many have been tried across the literature, and one widely quoted study of published factors argued for a test statistic above three. The 1.19 years in the files could support a claim only for a very high Sharpe.
What does India add?
Costs and rules change inside the history. Securities transaction tax on options and futures rose on 1 Apr 2026, lot sizes were revised in Nov 2024 and Oct 2025, and the weekly expiry day moved to Tuesday for contracts expiring from Sep 2025. A backtest that ignores those changes is fitted to a market that no longer exists, which is one more way to overfit. On claims, SEBI's framework lets only registered advisers, research analysts and empanelled algo providers make performance claims, and past performance claims are to be verified by a PaRRVA. A black box algo's provider must be a registered research analyst. A screenshot of a backtest is not a verified record.
Why does a skewed strategy flatter its Sharpe ratio?
The standard error above assumes returns close to normal. Many Indian derivative strategies are not. A rule that sells options earns a small amount most weeks and loses a large amount rarely, so its returns are negatively skewed and fat tailed. On a short history the rare loss may not appear at all, and the Sharpe ratio is high because the sample has not yet met the thing that the strategy is paid to carry. The standard error for such a rule is wider than one over the square root of years, and the correction in the deflated Sharpe ratio widens it for exactly this reason. The lesson on buyers and sellers shows what the tail looks like on the seller's side. A backtest of a seller that stops at a calm year is a measure of the calm.
Can a longer search be rescued by a longer history?
Partly. A longer history shrinks the standard error, so the noise bar for the best of N falls, and a real edge of a given size becomes visible sooner. But the search cost does not vanish, because it is a feature of how many things were tried and not of the length of the data. A thousand variants on twenty years can still produce a winner that was never an edge. The safest design runs the other way: write the rule down before touching the data, fix the count of variants in advance and keep a final stretch of history for one look.
Worked example: from a Sharpe of 2.0 to a probability
Two terms to hold on to from 19.13
Say a trader tries a grid of 4 values for each of five parameters and keeps the best, whose annual Sharpe on one year of daily data is 2.0. The grid is 1,024 variants, which the model rounds to the 1,000 of the chart. The standard error of one Sharpe on a year is 1.0. The best of 1,000 no skill rules is expected to reach 3.26, higher than the observed 2.0, so the z score is −1.26 and the winner is not distinguishable from the best accident.
A Sharpe of one needs close to 3.8 years; half of one needs 15.4
One path of 300 sessions is all the evidence the files hold
Read 2 more paragraphs, 105 words
Take the other route. How many years would a Sharpe of 2.0 need to count as significant on its own?
Taken alone, a Sharpe of 2.0 needs 1.0 years, and the files have 1.19, so the length of the history is not what sinks the claim. The search is. Counting 1,000 variants, the Sharpe falls short of the noise level, and the length test cannot rescue it. A single history is also a single path. The line chart of the Nifty close is the history the trader would have fitted; it shows one route through the year, and the rule has seen no other.
Read the claim yourself
Take any backtest you meet and ask five questions. How many variants were tried, including the ones thrown away? How many years of data stand behind the Sharpe, and what is the number of independent trades? Were the charges and rule changes of each period included? Is there a holdout that was looked at once? And can the strategy's own returns be published for each variant, so the overfitting probability can be computed by someone else? An answer of "I do not know" to the first is the most common, and the most telling.
Try it yourself
Pick a strategy idea you have already tested and list every version of it you ran. Estimate N and look up the expected best Sharpe for that N in the chart. Compare it with your best. If your best is below the noise level, the test has told you nothing. If it is above, work out the years of data needed for that Sharpe, and compare that with the history you used.
The common mistake
The belief: The backtest shows a Sharpe above two, so the strategy has an edge.
A Sharpe is a number with an error, and the best of many is biased upward. The right reading asks how many variants were tried, how long the history is and how the returns are distributed. A Sharpe of two from a single pre-registered rule over many years is strong evidence. The same number from a grid searched on one year is what noise produces. Deflating does not prove an edge is real, either; it says only that the evidence is not enough to tell it from luck. The remedy is a smaller search, a longer history, a holdout and an honest count.
Where this section sits in module 19.13
Key takeaways
- The best of many noisy estimates is biased upward.
- Count every variant tried, including grid points and dropped ideas.
- The expected best no skill Sharpe rises with the log of N.
- A Sharpe of one needs close to four years to stand out.
- PBO is the share of splits where the winner falls below median.
- A backtest screenshot is not a verified record.
Before you trade. Derivatives trading carries a high risk of loss. SEBI's study "Profitability of Individual Traders in the Equity Derivatives Segment (FY25 to FY26)" (August 2026) found that 87.7% of individual traders in equity F&O made net losses in FY26, with aggregate net losses of Rs 91,685 crore. Read the SEBI study