---
title: "Backtest Overfitting: Deflated Sharpe, PBO, Multiple Tests"
description: "How trying many variants inflates a backtest Sharpe: a model of the best of N no skill rules, the deflated Sharpe, PBO and the years of data needed."
source_url: "https://www.rupeecase.com/learn/path-19/module-19-13-backtest-overfitting"
---

[Path 19 13 of 16](/learn/path-19/)Backtest overfitting: deflated Sharpe, PBO and multiple testing[Next 19.14](/learn/path-19/module-19-14-financial-machine-learning)

The school's files hold 300 Nifty sessions, which is 1.19 years of daily data. Suppose a rule is tried in 100 variants on those sessions and the best of them shows an annual Sharpe ratio of 2.0. It looks like skill. It is also what noise alone is expected to produce: the best of 100 rules with no skill at all is expected to show a Sharpe of about 2.53 on a year of data. The backtest has not found an edge so much as the best accident. This lesson sets out how to count the variants, how to deflate the Sharpe ratio for them, how to measure the chance that the winner is a fluke, and how long a history it takes to say anything. It builds on [the lesson on biases](/learn/path-19/module-19-1-backtest-biases) and [walk forward testing](/learn/path-19/module-19-7-walk-forward-testing). The numbers are a model, and no rule here is a view.

3By the end you can

1. Say why the best of N variants is biased upward and how fast the bias grows.
2. Deflate a Sharpe ratio for the number of trials and work out the years of data needed.
3. Explain what the probability of backtest overfitting counts and how it is computed.

## The idea, plainly

### Why does trying more variants make the best one look better?

A Sharpe ratio is an estimate, and an estimate has error. For daily returns that are roughly independent and normal, the error in an annual Sharpe ratio is about one divided by the square root of the years of data. With one year it is about 1.0, so a rule with no skill shows a Sharpe between minus one and plus one about two times in three. Now run many rules and keep the best. The best of several noisy numbers is higher than a typical one, and the more rules, the higher it is. This is multiple testing, and it is the main way an honest backtest goes wrong. Nobody has to cheat. Trying 100 parameter settings by hand is enough.

Read one more paragraph, 51 words
The count is easy to understate. A grid of 4 values for each of five parameters is 1,024 variants. Every time you tweaked an exit rule after a poor result, that was another. Every discarded idea counts as well, because the ideas that failed are the reason the survivor looks good.

### How high is the best of N pure noise?

Under the assumptions above, the expected maximum of N independent Sharpe ratios with zero true skill is the standard error times a quantile of the normal distribution that rises slowly with N. The chart gives the values for one year of data. They climb quickly at first and then slowly, which is why the first dozen variants raise the bar more than the next few hundred, and why a thousand variants do not make the bar infinite.

With 100 variants the best no skill rule is expected to show 2.53

Model, not market data. Expected maximum of N independent normal Sharpe estimates with a standard error of 1.0, from the extreme value approximation. Real variants are correlated, which lowers the effective N. Source: RupeeCase model, no market data. As of 17 Oct 2026.

Maths: expected maximum and the deflated Sharpe ratio
E[max] ≈ SE × ((1 − γ) × Z(1 − 1/N)
+ γ × Z(1 − 1/(N × e)))

Z is the inverse normal distribution function, γ is Euler's constant (about 0.577) and SE is the standard error of one Sharpe estimate. The deflated Sharpe ratio takes the observed Sharpe, subtracts E[max] for the number of trials N, divides by the standard error and reads the result on the normal distribution. It is the probability that the true Sharpe is above zero once the search is allowed for. The full version also widens the standard error for skew and for fat tails, which the simple version here leaves out. Fat tails and negative skew, typical of option selling, make a Sharpe ratio flatter the strategy even more.

### What does deflating do?

It asks how likely the observed Sharpe is under the assumption that the true Sharpe is zero, given how many things were tried. With one variant and a Sharpe of 2.0 on one year, the z score is 2.0 and the probability that the true Sharpe is above zero is about 98%. With 100 variants the same Sharpe has to beat 2.53, the z score becomes −0.53 and the probability falls to about 30%. The number of trials moved a result that looked certain into one that is more likely than not to be luck.

### What is the probability of backtest overfitting?

A second tool looks at how the winner behaves on data it was not picked on. Cut the history into an even number of blocks, such as 8. Take every way of choosing half of them as in sample: that is 70 combinations for 8 blocks. In each, pick the best variant on the in sample half and see where it ranks among all variants on the other half. The probability of backtest overfitting is the share of combinations in which the in sample winner lands below the median out of sample. A value near one half means the selection is no better than a lottery. A value near zero says the winner carries over.

The overfitting probability asks how often the winner loses on data it did not choose on

A method diagram with no market figures. The procedure is combinatorially symmetric cross validation. It needs the returns of every variant tried, not only the winner's. Source: RupeeCase illustration of the probability of backtest overfitting. As of 17 Oct 2026.

### How long a history does it take?

The minimum record length is the years needed before a Sharpe ratio is distinguishable from zero. With a normal approximation and a two sided test at the usual level, it is about the square of the test statistic over the Sharpe: a Sharpe of one needs close to four years, and of a half more than fifteen. A rule that trades weekly has few trades a year, and a few years of data are a handful of independent bets. A discovery in finance should clear a higher bar than the usual one, because so many have been tried across the literature, and one widely quoted study of published factors argued for a test statistic above three. The 1.19 years in the files could support a claim only for a very high Sharpe.

### What does India add?

Costs and rules change inside the history. Securities transaction tax on options and futures rose on 1 Apr 2026, lot sizes were revised in Nov 2024 and Oct 2025, and the weekly expiry day moved to Tuesday for contracts expiring from Sep 2025. A backtest that ignores those changes is fitted to a market that no longer exists, which is one more way to overfit. On claims, SEBI's framework lets only registered advisers, research analysts and empanelled algo providers make performance claims, and past performance claims are to be verified by a PaRRVA. A black box algo's provider must be a registered research analyst. A screenshot of a backtest is not a verified record.

Watch in the terminal: how does a dynamic allocation share out its money? A short tutorial on sample data.

### Why does a skewed strategy flatter its Sharpe ratio?

The standard error above assumes returns close to normal. Many Indian derivative strategies are not. A rule that sells options earns a small amount most weeks and loses a large amount rarely, so its returns are negatively skewed and fat tailed. On a short history the rare loss may not appear at all, and the Sharpe ratio is high because the sample has not yet met the thing that the strategy is paid to carry. The standard error for such a rule is wider than one over the square root of years, and the correction in the deflated Sharpe ratio widens it for exactly this reason. [The lesson on buyers and sellers](/learn/path-14/module-14-3-option-buyer-vs-seller) shows what the tail looks like on the seller's side. A backtest of a seller that stops at a calm year is a measure of the calm.

### Can a longer search be rescued by a longer history?

Partly. A longer history shrinks the standard error, so the noise bar for the best of N falls, and a real edge of a given size becomes visible sooner. But the search cost does not vanish, because it is a feature of how many things were tried and not of the length of the data. A thousand variants on twenty years can still produce a winner that was never an edge. The safest design runs the other way: write the rule down before touching the data, fix the count of variants in advance and keep a final stretch of history for one look.

## Worked example: from a Sharpe of 2.0 to a probability

Two terms to hold on to from 19.13

From the module glossary. Source: Module 19.13, Backtest overfitting: deflated Sharpe, PBO and multiple testing.
Say a trader tries a grid of 4 values for each of five parameters and keeps the best, whose annual Sharpe on one year of daily data is 2.0. The grid is 1,024 variants, which the model rounds to the 1,000 of the chart. The standard error of one Sharpe on a year is 1.0. The best of 1,000 no skill rules is expected to reach 3.26, higher than the observed 2.0, so the z score is −1.26 and the winner is not distinguishable from the best accident.

A Sharpe of one needs close to 3.8 years; half of one needs 15.4

Model, not market data. Years = (1.96 ÷ Sharpe) squared, for normal independent returns. Skew and fat tails lengthen every bar. The files hold 1.19 years. Source: RupeeCase model, no market data. As of 17 Oct 2026.
One path of 300 sessions is all the evidence the files hold

Daily close. A rule fitted to this line has been tested against one history. Resampling and the walk forward folds of the earlier lesson ask how it would fare on others. Source: NSE end of day data, computed by RupeeCase. As of 15 Sep 2026.
Read 2 more paragraphs, 105 words
Take the other route. How many years would a Sharpe of 2.0 need to count as significant on its own?

Taken alone, a Sharpe of 2.0 needs 1.0 years, and the files have 1.19, so the length of the history is not what sinks the claim. The search is. Counting 1,000 variants, the Sharpe falls short of the noise level, and the length test cannot rescue it. A single history is also a single path. The line chart of the Nifty close is the history the trader would have fitted; it shows one route through the year, and the rule has seen no other.

### Read the claim yourself

Take any backtest you meet and ask five questions. How many variants were tried, including the ones thrown away? How many years of data stand behind the Sharpe, and what is the number of independent trades? Were the charges and rule changes of each period included? Is there a holdout that was looked at once? And can the strategy's own returns be published for each variant, so the overfitting probability can be computed by someone else? An answer of "I do not know" to the first is the most common, and the most telling.

### Try it yourself

Pick a strategy idea you have already tested and list every version of it you ran. Estimate N and look up the expected best Sharpe for that N in the chart. Compare it with your best. If your best is below the noise level, the test has told you nothing. If it is above, work out the years of data needed for that Sharpe, and compare that with the history you used.

## The common mistake

The belief: The backtest shows a Sharpe above two, so the strategy has an edge. A Sharpe is a number with an error, and the best of many is biased upward. The right reading asks how many variants were tried, how long the history is and how the returns are distributed. A Sharpe of two from a single pre-registered rule over many years is strong evidence. The same number from a grid searched on one year is what noise produces. Deflating does not prove an edge is real, either; it says only that the evidence is not enough to tell it from luck. The remedy is a smaller search, a longer history, a holdout and an honest count.

Where this section sits in module 19.13

Sections 1 to 4 of 4; the highlighted one is next. Source: Module 19.13, Backtest overfitting: deflated Sharpe, PBO and multiple testing.

## Key takeaways

* The best of many noisy estimates is biased upward.
* Count every variant tried, including grid points and dropped ideas.
* The expected best no skill Sharpe rises with the log of N.
* A Sharpe of one needs close to four years to stand out.
* PBO is the share of splits where the winner falls below median.
* A backtest screenshot is not a verified record.

Watch in the terminal: how bumpy was the ride? A short tutorial on sample data.
**Before you trade.** Derivatives trading carries a high risk of loss. SEBI's study "Profitability of Individual Traders in the Equity Derivatives Segment (FY25 to FY26)" (August 2026) found that 87.7% of individual traders in equity F&O made net losses in FY26, with aggregate net losses of Rs 91,685 crore. [Read the SEBI study](https://www.sebi.gov.in/reports-and-statistics/research/aug-2026/study-profitability-of-individual-traders-in-the-equity-derivatives-segment-fy25-fy26-_103835.html)

What does Backtest overfitting: deflated Sharpe, PBO and multiple testing cover?The best of many noisy rules looks like skill. Count the trials, deflate the Sharpe and ask how long the history must be. It works through The idea, plainly, Worked example: from a Sharpe of 2.0 to a probability, The common mistake, Key takeaways. It is module 19.13 of Path 19, Algo, HFT and Backtesting, about 11 minutes of reading, free.

  AF&O on RupeeCase

## Hedging only, in development
F&O at RupeeCase will do one job: protect a portfolio you already own, by a rule published before it runs. No speculation, no naked option selling, no leveraged bets. It is not available yet.

  [How hedging will work](/hedging)[Sign up for RupeeCase](https://invest.rupeecase.com/signup)

  [No hedgeFalls with the marketWith a putA floor, for a premiumIllustration with made up numbers, not a product and not a recommendation.](/hedging)

QQuick check

## 10 questions, 7 to pass.
Each attempt draws 10 of the 15 questions in this module's bank, and a retake draws a fresh set. A pass gives this module's Certificate of Completion. Your result stays on this device and counts towards the Path 19 certificate.

7 of 10 correct passes module 19.13

Every answer is explained when you check; a retake is always free. Source: RupeeCase Learn quiz rules (70% to pass).

  Passed

###

  Certificate ID

Download PNGPrint or save as PDF[Check the ID](/learn/certificates/verify)
Self paced and unproctored. A record of completing free education by RupeeCase; not a NISM certificate and not endorsed by NISM, NSE, BSE or SEBI. It qualifies no one to deal, advise or publish research.

CCheck yourself

## Path 19 so far.
Your quick check results on this device, stop by stop.

Stop 13 of 16 on Path 19

Algo, HFT and Backtesting: every live stop, this module ringed. Source: RupeeCase Learn, Path 19.

1. [19.1What a backtest is, and five biases that ruin itNot yet](/learn/path-19/module-19-1-backtest-biases)
2. [19.2Strategy metrics: CAGR, Sharpe, Sortino, drawdown and expectancyNot yet](/learn/path-19/module-19-2-strategy-metrics)
3. [19.3HFT, MFT and LFT comparedNot yet](/learn/path-19/module-19-3-hft-mft-lft)
4. [19.4Colocation, latency and fairness: NSE's historyNot yet](/learn/path-19/module-19-4-colocation-and-latency)
5. [19.5Building a compliant retail algoNot yet](/learn/path-19/module-19-5-building-a-compliant-algo)
6. [19.6Backtesting options on Indian dataNot yet](/learn/path-19/module-19-6-backtesting-options)
7. [19.7Walk forward and out of sample testingNot yet](/learn/path-19/module-19-7-walk-forward-testing)
8. [19.8Monte Carlo and bootstrap of trade resultsNot yet](/learn/path-19/module-19-8-monte-carlo-and-bootstrap)
9. [19.9Microstructure models: spreads, informed trading and price impactNot yet](/learn/path-19/module-19-9-microstructure-models)
10. [19.10Market making: inventory and quotingNot yet](/learn/path-19/module-19-10-market-making)
11. [19.11Optimal execution and market impactNot yet](/learn/path-19/module-19-11-optimal-execution)
12. [19.12HFT economics and the latency raceNot yet](/learn/path-19/module-19-12-hft-economics)
13. [19.13Backtest overfitting: deflated Sharpe, PBO and multiple testingNot yet](/learn/path-19/module-19-13-backtest-overfitting#quiz)
14. [19.14Financial machine learning done carefullyNot yet](/learn/path-19/module-19-14-financial-machine-learning)
15. [19.15Expiry day market integrity: a SEBI interim order, FutEq limits and surveillanceNot yet](/learn/path-19/module-19-15-expiry-day-market-integrity)
16. [19.16Capstone: a research note on an Indian derivatives questionNot yet](/learn/path-19/module-19-16-capstone-research-note)
17. [TPath test, 30 questionsNot yet](/learn/path-19/module-19-16-capstone-research-note#pathTestSection)

Path 19 in reading time: 172 minutes over 16 modules

Minutes of reading per module, this one in amber. Source: RupeeCase Learn, Path 19.

    Explore next

## Put the module to work.

* [ToolOption payoff builderLegs on a dated Nifty chain, breakevens and charges](/learn/tools/payoff-builder)
* [Free module 19.1What a backtest is, and five biases that ruin it9 min read](/learn/path-19/module-19-1-backtest-biases)
* [Free module 19.6Backtesting options on Indian data11 min read](/learn/path-19/module-19-6-backtesting-options)
* [Free module 19.7Walk forward and out of sample testing10 min read](/learn/path-19/module-19-7-walk-forward-testing)
* [Free module 19.8Monte Carlo and bootstrap of trade results10 min read](/learn/path-19/module-19-8-monte-carlo-and-bootstrap)
* [Free module 19.14Financial machine learning done carefully12 min read](/learn/path-19/module-19-14-financial-machine-learning)

      Community

## Join my WhatsApp group.

      Scan the code, fillFill one short form, and join my WhatsApp group. Slides, sources, factsheets and the rebalance logs live there, and members join a daily Google Meet with me.

      New to RupeeCase? [Getting started](/getting-started) [See the terminal](/terminal)

        [Sign up for RupeeCase](https://invest.rupeecase.com/signup)
        [Join the WhatsApp group](https://forms.gle/C6HMtM1Z2GfeocQV7)

      [One form, one group](https://forms.gle/C6HMtM1Z2GfeocQV7)Scan with your phone camera

SSources

## Go to the source.
8 sources, NSE the most cited

The publisher of each source listed below. Source: Module 19.13 source list.
1. [SEBI, circular of 4 February 2025, safer participation of retail investors in algorithmic trading](https://www.sebi.gov.in/legal/circulars/feb-2025/safer-participation-of-retail-investors-in-algorithmic-trading_91614.html) SEBI
2. [SEBI, circular of 3 August 2026, enrolment with PaRRVA for verified past performance](https://www.sebi.gov.in/legal/circulars/aug-2026/extension-of-timeline-for-enrolment-with-parrva-as-specified-in-sebi-circular-no-ho-38-14-4-2026-mirsd-pod-i-10557-2026-dated-april-29-2026_103314.html) SEBI
3. [NSE, securities transaction tax on equity derivatives](https://www.nseindia.com/static/products-services/equity-derivatives-securities-transaction-tax) NSE
4. [NSE, derivatives reports and daily end of day files](https://www.nseindia.com/all-reports-derivatives) NSE
5. [NSE, exchange communication circulars](https://www.nseindia.com/resources/exchange-communication-circulars) NSE
6. Bailey, D. and Lopez de Prado, M., The deflated Sharpe ratio: correcting for selection bias, backtest overfitting and non-normality, Journal of Portfolio Management 40(5), 2014 Paper
7. Bailey, D., Borwein, J., Lopez de Prado, M. and Zhu, Q., The probability of backtest overfitting, Journal of Computational Finance 20(4), 2017 Paper
8. Harvey, C., Liu, Y. and Zhu, H., ...and the cross-section of expected returns, Review of Financial Studies 29(1), 2016 Paper

GTerms

## 4 terms from this module.
Multiple testingRunning many tests on the same data. The best result of many is biased upward, so its significance must be judged against the number run.
Deflated Sharpe ratioA Sharpe ratio judged against the expected best of N no skill trials, with allowance for skew and fat tails.
Probability of backtest overfittingThe share of in sample and out of sample splits in which the variant chosen in sample ranks below the median out of sample.
Minimum track record lengthThe years of data needed before a Sharpe ratio is distinguishable from zero at a chosen confidence.
[All 230 terms in the glossary](/learn/glossary)

[Question people askDoes machine learning work for trading?A short answer with a dated chart](/learn/questions/does-machine-learning-work-for-trading)[Question people askWhat are overfitting, look ahead bias and survivorship bias?A short answer with a dated chart](/learn/questions/overfitting-and-biases)[Related module19.1 What a backtest is, and five biases that ruin it9 min, Intermediate](/learn/path-19/module-19-1-backtest-biases)[Related module19.6 Backtesting options on Indian data11 min, Advanced](/learn/path-19/module-19-6-backtesting-options)[Pillar guideDerivatives and Quant115 modules across paths 12 to 19](/learn/derivatives)

Education only, not investment advice, a research report or a recommendation to buy, sell or hold any security. RupeeCase (QC Alpha Technologies Pvt Ltd) is not registered with SEBI as an investment adviser or research analyst. Contracts and prices shown are for illustration only, come from NSE end of day files at least 30 days old, and are not recommendatory. Derivatives trading carries a high risk of loss.