Why most backtests lie — and six checks that catch it
A rising equity curve is the easiest thing in trading to produce and the least informative. Six checks that separate a real edge from a lucky fit, and how Evident runs them on every strategy.
A rising equity curve is the easiest thing in trading to produce. Try enough parameter combinations on enough data and one of them will look brilliant. That curve tells you almost nothing about next month. Here are six checks that catch the most common ways a backtest lies.
1. Too few trades
A strategy with 9 trades and a 78% win rate has not shown anything: a few lucky trades explain it. Below roughly 30 trades, treat any result as an anecdote. Evident shows the trade count next to every result and lowers its reliability label when it is small.
2. Rules that never fire, or always fire
AI-written strategies often contain a condition that is true on every candle (so it filters nothing) or one that is never true (so a whole branch of the strategy is dead). The equity curve does not show this. Evident audits every rule against the data and tells you, for example, that "RSI below 5" never happened in the tested period.
3. Costs
Many backtests ignore fees and slippage. A strategy that trades often can turn from profit to loss once you pay 0.1% per side. Evident includes fees and slippage in every backtest; you can change them to match your exchange.
4. Out-of-sample behaviour
If you tune a strategy on the whole history, you have measured how well you can fit the past. The honest question is what happens on data the strategy has not seen. Evident holds out the most recent part of the data and reports how the strategy behaved there separately. A strategy that collapses out of sample was fitted, not discovered.
5. Luck in the order of trades
The same set of trades in a different order produces a different drawdown. Monte Carlo reshuffles the trades many times and shows the range: not "the drawdown was 12%", but "in most orderings it was between 9% and 24%". If the bad end of the range would end your account, the strategy is too risky whatever its average.
6. Too many ideas tested
Test 40 indicators at the usual 5% threshold and about two will look significant by chance alone. That is not a hypothetical: in our open indicator research, 5 of 40 indicator–market combinations passed our evidence bar, and we say explicitly that about 2 would be expected from chance. The more ideas you try, the more skeptical each "discovery" deserves.
What a good result looks like
Not a perfect curve. A good result has enough trades, survives costs, behaves similarly on unseen data, has a tolerable worst case in Monte Carlo and uses rules that actually fire. When those agree, you have something worth paper trading. When they do not, the most valuable thing a backtest can tell you is "don't".
In Evident every strategy comes with these checks and a plain reliability label. The AI writes the rules; the engine measures them.
Nothing here is investment advice. A backtest describes the past only.
Evident does not give investment advice. Past performance does not indicate future returns. Leveraged and automated trading carries the risk of losing capital.