Optimising a strategy without fooling yourself
Parameter optimisation finds the best settings for the past, which is exactly the problem. How walk-forward folds and a held-out tail show whether the best settings mean anything.
Optimisation is the most dangerous button in any backtesting tool. Give it a moving-average length from 5 to 100 and a stop from 1 to 5 ATR, and it will find the combination that did best on your data. It always finds one. The question is whether that combination learned something about the market or just memorised the noise in one particular stretch of prices.
Evident's optimiser is built to answer that question, not to hide it.
The sweep
You choose which numbers to vary and list the values to try: indicator lengths, rule thresholds, stop, target and trailing-stop distances, position size. Evident runs a backtest for every combination and ranks them. That ranking on its own is the part to distrust. The more combinations you try, the more likely it is that the winner won by luck.
Hold back a tail
Before the sweep starts, Evident sets part of the data aside: by default the last 25% of candles (you can choose 20%, 25% or 30%, or turn it off), which the optimiser never sees. It searches only the earlier part. When it is done, the winning settings are run once on the held-out tail, as long as the tail has at least 30 candles.
That single run is the most honest number in the report. The settings were chosen without looking at those candles, so they behave there roughly the way they would behave on the future. If the in-sample result is excellent and the held-out result is flat or negative, the optimiser has fitted noise.
Walk-forward folds
A single tail is one test. Walk-forward folds add more. Evident cuts the in-sample part into two to five equal consecutive pieces (three by default) and runs the winning settings on each piece separately, reporting return, Sharpe ratio, drawdown and trade count for every fold.
What you want to see is consistency: most folds positive, none catastrophic, trade counts that aren't close to zero. What you often see instead is one great fold carrying the average while the others lose money. That strategy worked in one kind of market and you should know which one before you trade it.
A short checklist
- Vary few things. Two or three parameters over sensible ranges beat seven parameters over wide ones.
- Look at the neighbours. If the best length is 23 but 21 and 25 are terrible, 23 is an accident. A good setting sits on a plateau, and Evident's reading of the result tells you when the winner stands alone.
- Trust the held-out tail over the ranking. Choose from settings that also did reasonably out of sample, even if they weren't first.
- Check the trades. A fold with three trades says nothing.
- Run Monte Carlo on the result. Redrawing the trades a thousand times in random order shows how much of the drawdown, and the final result, could have been luck.
The honesty strip at the top of every Evident report already warns when there is no out-of-sample test, when the out-of-sample result turns negative while the in-sample one is positive, and when there are fewer than 10 trades.
Optimisation describes the past. Nothing on Evident is investment advice.
Evident does not give investment advice. Past performance does not indicate future returns. Leveraged and automated trading carries the risk of losing capital.