Curve fitting vs optimization | Page 4 | Crypto Craft

Options

Go to
Permalinks
Print Thread

Similar Threads

Optimization, statistics, and avoidance of curve fitting 21 replies

Curve Fitting: The Ultimate Conundrum 17 replies

EA Backtesting and Curve Fitting Concern 2 replies

why curve fitting doesn't work? 23 replies

"curve fitting" or just improving?? 5 replies

Attachments: Curve fitting vs optimization
Exit Attachments

Curve fitting vs optimization

  • #61
  • Edited 3:14pm May 27, 2024 12:34pm | Edited 3:14pm
Quoting yoriz
{quote} Just based on the equity curve of an optimization there is nothing you can say about the forward test. It is not hard to find beautiful equity curves with optimization, but that says nothing about the profitability of your strategy or whether it even has an "edge" at all. The fastest way to know whether your strategy and your optimization method makes sense is by doing a Walk Forward Optimization (WFO) over the last few years. See here for a diagram. To do a WFO...
I appreciate and understand your response and I already am doing WFO and have a successful "system".

My question was a bit different: I am going to post hypothetical.

Let's say after doing WFO, you finally found your "system" and your optimized result over your optimization period is 100% profit on your best optimized inputs.

Do you have an expectation? that trading forward with these best inputs that brought you 100% over the optimized interval will bring bring at least 10% in profit, 15% in profit, etc over the next live trading forward interval?

Which profit over the live forward trading period could be considered a robust system if optimized interval brought 100%(as in example above) in profit?
  • #62
  • Edited 7:40am May 28, 2024 1:59am | Edited 7:40am
Quoting driven18
Which profit over the live forward trading period could be considered a robust system if optimized interval brought 100%(as in example above) in profit?
Lets assume your backtests matches the live execution at your broker in terms of spread, slippage, delays, etc. For strategies that keep their positions open for hours that is usually the case, for scalping strategies that keep positions for a few minutes and tied trailing stops the backtest could differ from the live performance.

In that case, the backtest performance *is* your expected live performance. If you have done an extensive WFO starting from 2010 up to know in tiny steps of 1 week, then you have a very clear view on the typical behavior of your strategy. You have seen what drawdowns you can expect, how long stagnation periods typically are, etc. etc. There is no reason to assume the next week will suddenly be different as long as you keep doing the same optimizations you did during the hundreds of steps in the WFO.

However... if you say your strategy has a 100% win rate, that is a huge red flag to me. The markets are extremely noisy so even getting 55% of your guesses right is already a challenge. This means your 100% strategy must be a Martingale / grid type of strategy. These can not be backtested. You can "optimize" your input parameters to avoid the Black Swan in the testing interval, but you are just fooling yourself. It is just random chance whether you will encounter an unfortunate series of events that makes your strategy dig itself in too deep and break your account. The casino does occasionally roll 10 times red in a row...

So in short, if you use a Martingale / grid type of strategy, optimizing over a short time interval is useless. The Black Swan will be encountered only once in a while. There is no way to "optimize" that away.
  • #63
  • May 28, 2024 7:27am May 28, 2024 7:27am
Quoting yoriz
{quote} Lets assume your backtests matches the live execution at your broker in terms of spread, slippage, delays, etc. For strategies that keep their positions open for hours that is usually the case, for scalping strategies that keep positions for a few minutes and tied trailing stops the backtest could differ from the live performance. In that case, the backtest performance *is* your expected live performance. If you have done an extensive WFO starting from 2010 up to know in tiny steps of 1 week, then you have a very clear view on the typical...
Ok, I give up, 100% was just a hypothetical example.

i had a simple question about expectancy over trading forward interval.

One of us is toooo intellectual or perhaps both of us are. I will vote for both of us. Regards.
  • #64
  • May 28, 2024 7:36am May 28, 2024 7:36am
Quoting HeyYou
I was so obsessed with these things... I remember this was such a stressful period of my life... I miss it LOL I live in different reality now (real estate / passive investing..), which works but it's not as fun... so is there any system that actually works ? live stats plz
I think I know what you mean. Done!
  • #65
  • May 28, 2024 8:26am May 28, 2024 8:26am
Quoting driven18
Ok, I give up. i had a simple question about expectancy over trading forward interval.
Sorry if I misunderstood, but like I wrote: I think you can expect to get the same performance as you are seeing in a WFO. In fact, you can measure the difference in performance between In-Sample and Out-of-Sample during the WFO. You might for example discover that on average 50% annualized return IS gives 25% annualized return OOS.

Again, this assumes backtesting in your platform (MT5) is representative for the performance you see in live trading. You can easily verify by trading your strategy in the live market for a few days and then backtest these same days in your platform. You can then compare the spread, slippage, commission, etc. between your live trades and the backtested trades.

If the backtest is very similar to live trading, then the WFO is representative for the performance you will see in live trading. Similarly the ratio between In-Sample and Out-of-Sample performance can be estimates also from the WFO.

Does that answer your question?
  • #66
  • Edited 9:53am May 28, 2024 8:50am | Edited 9:53am
Quoting yoriz
{quote} Sorry if I misunderstood, but like I wrote: I think you can expect to get the same performance as you are seeing in a WFO. In fact, you can measure the difference in performance between In-Sample and Out-of-Sample during the WFO. You might for example discover that on average 50% annualized return IS gives 25% annualized return OOS. Again, this assumes backtesting in your platform (MT5) is representative for the performance you see in live trading. You can easily verify by trading your strategy in the live market for a few days and then...
I will post my real trading experience.

April 28 - May 20 - was my last optimization period before trading forward live May 20 - May 27. That optimization brought 20% in profit with 5%DD. (First image.)

Trading forward Live, May 20th - May 27 with above optimized inputs brought me 4% in profit for the week. (Second Image)

I was wondering if there is a way to quantify, what consider to be a good result trading forward. Perhaps there is a certain ratio in the science of optimization.

4%(which is 20% of 20%) is good enough, but 5% would be better.
Attached Image(s) (click to enlarge)
Click to Enlarge Name: screenshot.png Size: 12 KB
Click to Enlarge Name: screenshot.png Size: 13 KB
  • #67
  • May 28, 2024 9:35am May 28, 2024 9:35am
Quoting driven18
Perhaps there is a certain ratio in the science of optimization.
I found this article interesting: “All that Glitters Is Not Gold: Comparing Backtest and Out-of-Sample Performance on a Large Cohort of Trading Algorithms”.
They studied a huge amount of strategies. Two remarkable statements they make:

  1. Commonly reported backtest evaluation metrics like the Sharpe ratio offer little value in predicting out of sample performance
  2. Overfitting – the more backtesting a quant has done for a strategy, the larger the discrepancy between backtest and out-of-sample performance

But basically they are saying the ratio differs per strategy. Hence my advice to simply do a WFO and then measure yourself for your specific strategy what the ratio between IS and OOS performance is.

Edit: I noticed this journal is pay-walled. Here is a free digest of that article.

  • #68
  • May 28, 2024 9:45am May 28, 2024 9:45am
Quoting yoriz
{quote} I found this article interesting: “All that Glitters Is Not Gold: Comparing Backtest and Out-of-Sample Performance on a Large Cohort of Trading Algorithms”. They studied a huge amount of strategies. Two remarkable statements they make: Commonly reported backtest evaluation metrics like the Sharpe ratio offer little value in predicting out of sample performance Overfitting – the more backtesting a quant has done for a strategy, the larger the discrepancy between backtest...
That is it, right there...will research further based on this article.
  • #69
  • Last Post: May 28, 2024 3:09pm May 28, 2024 3:09pm
Quoting hanover
{quote} My 2c fwiw: Excellent posts by topherhk88 here and merlin here. Bruce Babcock, a well known futures trader during the 1980s-90s, wrote a lot of good material about optimization. I found this as a starting point; you might be able to find more of his writing using Google....
Quoting remonpilip
{quote} Hey, thanks for sharing babcock's reviews blog i rwad it all took me 3hrs to finish reading it very infornative and changed the way I see optimizing. To shorten the 3hrs read from my POV: To avoid curve fitting you system must be able to trade on at least 10 other markets with a degree of profitability. Start with a core strategy and if it is profitable on 10 other markets then apply a filter to it and test it again on 10 different markets to see if yout filter is over optimized or nah He also notes that if you have x systems and each of...
It feels THIS good to read something that resembles my own trading, so thanks a lot for sharing !!! P.S hanover, your shares are a rare inspiration, thank you dearly for everything.
Attached Image (click to enlarge)
Click to Enlarge Name: eafdfdsfdf.jpg Size: 91 KB
If I don't mention SL and/or TP = longer-term setup possibly in the works!
0 traders viewing now