When you create a trading system, or algorithm, for your trading rules the first item on the agenda is backtesting. You need to backtest to see if your ideas are indeed profitable. If they have not been profitable over time why would you think they will be profitable in the future?
When you create a trading system, or algorithm, for your trading rules the first item on the agenda is backtesting to see if your ideas are indeed profitable
Backtesting is the process of feeding historical data to your trading strategy. You do this to see how it would have performed with the expectation the past results we be a predictor of future results.
Testing an algorithm over historical data is an essential part of the strategy development process. Estimating the performance of the algorithm allows the trader to refine the strategy and modify parameters before trying it with live data. Backtesting is a complicated task requiring survivorship bias free data and special software to run the algorithms. QiT uses Amibroker to run its algorithms and our data provider is Norgate Premium Data.
Backtesting is a simulation of how a trader would have done if he/she responded to market data according to the model, the rules of your trading idea, over time. How would a trader buy, sell, short, cover a symbol and when? Backtesting is a way of testing the signals given by a trading system in order to see whether it would have been profitable in the past.
As with any simulation, the more realistic the backtest, the more useful it is. Obviously, the goal of any trader is to maximize return after slippage and transaction costs (while avoiding unnecessary risk) and these costs should, of course, be included in the test.
Overfitting data (curve fitting)
The problems with backtesting though are difficult to overcome but you need to in order to have faith in any algorithm you’ve written and ultimately use in your trading. One of the biggest problems is something called “overfitting.” This happens when you continually tweak and add parameters in order to increase the overall profitability. Do this to excess and you are likely to end up with a brilliant system for the period of time under scrutiny, but one that fails miserably as market conditions change. The best systems tend to be fairly simple.
Overfitting happens when you continually tweak and add parameters in order to increase the overall profitabilit
Problems with Overfitting
A piece of logic too closely fit your data is overfitting, a modeling error. This generally happens when a model gets overly complex to explain idiosyncrasies in the data under study. In reality, the data being studied often has some degree of error or random noise within it. Thus attempting to make the model conform too closely to this noise can infect the model with substantial errors and reduce its predictive power.
Here’s an example. We created a set of rules and backtested to 2007 but backtesting revealed a large drawdown in August 2011, when the US debt was downgraded. We went back and changed the model to lessen the pain of that month with more parameters and more stringent inputs but unfortunately, it changed the model and the whole algorithm fell apart. That month was part of reality and no model is going to rule out black swan events nor should it.
How do you overcome overfitting? You use In-Sample and Out of Sample data and do Walk Forwards.
Look-Ahead Bias
Look-Ahead Bias happens when backtesting results are distorted because trading decisions are based on information that was not yet available. This generally happens because extra data is accidentally included and we simply don’t realize it. These errors are often very slight, which is why they are so easily overlooked. They can, however, have a dramatic impact on the overall results.
An example of Look-Ahead Bias would be creating a system that bought daily lows and sold daily highs. Sounds good, doesn’t it? I would guess that this system would produce some awesome results. Problem is, in reality, you don’t know the daily high or low until the end of the day, whereas when backtesting you do know.
Believe it or not there are a lot of trading systems that use this type of logic.
Survivorship Bias
Survivorship Bias addresses the concept that many data sets do not include assets that are no longer listed. The result is that these data sets only include the assets that have survived to this point. This makes them skewed to the positive side because they do not include assets that have not survived.
Here’s a scenario. You want to test how well your strategy does with the S&P 500 over the last 10 years. Would you not need a database that was able to identify which stocks were actually in the S&P 500 10 years ago? Would it be of any value if you tested only the stocks that were are in the S&P 500 today? I think not.
For more information on Survivorship Bias and why QiT is adamant about not using data with it click here
For examples of stocks that did not survive, look at the trades-data and every stock that has a date beside the symbol no longer exits. The date is the day it became delisted. You’ll be amazed as to how many there are.
Tolerance Bias
Every trader thinks that they will be able to withstand more drawdown than they will actually be able to tolerate. Tolerance Bias is a trader overestimating his ability to tolerate drawdowns. This bias is completely psychological.
Let’s take an example. You’re looking at a strategy with 20% drawdown and a 35% trade drawdown. You start trading and it immediately starts into a drawdown. You go back and check the algorithms metrics and you see over the last 7 years the max duration drawdown is thirteen months. Can you stick with this system for thirteen months underwater? Probably not. You need to take these numbers into account when deciding on a mechanical system. Well, actually you need to take these numbers into account when trading any system.
Your Challenge
While it is easy to look at the overall return of the system and agree that it was profitable, it is naive to think that it would not be a challenge to stick to the system at its low point. If you find a trading system that is profitable, you need to trade it long term. While this can be challenging, you must find a way to stick to the plan, even if it seems boring or not worth your time in the beginning. In my opinion, trading should be boring if you are going to succeed.
We’re the Plan in “Plan your Trade and Trade your Plan” TraderJanie
Check our out the Performance Matrix then grab your 30-day free trial.
