Alex WhitePreviously I wrote about the dangers of unscientific backtesting, and promised to show how it might be mitigated. Because the ideas are quite abstract, and because it’s where this issue arguably poses the greatest risk, where relevant I will use style premia as an illustration of the basic idea.
The first, and arguably most important mitigant, is to recognize that we do not start knowing nothing[1]. As such, we should need less evidence to convince ourselves of sensible ideas than ridiculous ones. A newspaper article would convince me that a celebrity had died of a heart attack, but I’d need a lot more evidence before I believed someone had been abducted by alien wizards. Similarly, before we believe a strategy works, we should have an intuitive idea of why it should work. By insisting on an economic rationale before believing any backtest, we can effectively cut down the number of trials, and do so substantially.
With style premia, value investing is perhaps the clearest example of an intuitive way of investing; you look for companies that are under-priced (in the most objective terms available), buy them, and wait. Obviously, company accounts are never that simple, but measures such as P/E and EV/EBITDA should give some guide, and buying the unfashionable, boring companies that- effectively- cost less per pound of profit should intuitively give you less downside and less exposure to bubbles.
This example also leads us on to the more general point that, for any idea, there should be more than one way of implementing it. Some ways may be better than others, but if the success of a strategy depends too much on its method of implementation, it is unlikely to be a viable risk premium.
However, while having an economic rationale is necessary, it is not sufficient; after all, it is relatively easy to rationalize and justify almost any strategy, especially when the numbers appear to back it up. A second, simpler mitigating strategy is to insist on out-of-sample performance- does the theory work on a different data set from the one on which it was built? In the simplest case, this means insisting on a long backtest. The good news is that the length of backtest needed to validate a theory increases with the logarithm of the number of trials attempted[2]; so for longer backtests, the number of trials that can sensibly be run without too much risk of overfitting not only increases, it does so exponentially.
Momentum investing- buying recent winners and selling recent losers- is a good example, as there is a 212 year backtest in US stock markets[3]. The following image is taken from “212 Years of Price Momentum” – Geczy, Samonov – August 2013. While there are several decades of negative returns to the strategy, buying winners and selling losers has, historically, been a successful approach.
Of course, we may still be wrong. There is always some second-order survivor bias[4], and inefficiencies can still be arbitraged away. However there has to come a point where, after weighing up all the risks, and factoring in survivorship bias and the potential for poor backtesting, the evidence just becomes convincing. Personally, I find a number of the smart beta arguments fall into this category; as such, they have a role to play as sources of return within a diversified portfolio.
