On Back-testing

Not everyone needs to do back-testing and therefore to some it is not a worthwhile endeavor. There are thousands of ways to play the stock market game and a lot of them do not require back-testing at all.

For instance, I am sure Mr. Buffett does not do any kind of back-testing nor does his staff has any time for it. Mr. Buffett's investing methodology is based on his great understanding of the fundamentals of the markets (reads some 200 annual reports a year), his wide experience and the fact that he views the market in a long term uptrend that has lasted for decades and he is ready to bet, and has bet, that this secular trend will continue for some time to come. For other reasons, there is no need to back-test for momentum and discretionary traders following their hard earned experience on how to play and win.

Back-testing is for those wishing to automate high trade volume over shorter trading intervals. For those wishing to delegate the decision making process to a program that will execute without emotions its required task; and for those wishing to find a long term  tradable and profitable “edge”. All a back-test can say is: if statistically your chosen trading rules would have in all probability produce a profit in the past.

The shorter the trading interval, the higher the random-like nature of price movements, the harder to find trading methods that consistently make money out of the chaos and that can outperform someone like Mr. Buffett over the long run. It is not enough to find a profitable “edge”; it must also beat, as a bear minimum, the Buy & Hold strategy. Outperforming Mr. Buffett long term is the real ball game.

The Challenges

The challenge is considerable, you need to find a trading methodology that can thrive in a harsh almost random-like trading environment with unforeseen and unforeseeable events like earthquakes, tsunamis or nuclear melt-downs; not to mention all the cheating that can go on in the markets at every level: frauds, book cooking, manipulations, mismanagement, and outright incompetence. Add product obsolescence, technological innovations, failed bio-tech trails to the mix and it becomes normal to view the future as a series of uncontrollable random events. And with globalization, all markets get interconnected; so indirectly one would have to predict future prices from a global point of view.

No back-test will be able to predict unforeseeable events. “Black swans” are statistical aberrations (outliers) and as such can not be predicted. However, you can play the market in such a way as to profit from “black swans” went they do occur. That is a totally different problem.

It is not because you have detected a statistically “profitable” scenario over past data from a specific group of stocks that this back-tested “edge” will necessarily be maintained in the future. Far from it, the future with have its own unique signature. This is probably why most trading systems break down when going forward.

I do think, however, that the main reason for trading system breakdowns is due to simplistic and incomplete trading procedure designs (like designing a dip-buyer strategy with no stops). They break down because their trading strategies were flawed from the very beginning, even on positively back-tested data. There is no one size fits all and there is no single strategy that will outperform no matter what is thrown at it. As a minimum, the selected stocks should have a tendency to move in the same direction as the underlying trading philosophy; it makes no sense to buy or accumulate a triple inverse ETF in an up-trending market.

Automated Trading Rules

A back-test is simply a set of automated trading rules and procedures performed on past market data. Its purpose should be mostly to provide a kind of proof of concept. Doing this or that, would these procedures have a positive outcome? If not, why pursue that avenue of research? And if yes, then what were the reasons for the out-performance, will they still prevail in any future scenario, can they be enhanced or scalable, are they still valid over other groups of stocks, over other trading intervals? All legitimate questions that need answers.

There are many kinds of traders who do not need to do back-testing based on their respective trading methodologies (ex: discretionary traders). However, even thou back-testing is not for everyone, it does not mean that back-testing has no value; it is just another tool and like any tool, it can be abused and misused. It can also be a great tool  guiding one's search in the right direction for automated trading profits. Should your back-test show negative outcomes for various groups of stocks with different trading intervals, wouldn't it be wise not to implement those trading rules on real life data? Surprisingly, I like it when I find very bad trading rules as I immediately explore their opposites.

Your Unique Perspective

In a way, your back-test, based on your understanding of how prices move and how you should react (trade) based on these price fluctuations, could be considered as your own game within the game. You are carving out procedures out of the thousands available, or creating your own set of rules based on your vision of this trading universe and as such become a unique, one of a kind player, trader or investor. In final analysis, you are the one pulling the trigger, be it by hand or by program, based on your vision and understanding of your trading environment where you have to set when to trade, determine the risk you are willing to take and quantify within your portfolio constraints the size of your next trade knowing that this last trade will not be the only one.

Back-testing over past market data provides no profits at all; not a penny to be made. However, from back-testing one can design trading rules that will prevail in the future even if the future may come around only once. And maybe there lies the problem: you can do thousands of back-tests on thousands of data sets of varying length using thousands of different trading rules, but in the end, you will have to choose one, the one that you will play your future on, even just for a time, once.


A Special Note:

The old Wealth-Lab 4 web site Chartscript Center has over 1,800 scripts and represent a considerable bank of knowledge on trading methods and concepts, all available free. Trading scripts span 10 years, and all scripts prior to 2005 will not have seen any current data; thereby a special kind of walk-forward and no script has been modified since mid-2008 except for the last 9. A lot can be learned from all these trading scripts.

 


Created on ... September 06, 2011 © Guy R. Fleury. All rights reserved.

Alpha Project