February 7, 2011
This has been discussed many times. It is a recurring theme. It is also the one subject which if not treated with respect can be the main cause of one’s future dismal performance.
I will try to give it a singular perspective.
First, there is nothing wrong with optimizing or over-optimizing for that matter. Optimization should be used to search for trading ideas and concepts; not hard numbers like top performance.
By optimizing, we will always get better and better answers to what we think might be the real problem: finding our best solution to the stock market trading game. However, the whole process of multiple reiterations based on past data has its pitfalls.
In my opinion, any type of optimization must, first and foremost, satisfy integrity. We “cheat” in your backtests, we are only deceiving ourselves. We peek in the future to obtain better results, or we select from hindsight the best-performing stocks for our trading method, and again, the only persons we can hurt are ourselves. It is when we try to sell our over-optimized script to others that our “integrity” should take a hit as now other people will also surely have to pay for our lack of “self-honesty”. So, my first advice is: always develop honest scripts, and, only then can you consider offering them to the public. Before getting blasted by some, please note that I have no scripts that have ever been offered to anyone.
It seems like that any type of optimization we are trying to do might translate into dismal future results when switching to live trading conditions and with real money on the line. For whatever we do more than once, in trying to optimize a trading strategy, we are over-optimizing.
Here are some test conditions which invariably seems to lead to over-optimization and curve-fitting:
- Improving past performance using the same data set for every test.
- Trying to find the best range of parameters for a specific group of stocks over the same investment period.
- Picking from hindsight stocks to include/exclude in our back tests.
- Using past statistical data for evaluating ranges or projections.
- Using statistical data that can only be available from bar.count-1.
- Using optimized past moving indicator values.
- Using too short a testing interval.
- Using too few or hand-picked stocks to include in our tests.
- Using only upswing investment periods (ex.: 80’s and 90’s).
- Ignoring bankrupt, delisted or merged companies.
- Trying to flip 50 000 shares or more on a stock every day.
- Peeking in the future or using data only available from (bar.count-1).
- Relying on past data as if it were really accurate (data glitches).
- Trying to ignore outliers or bad investment periods.
- Trading 5%-10% of our portfolio on every trade as in time these trades will come to represent tens of thousands of shares.
- Flipping stocks where your trades represent more than 10% of the average trading volume for that day.
- Putting all the money on the line on the latest multi-position dip-buyer script developed on market survivors only.
The list of things one can do when over-optimizing a script is a lot longer. All those listed or any combination thereof can be detrimental to our portfolios or our follower’s portfolios. This is why integrity should be priority number one. First for ourselves, and ultimately, should we elect to spread our “trading wisdom”, for others.
It’s like whatever we do to improve performance, because of the iteration process itself, will result in over-optimization. We find where in the past our system did not perform well or that our selection behaved poorly and then hard code our strategy to skip over it or profit from it. Result: better past performance but a very bad idea. The over-optimization process will also give a sense of false confidence that can only result, as a conclusion, in losing more money.
Any weakness in concept, any superficial understanding of what is, or wrongly based market beliefs will produce dismal results when incorporated in our trading strategies. We set any type of unrealistic conditions in our strategies and these will rip us apart in future market conditions. The more we over-optimize, the more we are hammering nails in our portfolio’s coffin.
In this trading business, we rarely hear about the losers; they are just out of the game without even a whimper. But as a group, they represent about 80% of traders. It usually takes less than 18 months to transform a wannabe trader into a dropout with the main reason for quitting the game being a destroyed portfolio (no money, no game, ask any broker). Playing the market is a tough game. We want to be right, we will pay for it. We want the market to do what we want; we will pay for that too. We don’t quite understand the game; we will pay for every lesson we want to learn (as long as we still have cash available to play the game). We don’t believe in stop losses, the market will show us that maybe not only we should but that it is a must just to survive. We want to double down, no problem; the market really likes our money and will invite us to double again and again. Can you say in a single phrase: out of the game, next!
The market has no memory (certainly not of us), it has no mercy and will not discriminate (it will take anybody’s money). The market owes us absolutely nothing. However, it will take all we want to give it; all our time, all our talent, all our savings and more if we let it. It is truly up to us to decide before putting our money on the table what our optimum betting system will be. In my opinion, we can extract from the market what we want or let it take all we have. It is our choice.
The market has changed a lot in recent years. A trader needs a fast computer, adequate trading software and a very good understanding of the game just to stay alive. His competition now comes mostly from machines with sophisticated trading software ready to respond in a microsecond to the changing market environment. The use of high-speed computers connected directly to the floor of the exchange using fast data feeds that enable them to front-run most of the market itself. The competition is fierce, and the reward is huge for the big player that is ready to play and even to front-run his own clients like in the Merrill Lynch case recently. The other side of your trade will try anything to push you to trade at the wrong time or at the wrong price. The simple fact of leaving your stop-loss as an open order at your broker can be devastating as in the May 6th flash crash where all stops on the books were executed down to a -60% decline. Tens of thousands of traders had a very hard lesson to learn that day. And where was the SEC? Well, not on their side for sure.
We optimize because we believe that the game is fair. It’s natural because on our side of the game we can only play fair. We trade on the prices we see. It is not the same for the other side; 60% of the time we are dealing with a machine. Thinking that having Level II is the cure and levels the playing field; well look again. Some 20% of orders on the books are of the iceberg type, over 90% of orders are canceled not to mention the 10 thousand orders that are flashed and removed within seconds after being posted just to occupy a quote server while trading on regional exchanges.
We need to develop scripts that will withstand the future, not the past. Nothing that was should be expected to repeat in the future. It is our responsibility to first protect our capital from whatever will happen in the future and then find ways to improve our performance beyond the Buy & Hold trading strategy. Like I have said before, it is not an easy game. But I do think that we can give ourselves a chance to succeed. And it all starts from simple beliefs: 1) believe in yourself, 2) don’t believe all the scripts you see or test, 3) make your own or modify somebody else’s script to do what you want it to do, 4) make sure that what you do has a real foundation in reality, 5) always keep in mind that the money you make trading the markets is yours, you’ve earned it the hard way.
When we cheat in our backtest, it does not change the future. However, do not expect the future to evolve following those cheats. It is up to us to realistically extract from the game what we honestly have seen from our backtests what we could realistically extract.
My solution to the over-optimization problem was to design randomly generated datasets which would have quasi-random behaviors that closely mimicked real market data including fat tails. By having each test as a unique dataset with absolutely no predictable price movement; I was assured of not falling in the over-optimization criteria listed above: no survivorship bias, no hindsight selections, no favorable selected investment periods, and no forecasting feature on any one criterion. If my script could survive under those conditions; then I thought that it would be well prepared to tackle the future.
Created on ... February 7, 2011, © Guy R. Fleury. All rights reserved.