August 18, 2018

Recently, on a Quantopian thread, the debate touched on over-fitting stock trading strategies. I tend to have a different point of view on this subject than most since I find the very definition of strategy over-fitting as something that describes very little of what is. It resulted in the following post, which is replicated here.

The notion of over-fitting a trading strategy might technically turn out to be quite a misnomer. It might require to do so many compromises that its very notion becomes almost irrelevant.

I see the problem of strategy over-fitting as two separate, sequenced, and interacting problems: first, a stock selection process, and then a trading process.

The selection process is like a stock screener of some kind. A way of selecting for some reason or other some stocks to trade on. In my post on gaming multiple strategies, it was said that selecting 500 stocks out of some 2,000, like in the QTU stock universe, had 5.6486 possible combinations. Every day, every week, or every month you would have such a selection to make from this huge stock universe. This number is so large that even a slight shift in criteria could yield quite a different list of stocks, each with their own idiosyncrasies. You could schedule your trading procedure 30 minutes later and get different answers, while during that time, global activity on the planet would not have changed that much.

On the other hand, the specifications could be so specific that a single list might be the only thing that comes out.

The Selection Process

The outcome of the selection process is what will be fed to your dynamic trading process. It is, therefore, critical to understand which stocks are being picked and for what reasons.

Is it over-fitting if you change the selection process resulting in a new selection that might not be comparable to the previous one? We could all argue that that would be sufficient to get different answers even if none of the trading procedures or program logic were changed an iota.

Over-fitting implies that you have something “fitted” to compare it to (from where you derived the expression over-fitted) or that you have taken enough samples from the available selectable universe to reach the notion of averages. How many samples would that be? One might not be enough!

The over-fit notion might possibly apply if we had averages where we could say: above or below those averages. But there, the problem is that we have no means to locate or identify what the averages might be. The selectable universe is simply too large. You take a million samples (which requires running a million simulations and tallying all the results), and all you have is 106 out of 5.6486, leaving 5.6480 untested scenarios.

Changing the portfolio selection process does not make it over-fit, just different.

It is one combination out of 5.6486. And, a few days later, the whole selection process will need to be done again, and again be a 1 in 5.6486. It would be sufficient to change a single factor to get a different selection.

To grasp the magnitude of 5.6486, take 1010 supercomputers (that's 10 billion machines) times 1016 operations per second times 1014 seconds over some 10 million years of trading. This adds up only to 1040.

NO Exhaustive Search

So yes, nobody has yet to make an exhaustive search, Monte Carlo style, or otherwise. Not even close. One million more of those supercomputers raises the total to 1046. Still, an infinitesimal fraction of the total (it is: 0.0 followed by 439 zeros, and then a 1). I think you see the point. The selections we make could be pretty unique and come with no information about the world they come from (mean, standard deviation, and what have you).

How could we declare something over-fit if there is nothing that we can compare it to or somehow declare as a representative average of this huge set of possible scenarios? Even the indices we use are just 1 in 5.6486 other possible choices. So, we make compromises and simply declare this, whatever something, as an average of what is out there.

Whatever you select as your trading universe will most probably be different from everyone else unless they use the exact same trading program you do on the same data from the same data provider and did those simulations under the same conditions using the same trade times over the same time interval.

Survivorship Bias

The best you could do is bias your selection toward survivors. The financial literature recommends avoiding survivorship biases as much as you can. But that is exactly what you should do: seek out survivors by all means. You want to trade the best that is out there, not just anything that comes out of this huge universe of possibilities.

Your selectable and tradable stocks usually should not be penny stocks or those on the verge of bankruptcy, even though you could specialize in those. Going forward, you will want to play only survivors anyway. Those that have positive future prospects of doing more business. None of the nearly bankrupt candidates should ever be able to stay on your tradable stock list. If it is a bias to want to play the best stock candidates you can find, then so be it, do it. It is just plain common sense. And it is not because you are programming something that you should lose it.

However, the caveat is that everybody else is trying to do the same thing (the millions of them). And guess what? A lot of them navigate toward the same solution. For instance, all the traders that sort their tradable universe by highest market cap should all get the same answer all the time.

There was only one possible past universe to select from, and it is of historical record.

Therefore, whatever their respective trading strategies, they will all be a variation on the same theme which is: how can you take advantage of large caps sorted by size?

Just by putting this observation on the table, the problem changed.

The Problem Changed

Now, we have traders (he/she) playing a variation on a theme. Whatever trading strategy they design, it will deal with an ordered list of the highest caps, the exact same selection set as the other guy. For whatever, when you analyze a lot of strategies, it is what you find. This reduces the problem to a single set in this huge stock universe... which furthermore becomes the same for everyone on this theme. Is this an over-fit? Again, I don't think so. It is just a selection process.

I presume, except for published strategies, that everyone has a different selection process making their tradable stock universe unique. And from day to day, that universe could change. It is not from the stock selection process that we could declare a trading strategy in some over-fitted territory. All you have is a selection, whatever it is. And whatever criteria served in that selection process can also be applied going forward.

The problem is different going forward since you will not know what your selection process will choose. It is not a question of over-fitting, nor is it a notion of random-like selection. Going forward, your strategy will select its stocks, but you will not be able to know which factor dominated, or why next year at this date a particular stock made the list. You cannot even know if it will be there or not. However, some do have much higher probabilities of being there than others. Does your trading strategy suddenly become over-fitted because your stock was there or not there? Again, it is not from the selection process itself that we could speak of over-fitting. For the trader, that stuff did not make any money. Or did it?

A long-term investor would argue that it did and that the selection process was the most important of all. It should be observed that you do not see traders in the Fortune 500, only bag-holders that have held on to their shares for years and years, which, in a way, defends their argumentation.

For the traders, their respective trading strategies are what will differentiate them.

Created... August 18,  2018, © Guy R. Fleury. All rights reserved