January 30, 2015
When designing stock trading systems, it is a good idea to view the problem not only with a vision of what a trading program could or should do but also with an understanding of the environment in which this program will have to operate.
In a software trading program, in which we can make it do whatever we want it to do, we only have logical decisions, calculations, and statements in code to execute.
No feelings, no moods, no hunches, and no trader psychology, even if these in some way could also be programmed in. But this won't stop anyone from just gambling, be it in a discretionary manner or by using a partially or totally automated solution.
A trading program will just do what it is programmed to do. It does not know the difference between good and bad code. It is up to the strategy designer to put in his program whatever he thinks is appropriate or required to make that program profitable over the long term. This may include technical and fundamental data that can be used or not, sentiment indicators, or outside input such as long-term prognosis on individual companies, and minor or major world economic trends. Every strategy designer carries his own information set, to be used or not, in the making of trading decisions.
The not-executed trade makes absolutely no profit and evidently no loss either.
The "I told you so", "I knew", or "you could or should have" also do not generate profits. It's the set of executed trading decisions, on whatever basis they were made or will be made, that matters. It's the sum of all these trading/investment decisions that will determine if, in the end, you win or lose. This does not imply that you can avoid drawdowns; they are an inherent part of the investment/trading process.
The primary objective is not just to make money; the primary objective is first to survive the game and then also prosper. Capital preservation should be the cornerstone of any stock trading/investment strategy. Be it investing or speculating, be it discretionarily, partially, or totally automated, no capital means no trade, no game, and no return.
Why Automate?
You design an automated trading strategy because you view the problem as amenable to logical trading rules and the task too considerable to do it all by hand. You also want to know if the trading strategy you have in mind could have at least survived its past; because if it didn't, this would not be a good omen that it could outperform going forward.
What should transpire in your trading script is your expertise, your knowledge, and understanding of what a stock trading strategy should do. But whatever the code's simplicity or complexity, for that matter, the final outcome will entirely depend on the applied decision process that determines when and how much to buy, sell, or hold.
You are, technically, designing an expert system. No wonder some use quite sophisticated tools to get there. It's not an easy task, as evidenced by all the academic literature available and the huge number of people pushing their particular brand of trading methodology, philosophy, or other.
You do automation because you can not type fast enough or handle more than just a few trades and stocks at a time. As a portfolio grows, it becomes more and more difficult to follow everything. Automation is an answer to a more systematic trading methodology. What you can do by hand can be done by machines at a much faster pace and grander scale.
As an individual or organization, your interest should be for the long term. And here, long-term means 20+ years. The idea is simple as to why: if your trading strategy blows up at any time before the 20+ years are up, you lose. Period. It's not just that you lost the game. You also lost the capital. All that time wasted, all those resources wasted. Maybe most important of all, it was your time wasted with absolutely nothing to show for it except your vaporized account.
Performing a task that might have been doomed from the start because your strategy design was inadequate to properly handle its future long-term trading environment is not a desirable outcome. It's something that should have been corrected before you started to "play" this stock market "game".
Any trading strategy, be it discretionarily driven up to totally automated, needs to be based on something that is there. This from a single phenomenon to any combination of notions that could describe what is happening in the market. A trade needs to be executable. If not, even if it is hardcoded in your trading strategy, it might have absolutely no economic value.
That is why you do long-term backtests. To see how your trading strategy would have behaved in general terms over long-term historical data. It is not enough to simulate on a single stock or instrument. The strategy must work at the portfolio level. This means multiple stocks over a long trading interval. These tests will also give you added knowledge as to general trading behavior and order placements and maybe help uncover new avenues to explore as you see more potential in some of your trading procedures.
If you do not do these backtests, on what grounds could you claim that your particular brand of trading strategy has some value going forward? If you want to sell me that your short-term trading strategy operating on breakouts is worthwhile, bring me a 20+ year test at the portfolio level (say 30 stocks or more), showing that it was at least feasible in the past. If not, I pass. And there are many reasons why.
The Math
The most concise form of expressing the evolution of a portfolio of stocks of any size over any duration can be given using a payoff matrix equation of the form: A(t) = A(0) + Σ(H.*ΔP). The symbol (.*) represents element-wise matrix multiplication. Easily done in Excel. The expression: H.*ΔP resumes all the trading activity of any trading strategy over the life of a portfolio.
I remember seeing this equation for the first time. It took me about 15 minutes to convert all my stuff to adhere to it, most of it was already in that general form, only a few minor modifications were required. It is what is implied by this equation that can be revealing. The output of the payoff matrix Σ(H.*ΔP) is a vector of the cumulated profits and losses generated by each of the stocks in the portfolio over its entire history. What an easy way to express such a complex problem.
An equation representing a price series can be given by P(t) = P(0) + Σ ΔP, an initial price to which is added the cumulative sum of all price variations up to time t. It's not the only representation available. One could consider a stock price as if composed of a series of n consecutive numbers: P(t) = P(0), P(1), ..., P(n) spanning the trading or investment interval.
The holding inventory of about anything could also be written as H(t) = H(0) + Σ ΔH, also saying an initial inventory to which is added the sum of all inventory changes. All the terms used in the payoff matrix could be viewed as starting with an initial value to which is added the sum of all variations over any duration. These series could be fragmented into as many segments as desired or required for whatever reason.
If there is no variation, there is no change. And if the sum of all variations over any duration is zero (ΣΔP = 0), you are left with the same thing: the initial value. It should be evident that the object of the "game" is to achieve: Σ(H.*ΔP) > 0, meaning making an overall profit, no matter the size of the portfolio payoff matrix over whatever the duration.
Both the price series and the inventory series can be chopped in any way one thinks appropriate. It's only when one decides to play the stock market game live that the manner in which one will chop, slice, and dice these two particular data series will become critical. This requires a decision process: how much of this or that, when, and at what price?
That is the problem to be solved, with a profitable outcome: Σ(H.*ΔP) > 0. A portfolio payoff matrix ending in the red (a loss): Σ(H.*ΔP) < 0 should not be part of acceptable solutions, neither should one design the output of a trading strategy to tend to zero: Σ(H.*ΔP) → 0 and this from either side of zero.
One needs to introduce a decision process in order to play the game. But there, there is no perfect equation for that. Yet, it's what will bring a change in inventory! I can write: D(t) = D(0) + Σ ΔD. The equation is right, but it has absolutely no monetary value. It is totally disembodied. However, if you play the game live, it is of the utmost importance. It will be this decision process that will determine what you will trade at what time and in what quantity. Therefore, this decision process needs to be divided into its components; you have time, prices, and quantity of shares bought or sold.
All this comes back full circle to A(t) = A(0) + Σ(H.*ΔP), which resumes it all! The equation could have been written as A(t) = A(0) + Σ(D.*(B.-S).*ΔP) where the decision process is involved in the buying and selling of part or all of the outstanding inventory in each of the selected stocks in the portfolio. The holding inventory matrix H varies according to the result of the trading decision process in action: H = D.*(B.-S).
Any trading strategy that needs to survive over long periods of time (read 20+ years) needs the following properties: feasibility, executability, marketability, and maintainability for the entire duration of the investment period. It's imperative that it be feasible, just as it needs to be logical and respond to just basic common sense.
Things you can't do in real life, even if they are programmed and backtested using your trading software, are not executable and, therefore, worthless. Things like trading a million shares a day when the average daily volume is less than 100k shares are not feasible in real life, but your backtest program could still process price data as if it were. This would result in a doomed and worthless trading strategy and this from the start and by design.
The Payoff Matrix
The construction of a price matrix P can be the closing price of every stock in the DOW 30 over the last 25 years (making this a representative sample of the market as a whole). Every price element in the matrix p(i,j) would be the close of that day (i) for a particular stock (j). Since we are looking at the past, all p(i,j) would be recorded historical data and easily accessible. The Dow 30 price matrix P would grow in size, day by day, by j = 30 new closing prices, and for i = 0 to about 6,500 trading days (195,000 prices in all). The 195,000 entries in the P matrix would be the same closing prices for everyone.
Looking at the P matrix, each column (j) is the actual closing price of a single stock. But the payoff matrix needs the difference in price from entry to exit with all price changes in between ΔP. This difference matrix is easily constructed: Δp(i,j) = p(i,j) – p(i-1,j), subtract the previous price from the current price for all the stocks in the P matrix, and you get ΔP. Since all the data in P was historical data, their daily change or price variations are also part of recorded history. Again, whatever the composition of the historical price matrix P, its price variation matrix ΔP is also the same for everyone. One could view P and ΔP as a portfolio's stock selection which can be applied to any trading strategy H.
So if I want to design an EOD (end-of-day) trading strategy from the above matrix, the price universe would be composed in the above case of 195,000 data points. Both P and ΔP are totally determined with no option to modify the price data since being taken as it is from historical records. If I design two different EOD trading strategies, they both will have to contend with the same closing price data.
If I want to extract some other information from the price series, it will be my doing. Say I want a moving average, well the data is there, so I can do it. Will it be useful? This depends on what I want to do with it. It opens the door to a multitude of data interpretations. From the original data series, which was the same for everyone, I can now transform it into any suitable form in order to feed my trading decision surrogate.
The main course of a trading strategy is H, the holding inventory matrix. Evidently, it will have the same size as P and ΔP: 195,000 data elements. Each h(i,j) gives the state of the inventory level, the quantity held in each of the stocks on that particular day. It is with the decision matrix that we can slice and dice at will the price series and for whatever reasons we see fit: H = D.*(B.-S).
The Buy & Hold is the easiest payoff matrix to build: H.*ΔP where H is a matrix having its first-row h(0,j) equal to the number of initial shares bought in each of the stocks, and all other rows, all 6,499 of them are identical to the first since the initial inventory is held constant for the entire duration.
I prefer writing the Buy & Hold strategy using scalar multiplication as A(t) = A(0) + Σ(hoI.*ΔP) where ho is the initial scalar vector the same size as j, while I is a matrix the same size as ΔP but entirely composed of ones. This produces a matrix where each column has for data element: h(i,j) = hoj,j. Using the 30 Dow stocks case, it can be considered as a Buy & Hold benchmark where the quantities held are constant, and ΔP the difference in the closing prices of the stocks in the DJIA index. The same principles could be applied to the S&P100 or the S&P500 and respectively generate matrices of 650,000 or 3,125,000 data elements over a 25 year period.
Becoming a benchmark, all the characteristics of the index would equally apply to a Buy & Hold strategy. The historical Sharpe ratio, as well as any other ratios you might want to choose, long-term CAGR, drawdowns, including other statistics and metrics.
Another useful tidbit is that you can now compare other trading strategies to this basic benchmark, just as you can compare one trading strategy to another over the same stock selection and duration: is Σ(H(S#n).*ΔP) >? Σ(H(S1).*ΔP) >? Σ(H(B&H).*ΔP) > 0 ? Is your latest trading strategy better than your previous ones and better than a Buy & Hold?
Where is the Money or Where is the Strategy
Designing a trading strategy should have for primary purpose to do better than a Buy & Hold benchmark. Otherwise, why bother? Do a Buy & Hold instead. Simply buy some index funds and be done with it. At least an index fund is designed to generate a long-term average performance that will tend to about the same performance level as a Buy & Hold strategy. Why strive and put the effort to, in the end, underperform an index?
Doing less than the Buy & Hold over the same period of time is surely less than optimum, if not downright useless, considering the small effort required in executing a Buy & Hold. Consider all the time saved not constantly monitoring the market, all that effort just to underperform.
A strategy S1 playing EOD Dow stocks can be written as: A(t) = A(0) + Σ(H(S1).*ΔP(Dow)). From its equation, one can surmise what is required: Σ(H(S1).*ΔP(Dow)) > Σ(hoI.*ΔP(Dow)). Whatever you design as a trading strategy, it must at least beat the Buy & Hold as its final outcome. The closer your trading strategy is to the Buy & Hold, the closer your performance level will tend to the Buy & Hold. If H(S1) → hoI, then the output of the payoff matrix will be about the same or close to it: Σ(H(S1).*ΔP(Dow)) → Σ(hoI.*ΔP(Dow)). Should S1 be the strategy of an index fund, then one should not be surprised if its payoff is about the same as the index.
One could compare multiple trading strategies by ordering them from most to least productive. The differences would be the result of how the trading strategy changed the inventory over time. How they would have sliced and diced the price series for all their trades. Not just one in and there, but all of them, since all trades will be accounted for over the trading interval. The question becomes: is H(S#n) >? H(S3) >? H(S2) >? H(B&H)? Is a particular trading strategy better than the others, including better than the Buy & Hold or an index fund?
From the holding matrix equation: H = D.*(B.-S), the decision surrogate D determines if you buy or sell. It will slice the time series and thereby determine how long you will hold onto bought shares. But there is one ingredient still undefined: q(i,j), the quantity to be bought or sold for each stock over the entire trading interval.
The price of a transaction in an EOD system has already been determined. It is p(i+1,j), which is the next day at the open using a market order, as demonstrated in the DEVX V6 strategy. Market orders at the open for the next day are not the most efficient way to set entry prices. But all you want in the initial stage of designing a trading strategy is to find out if it is worthwhile even if it trades under adverse conditions.
What is left to determine is B.*P and S.*P. I opted to use a buying trade unit (u) as a fixed amount: u = q(i+1,j)p(i+1,j), with u having position sizes like $5k, $10k, $50k, or more. This way I can scale the trading functions using this trade unit, and as a result, the payoff matrix would be scalable to any value I might want, need or consider feasible.
This would result in u/p(i+1,j) = b(i+1,j), the number of shares to be bought on the next day at the opening price, which will be added to the then current inventory: h(i+1,j) = h(i,j) + b(i+1,j). The same would hold when selling shares: h(i+1,j) = h(i,j) - s(i+1,j). A strategy H(S?) can be applied to a portfolio of selected stocks over the entire trading interval. For the Dow 30 stocks this would give: A(t) = A(0) + Σ(H(S?).*ΔP(Dow)), and for the S&P100 100 stocks, one would get: A(t) = A(0) + Σ(H(S?).*ΔP(SP100)).
Having set a buying trade unit (u), I can already answer some investment questions. If I double the trade unit by doubling the initial capital, what will happen? Easy: 2A(t) = 2A(0) + Σ(2H.*ΔP). The position size, the trade unit, would be twice as big producing twice as much profits or losses as before. It's the same as if I traded 200 shares at a time compared to only 100 each trade; I would get twice as much profit or loss. I could scale the portfolio to any size I want as long as the trading strategy remains feasible, executable, marketable, and sustainable.
By analyzing the whole trading strategy as a block, not in the Markowitz sense from period to period, but from start to finish, I can extract a different view of the portfolio management process. It is not anymore the determination of what can come next from period to period, from trade to trade, but more what is the combined and final output of all those trading decisions (all 195,000 or 650,000 decisions depending on the Dow 30 or S&P100 case). A global view of the problem is now required, and it is all resumed in its payoff matrix: Σ(H.*ΔP).
Strategy Design
This provides a framework to develop trading strategies. Using a single price series as an example of a time-ordered sequence of prices, one could take from it a number of time slices (n). At the limit, on an EOD strategy, one could buy or sell at the opening of the next trading day. Thereby having n = 6,499 one day trades. The result would not be difficult to guess. It's equivalent to a Buy & Hold minus the trading expenses. Only slightly underperforming due to frictional costs. I could express such a strategy: Σ(H(S1).*ΔP(Dow)) → Σ(hoI.*ΔP(Dow)), as tending toward the Buy & Hold.
If I took a trade every other week and sold on the next, I would be in the market about ½ the time, or about 3,250 trading days. I could opt to be in the market only one day a week (1,300 days) set a fixed day or select at random one day a week. However, I should not expect to perform at the same level as if at a full market participation.
This raises basic and simple considerations to take care of. One of which is market exposure. A strategy might try splicing price series in such a way as to extract the best parts, those generating a profit. In reality, one might find that averaging out participating in the market half the time might also result in having one's equity working half the time as well. The equation: A(t) = A(0) + 1/2*Σ(H.*ΔP) can express that the equity is only participating over half the trading interval, thereby generating half of the profit or loss even if this splicing is randomly generated.
Another consideration is the position size. Trading smaller positions of, say, half the usual trade size unit (u/2) will also have for equation A(t) = A(0) + 1/2*Σ(H.*ΔP), showing that reducing the bet size also reduces average generated profits. Understandably, q is a profit/loss scaling factor as in qΔp.
The Buy & Hold is a full exposure and full participation trading strategy. As such, historically, it has managed to generate, on average, including reinvested dividends, a little less than a 10% CAGR over long holding periods. As a matter of fact, holding for 20+ years will have a probability of generating a portfolio profit asymptotically approaching 1: E[Σ(hoI.*ΔP(Dow)) > 0] → 1. As Mr. Buffett has said many times before: "... an almost guarantee to win in the long run" simply due to market participation.
If you reduce exposure and participation, then you are operating with some drawbacks. It means that a trading strategy will have to compensate for these shortcomings. Say you use only half your capital and reduce your trade unit by half as well; this would result, on average, in A(t) = A(0) + 1/2*1/2*Σ(H.*ΔP), the equivalent of putting only ¼ of your capital at work. Should you want to beat the Buy & Hold with that, you better have an outstanding trading strategy since it might require something around a 16.27% compounded rate of return just to compensate and reach the same level as the Buy & Hold over a 25-year investment period: (1+r)t = 1/4*(1+g)t, or in numbers: (1+0.10)25 = 1/4*(1+0.1627)25. Much more if you shorten the investment interval. Over a ten-year period, it would require a CAGR of about 26.35% to be equivalent to its Buy & Hold counterpart: (1+0.10)10 = 1/4*(1+0.2635)10. This is a major drawback entirely related to the actual trading strategy H being used.
Not only that but depending on your strategy design, you might find yourself with an added problem: gradual performance degradation. This one is hidden and kind of insidious. It is aimed at the sequential and linear trader who is trying to average out his/her performance level over the year.
Say you have a trading strategy designed to generate, on average, $300k/year from a starting capital of $1M, thereby an expected 30% return on your first year. Even though each year you make $300k, each year it would represent less and less as a percentage of the accumulating account. After just 9 years, your $300k gain would represent less than a 10% increase for the year. After 25 years, your original 30% return on the account is now less than a 4% increase.
Portfolio Return Degradation
(click to enlarge)
It is not the only source of long-term portfolio degradation. For instance, take a look at Fixed Fraction as a trading technique where one sets the same percentage gain or loss by setting a percent profit target and equal percent stop loss. The paper covers the subject and provides easy solutions to this problem as this one relates to the number of trades taken over the life of a portfolio. And since my trading strategies go for a large number of trades, it becomes wise to compensate for the fixed fraction shortcomings by simply over-compensating.
I would say that it is imperative to address all these potential problems (and others) when designing a trading strategy in order to deliberately compensate for these deteriorating performance agents before or as they arise. And it is the responsibility of the strategy designer to do these things.
Not compensating for these problems will not make them go away.
It is not after that you need to compensate for these problems; it is before and right from the start. And that is by designing trading strategies that will take care of these problems or at least account for them and compensate where you can. Otherwise, you will have designed a trading strategy with inherent flaws that will inevitably show up as you spend more and more time in the pursuit of your alpha generation.
Another consideration is the need not only to keep up with the Buy & Hold but to outperform. Not just once in a while but over the long term where it really counts. If your endgame is less than what could have been achieved by a simple Buy & Hold, then your efforts were kind of useless. And here again, the objective remains: Σ(H(S#n).*ΔP(Dow)) > Σ(hoI.*ΔP(Dow)) > 0. You definitely must outperform the Buy & Hold, otherwise... the Buy & Hold was a solution.
All that is being said is that if you deal with the same price series as everybody else, then the differences in payoff matrices, the outcome of the game, will have to come from the trading methodology itself.
... to be continued ...
Created... January 30, 2015, © Guy R. Fleury. All rights reserved.