 |
|
Software For Programmed Trading Of Equities Over The Internet: |
|
The Tsinvest Program - Optimal Concurrent Investments In Equities |
||
Home | Installation | Usage | FAQs | Utilities | Architecture | QA | Tests | Links | Mailing List | License | Author | Download | Thanks
 |
 |
 |
 |
 |
 |
 |
 |
 |
 |
 |
 |
 |
 |
|
tsinvest [-a 0|1|2] [-C] [-c] [-D D] [-d 1|2|3|4|5|6] [-I I] [-i] [-j] [-M M] [-m m] [-o o] [-P] [-p] [-q q] [-r] [-s] [-t] [-u] [-v] [filename] DESCRIPTIONTsinvest is for simulating the optimal gains of multiple equity investments. The program decides which of all available equities to invest in at any single time, by calculating the instantaneous Shannon probability and statistics of all equities, and then using statistical estimation techniques to estimate the accuracy of the calculated statistics. The input file structure is a text file consisting of records, in temporal order, one record per time series sample of each equity. Blank records are ignored, and comment records are signified by a '#' character as the first non white space character in the record. Each data record represents an equity transaction, and consists of exactly three fields, separated by white space. The fields are ordered by time stamp, equity ticker identifier, and closing price, for example:
1 ABC 333.6
2 DEF 7.2
3 GHI 335.9
The output file structure is a Unix tab delimited text file, with the optional time stamp field, (if the -t argument is given,) the optional index value, (if the -i option is given,) the portfolio value, the optional margin fraction, (if the -q option is given,) followed by an optional list of the equity ticker identifiers, (if the -s option is given,) each followed by an optional equal, '=' character and the fraction of the portfolio that should invested in that equity, (if the -a option is given,) for example:
1 999.64 1000.00 0.65 ABC=0.10 DEF=0.10 GHI=0.10
2 998.19 1010.00 0.65 ABC=0.10 DEF=0.10 GHI=0.10
3 998.19 1017.55 0.65 ABC=0.11 DEF=0.12 GHI=0.08
4 997.52 1020.76 0.65 ABC=0.10 DEF=0.07 GHI=0.12
which at time interval 4, the index value was 997.52, the portfolio value was 1020.76, and 65% of the portfolio should be on margin, 10% of the portfolio's total value should be invested in the equity with equity identifier ABC, 7% in DEF, and, 12% in GHI. The fields are tab delimited. INTRODUCTIONOne of the prevailing concepts in financial quantitative analysis, (eg., "financial engineering,") is that equity prices exhibit "random walk," (eg., Brownian motion, or fractal,) characteristics. The presentation by Brian Arthur [Art95] offers a compelling theoretical framework for the random walk model. William A. Brock and Pedro J. F. de Lima [BdL95], among others, have published empirical evidence supporting Arthur's theoretical arguments. There is a large mathematical infrastructure available for applications of fractal analysis to equity markets. For example, the publications authored by Richard M. Crownover [Cro95], Edgar E. Peters [Pet91], and Manfred Schroeder [Sch91] offer formal methodologies, while the books by John L. Casti [Cas90], [Cas94] offer a less formal approach for the popular press. There are interesting implications that can be exploited if equity prices exhibit fractal characteristics:
Note that 1) and 2) above can be exploited to formulate an optimal hedging strategy; 3), and 4) would tend to imply that "market timing" is not attainable; 5), and 6) can be exploited to formulate an optimal buy-sell strategy; and 7) would tend to indicate that, under certain circumstances, "market timing", at least in a probabilistic sense, may be viable. DERIVATIONAs a tutorial, the derivation will start with a simple compound interest equation. This equation will be extended to a first order random walk model of equity prices. Finally, optimizations will derived based on the random walk model that are useful in optimizing equity portfolio performance. If we consider capital, V, invested in a savings account, and calculate the growth of the capital over time:
V(t) = V(t - 1)(1 + a(t)) ......................(1.1)
where a(t) is the interest rate at time t, (usually a constant[1].) In equities, a(t) is not constant, and fluctuates, perhaps being negative at certain times, (meaning that the value of the equity decreased.) This fluctuation in an equity's value can be represented by modifying a(t) in Equation (1.1):
a(t) = f(t) * F(T) ............................(1.2)
where the product f * F is the fluctuation in the equity's value at time t. An equity's value, over time, is similar to a simple tossed coin game [Sch91, pp. 128], where f(t) is the fraction of a gambler's capital wagered on a toss of the coin, at time t, and F(t) is a random variable[2], signifying whether the game was a win, or a loss, ie., whether the gambler's capital increased or decreased, and by how much. The amount the gambler's capital increased or decreased is f(t) * F(t). In general, F(t) is a function of a random variable, with an average, over time, of avgf, and a root mean square value, rmsf, of unity. Note that for simple, time invariant, compound interest, F(t) has an average and root mean square, both being unity, and f(t) is simply the interest rate, which is assumed to be constant. For a simple, single coin game, F(t) is a fixed increment, (ie., either +1 or -1,) random generator. From an analytical perspective, it would be advantageous to measure the statistical characteristics of the generator. Substituting Equation (1.2) into Equation (1.1)[3]:
V(t) = V(t - 1)(1 + f(t) * F(t)) ...............(1.3)
and subtracting V(t - 1) from both sides:
V(t) - V(t - 1) = V(t - 1) (1 + f(t) * F(t)) -
V(t - 1) .......................................(1.4)
and dividing both sides by V(t - 1):
V(t) - V(t - 1)
--------------- =
V(t - 1)
V(t - 1) (1 + f(t) * F(t)) - V(t - 1)
------------------------------------- ..........(1.5)
V(t - 1)
and combining:
V(t) - V(t - 1)
--------------- =
V(t - 1)
(1 + f(t) * F(t) ) - 1 = f(t) * F(t) ...........(1.6)
We now have a "prescription," or process, for calculating the characteristics of the random process that determines an equity's price, over time. That process is, for each unit of time, subtract the value of the equity at the previous time from the value of the equity at the current time, and divide this by the value of the equity at the previous time. The root mean square[4] of these values are the root mean square value of the random process. The average of these values are the average of the random process, avgf. The root mean square of these values can be calculated by any convenient means, and will be represented by rms. The average of these values can be found by any convenient means, and will be represented by avg[5]. Therefore, if f(t) = f, and assuming that it does not vary over time:
rms = f ........................................(1.7)
which, if there are sufficiently many samples, is a metric of the equity's price "volatility," and:
avg = f * F(t) .................................(1.8)
and if there are sufficiently many samples, the average of F(t) is simply avgf, or:
avg = f * avgf .................................(1.9)
which are the metrics of the equity's random process. Note that this is the "effective" compound interest rate from Equation (1.1). Equations (1.7) and (1.9) are important equations, since they can be used in portfolio management. For example, Equation (1.7) states that portfolio volatility is calculated as the root mean square sum of the individual volatilities of the equities in the portfolio. Equation (1.9) states that the averages of the normalized increments of the equity prices add together linearly[6] in the portfolio. Dividing Equation (1.9) by Equation (1.7) results in the two f's canceling, or:
avg
--- = avgf ....................................(1.10)
rms
There may be analytical advantages to "model" F(t) as a simple tossed coin game, (either played with a single coin, or multiple coins, ie., many coins played at one time, or a single coin played many times[7].) The number of wins minus the number of losses, in many iterations of a single coin tossing game would be:
P - (1 - P) = 2P - 1 ..........................(1.11)
where P is the probability of a win for the tossed coin. (This probability is traditionally termed, the "Shannon probability" of a win.) Note that from the definition of F(t) above, that P = avgf. For a fair coin, (ie., one that comes up with a win 50% of the time,) P = 0.5, and there is no advantage, in the long run, to playing the game. However, if P > 0.5, then the optimal fraction of capital wagered on each iteration of the single coin tossing game, f, would be 2P - 1. Note that if multiple coins were used for each iteration of the game, we would expect that the volatility of the gambler's capital to increase as the square root of the number of coins used, and the growth to increase linearly with the number of coins used, irregardless of whether many coins were tossed at once, or one coin was tossed many times, (ie., our random generator, F(t) would assume a binomial distribution and if the number of coins was very large, then F(t) would assume, essentially, a Gaussian distribution.) Many equities have a Gaussian distribution for the random process, F(t). It may be advantageous to determine the Shannon probability to analyze equity investment strategies. From Equation (1.10):
avg
--- = avgf = 2P - 1 ...........................(1.12)
rms
or:
avg
--- + 1 = 2P ..................................(1.13)
rms
and:
avg
--- + 1
rms
P = ------- ...................................(1.14)
2
where only the average and root mean square of the normalized increments need to be measured, using the "prescription" or process outlined above. Interestingly, what Equation (1.12) states is that the "best" equity investment is not, necessarily, the equity that has the largest average growth. A better investment criteria is to choose the equity that has the largest growth, while simultaneously having the smallest volatility. Continuing with this line of reasoning, and rearranging Equation (1.12):
avg = rms * (2P - 1) ..........................(1.15)
which is an important equation since it states that avg, (and the parameter that should be maximized,) is equal to rms, which is the measure of the volatility of the equity's value, multiplied by the quantity, twice the likelihood that the equity's value will increase in the next time interval, minus unity. As derived in the Section, OPTIMIZATION, below, the optimal growth occurs when f = rms = 2P - 1. Under optimal conditions, Equation (1.14) becomes:
rms + 1
P = ------- ...................................(1.16)
2
or, sqrt (avg) = rms, (again, under optimal conditions,) and substituting into Equation (1.14):
sqrt (avg) + 1
P = -------------- ............................(1.17)
2
giving three different computational methods for measuring the statistics of an equity's value. Note that, from Equations (1.14) and (1.12), that since avgf = avg / rms = (2P - 1), choosing the largest value of the Shannon probability, P, will also choose the largest value of the ratio of avg / rms, rms, or avg, respectively, in Equations (1.14), (1.16), or (1.17). This suggests a method for determination of equity selection criteria. (Note that under optimal conditions, all three equations are identical-only the metric methodology is different. Under non-optimal conditions, Equation (1.14) should be used. Unfortunately, any calculation involving the average of the normalized increments of an equity value time series will be very "sluggish," meaning that practical issues may prevail, suggesting a preference for Equation (1.17).) However, this would imply that the equities are known to be optimal, ie., rms = 2P + 1, which, although it is nearly true for most equities, is not true for all equities. There is some possibility that optimality can be verified by metrics:
2
if avg < rms
then rms = f is too large in Equation (1.12)
2
else if avg > rms
then rms = f is too small in Equation (1.12)
2
else avg = rms
and the equities time series is optimal, ie., rms
= f = 2P - 1 from Equation (1.36), below
HEURISTIC APPROACHESThere have been several heuristic approaches suggested, for example, using the absolute value of the normalized increments as an approximation to the root mean square, rms, and calculating the Shannon probability, P by Equation (1.16), using the absolute value, abs, instead of the rms. The statistical estimate in such a scheme should use the same methodology as in the root mean square. Another alternative is to model equity value time series as a fixed increment fractal, ie., by counting the up movements in an equity's value. The Shannon probability, P, is then calculated by the quotient of the up movements, divided by the total movements. There is an issue with this model, however. Although not common, there can be adjacent time intervals where an equity's value does not change, and it is not clear how the accounting procedure should work. There are several alternatives. For example, no changes can be counted as up movements, or as down movements, or disregarded entirely, or counted as both. The statistical estimate should be performed as in Equation (1.14), with an rms of unity, and an avg that is the Shannon probability itself-that is the definition of a fixed increment fractal. MARKETWe now have a "first order prescription" that enables us to analyze fluctuations in equity values, although we have not explained why equity values fluctuate the way they do. For a formal presentation on the subject, see the bibliography in [Art95] which, also, offers non-mathematical insight into the subject. Consider a very simple equity market, with only two people holding equities. Equity value "arbitration" (ie., how equity values are determined,) is handled by one person posting (to a bulletin board,) a willingness to sell a given number of equities at a given price, to the other person. There is no other communication between the two people. If the other person buys the equity, then that is the value of the equity at that time. Obviously, the other person will not buy the equity if the price posted is too high-even if ownership of the equity is desired. For example, the other person could simply decide to wait in hopes that a favorable price will be offered in the future. What this means is that the seller must consider not only the behavior of the other person, but what the other person thinks the seller's behavior will be, ie., the seller must base the pricing strategy on the seller's pricing strategy. Such convoluted logical processes are termed "self referential," and the implication is that the market can never operate in a consistent fashion that can be the subject of deductive analysis [Pen89, pp. 101][8]. As pointed out by [Art95, Abstract], these types of indeterminacies pervade economics[9]. What the two players do, in absence of a deductively consistent and complete theory of the market, is to rely on inductive reasoning. They form subjective expectations or hypotheses about how the market operates. These expectations and hypothesis are constantly formulated and changed, in a world that forms from others' subjective expectations. What this means is that equity values will fluctuate as the expectations and hypothesis concerning the future of equity values change[10]. The fluctuations created by these indeterminacies in the equity market are represented by the term f(t) * F(t) in Equation (1.3), and since there are many such indeterminacies, we would anticipate F(t) to have a Gaussian distribution. This is a rather interesting conclusion, since analyzing the aggregate actions of many "agents," each operating on subjective hypothesis in a market that is deductively indeterminate, can result in a system that can not only be analyzed, but optimized. OPTIMIZATIONThe only remaining derivation is to show that the optimal wagering strategy is, as cited above:
f = rms = 2P - 1 ..............................(1.18)
where f is the fraction of a gambler's capital wagered on each toss of a coin that has a Shannon probability, P, of winning. Following [Rez94, pp. 450], or [Kel56], consider that the gambler has a private wire into the future, (ie., an inductive hypothesis,) who places wagers on the outcomes of a game of chance. We assume that the side information which he receives has a probability, P, of being true, and of 1 - P, of being false. Let the original capital of gambler be V(0), and V(n) his capital after the n'th wager. Since the gambler is not certain that the side information is entirely reliable, he places only a fraction, f, of his capital on each wager. Thus, subsequent to n many wagers, assuming the independence of successive tips from the future, his capital is:
w l
V(n) = (1 + f) (1 - f) V (0) ................(1.19)
where w is the number of times he won, and l = n - w, the number of times he lost. These numbers are, in general, values taken by two random variables, denoted by W and L. According to the law of large numbers:
1
lim - W = P .........................(1.20)
n -> infinity n
1
lim - L = q = 1 - P .................(1.21)
n -> infinity n
The problem with which the gambler is faced is the determination of f leading to the maximum of the average exponential rate of growth of his capital. That is, he wishes to maximize the value of:
1 V(n)
G = lim - ln ---- ...................(1.22)
n -> infinity n V(0)
with respect to f, assuming a fixed original capital and specified P:
W
G = lim - ln (1 + f) +
n -> infinity n
L
+ - ln (1 - f) ...........(1.23)
n
or:
G = P ln (1 + f) + q ln (1 - f) ...............(1.24)
which, by taking the derivative with respect to f, and equating to zero, can be shown to have a maxima when:
dG P - 1 1 - P
-- = P(1 + f) (1 - f) -
df
1 - P - 1 P
(1 - P)(1 - f) (1 + f) = 0 ..........(1.25)
combining terms:
P - 1 1 - P
0 = P(1 + f) (1 - f) -
P P
(1 - P)(1 - f) (1 + f ) .....................(1.26)
and splitting:
P - 1 1 - P
P(1 + f) (1 - f) =
P P
(1 - P)(1 - f) (1 + f) ......................(1.27)
then taking the logarithm of both sides:
ln (P) + (P - 1) ln (1 + f) + (1 - P) ln (1 - f) =
ln (1 - P) - P ln (1 - f) + P ln (1 + f) ......(1.28)
and combining terms:
(P - 1) ln (1 + f) - P ln (1 + f) +
(1 - P) ln (1 - f) + P ln (1 - f) =
ln (1 - P) - ln (P) ...........................(1.29)
or:
ln (1 - f) - ln (1 + f) =
ln (1 - P) - ln (P)...........................(1.30)
and performing the logarithmic operations:
1 - f 1 - P
ln ----- = ln ----- ...........................(1.31)
1 + f P
and exponentiating:
1 - f 1 - P
----- = ----- .................................(1.32)
1 + f P
which reduces to:
P(1 - f) = (1 - P)(1 + f) .....................(1.33)
and expanding:
P - Pf = 1 - Pf - P + f .......................(1.34)
or:
P = 1 - P + f .................................(1.35)
and, finally:
f = 2P - 1 ....................................(1.36)
Note that Equation (1.24), which, since rms = f, can be rewritten:
G = P ln (1 + rms) + (1 - P) ln (1 - rms) .....(1.37)
where G is the average exponential rate of growth in an equity's value, from one time interval to the next, (ie., the exponentiation of this value minus unity[11] is the "effective interest rate", as expressed in Equation (1.1),) and, likewise, Equation (1.36) can be rewritten:
rms = 2P - 1 ..................................(1.38)
and substituting:
G = P ln (1 + 2P - 1) +
(1 - P) ln (1 - (2P - 1)) .................(1.39)
or:
G = P ln (2P) +
(1 - P) ln (2 (1 - P)) ....................(1.40)
using a binary base for the logarithm:
G = P ln (2P) +
2
(1 - P) ln (2 (1 - P)) ....................(1.41)
2
and carrying out the operations:
G = P ln (2) + P ln (P) +
2 2
(1 - P) ln (2) + (1 - P) ln (1 - P)) ......(1.42)
2 2
which is:
G = P ln (2) + P ln (P) +
2 2
ln (2) - P ln (2) + (1 - P) ln (1 - P) ....(1.43)
2 2 2
and canceling:
G = 1 + P ln (P) + (1 - P) ln (1 - P) .........(1.44)
2 2
if the gambler's wagering strategy is optimal, ie., f = rms = 2P - 1, which is identical to the equation in [Sch91, pp. 151]. FIXED INCREMENT FRACTALIt was mentioned that it would be useful to model equity prices as a fixed increment fractal, ie., an unfair tossed coin game. As above, consider a gambler, wagering on the iterated outcomes of an unfair tossed coin game. A fraction, f, of the gambler's capital will be wagered on the outcome of each iteration of the unfair tossed coin, and if the coin comes up heads, with a probability, P, then the gambler wins the iteration, (and an amount equal to the wager is added to the gambler's capital,) and if the coin comes up tails, with a probability of 1 - P, then the gambler looses the iteration, (and an amount of the wager is subtracted from the gambler's capital.) If we let the outcome of the first coin toss, (ie., whether it came up as a win or a loss,) be c(1) and the outcome of the second toss be c(2), and so on, then the outcome of the n'th toss, c(n), would be:
[win, with a probability of P
c(n) = [
[loose, with a probability of 1 - P
for convenience, let a win to be represented by +1, and a loss by -1:
[+1, with a probability of P
c(n) = [
[-1, with a probability of 1 - P
for the reason that when we multiply the wager, f, by c(n), it will be a positive number, (ie., the wager will be added to the capital,) and for a loss, it will be a negative number, (ie., the wager will be subtracted from the capital.) This is convenient, since the increment, by with the gambler's capital increased or decreased in the n'th iteration of the game is f * c(n). If we let V(0) be the initial value of the gambler's capital, V(1) be the value of the gambler's capital after the first iteration of the game, then:
V(1) = V(0) * (1 + c(1) * f(1)) ...............(1.45)
after the first iteration of the game, and:
V(2) = V(0) * ((1 + c(1) * f(1)) *
(1 + c(2) * f(2))) ....................(1.46)
after the second iteration of the game, and, in general, after the n'th iteration of the game:
V(n) = V(0) * ((1 + c(1) * f(1)) *
(1 + c(2) * f(2)) * ...
* (1 + c(n) * f(n)) *
(1 + c(n + 1) * f(n + 1))) .............(1.47)
For the normalized increments of the time series of the gambler's capital, it would be convenient to rearrange these formulas. For the first iteration of the game:
V(1) - V(0) = V(0) * (1 + c(1) * f(1)) - V(0) .(1.48)
or
V(1) - V(0) V(0) * (1 + c(1) * f(1)) - V(0)
----------- = ------------------------------- .(1.49)
V(0) V(0)
and after reducing, the first normalized increment of the gambler's capital time series is:
V(1) - V(0)
----------- = (1 + c(1) * f(1)) - 1
V(0)
= c(1) * f(1) .....................(1.50)
and for the second iteration of the game:
V(2) = V(0) * ((1 + c(1) * f(1)) *
(1 + c(2) * f(2))) .....................(1.51)
but V(0) * ((1 + c(1) * f(1)) is simply V(1):
V(2) = V(1) * (1 + c(2) * f(2)) ...............(1.52)
or:
V(2) - V(1) = V(1) * (1 + c(2) * f(2)) - V(1) .(1.53)
which is:
V(2) - V(1) V(1) * (1 + c(2) * f(2)) - V(1)
----------- = ------------------------------- .(1.54)
V(1) V(1)
and after reducing, the second normalized increment of the gambler's capital time series is:
V(2) - V(1)
----------- = 1 + c(2) * f(2)) - 1
V(1)
= c(2) * f(2) .....................(1.55)
and it should be obvious that the process can be repeated indefinitely, so, the n'th normalized increment of the gambler's capital time series is:
V(n) - V(n - 1)
--------------- = c(n) * f(n) .................(1.56)
V(n)
which is Equation (1.6). DATA SET SIZE CONSIDERATIONSThis section addresses the question "is there reasonable evidence to justify investment in an equity based on data set size?" The Shannon probability of a time series is the likelihood that the value of the time series will increase in the next time interval. The Shannon probability is measured using the average, avg, and root mean square, rms, of the normalized increments of the time series. Using the rms to compute the Shannon probability, P:
rms + 1
P = ------- ...................................(1.57)
2
However, there is an error associated with the measurement of rms do to the size of the data set, N, (ie., the number of records in the time series,) used in the calculation of rms. The confidence level, c, is the likelihood that this error is less than some error level, e. Over the many time intervals represented in the time series, the error will be greater than the error level, e, (1 - c) * 100 percent of the time-requiring that the Shannon probability, P, be reduced by a factor of c to accommodate the measurement error:
rms - e + 1
Pc = ----------- ..............................(1.58)
2
where the error level, e, and the confidence level, c, are calculated using statistical estimates, and the product P times c is the effective Shannon probability that should be used in the calculation of optimal wagering strategies. The error, e, expressed in terms of the standard deviation of the measurement error do to an insufficient data set size, esigma, is:
e
esigma = --- sqrt (2N) ........................(1.59)
rms
where N is the data set size = number of records. From this, the confidence level can be calculated from the cumulative sum, (ie., integration) of the normal distribution, ie.:
Note that the equation:
rms - e + 1
Pc = ----------- ..............................(1.60)
2
will require an iterated solution since the cumulative normal distribution is transcendental. For convenience, let F(esigma) be the function that given esigma, returns c, (ie., performs the table operation, above,) then:
rms - e + 1
P * F(esigma) = -----------
2
rms * esigma
rms - ------------ + 1
sqrt (2N)
= ---------------------- ........(1.61)
2
Then:
rms * esigma
rms - ------------ + 1
rms + 1 sqrt (2N)
------- * F(esigma) = ---------------------- ..(1.62)
2 2
or:
rms * esigma
(rms + 1) * F(esigma) = rms - ------------ + 1 (1.63)
sqrt (2N)
Letting a decision variable, decision, be the iteration error created by this equation not being balanced:
rms * esigma
decision = rms - ------------ + 1
sqrt (2N)
- (rms + 1) * F(esigma) ...........(1.64)
which can be iterated to find F(esigma), which is the confidence level, c. Note that from Equation (1.58):
rms - e + 1
Pc = -----------
2
and solving for rms - e, the effective value of rms compensated for accuracy of measurement by statistical estimation:
rms - e = (2 * P * c) - 1 .....................(1.65)
and substituting into Equation (1.57):
rms + 1
P = -------
2
rms - e = ((rms + 1) * c) - 1 .................(1.66)
and defining the effective value of rms as rmseff:
rmseff = rms - e ..............................(1.67)
From Equations (1.16) and (1.17) it can be seen that if optimality exists, ie., f = 2P - 1, or:
2
avg = rms ....................................(1.68)
or:
2
avgeff = rmseff ..............................(1.69)
As an example of this algorithm, if the Shannon probability, P, is 0.51, corresponding to an rms of 0.02, then the confidence level, c, would be 0.996298, or the error level, e, would be 0.003776, for a data set size, N, of 100. Likewise, if P is 0.6, corresponding to an rms of 0.2 then the confidence level, c, would be 0.941584, or the error level, e, would be 0.070100, for a data set size of 10. Robustness is an issue in algorithms that, potentially, operate real time. The traditional means of implementation of statistical estimates is to use an integration process inside of a loop that calculates the cumulative of the normal distribution, controlled by, perhaps, a Newton Method approximation using the derivative of cumulative of the normal distribution, ie., the formula for the normal distribution:
2
1 - x / 2
f(x) = ------------- * e ............(1.70)
sqrt (2 * PI)
Numerical stability and convergence issues are an issue in such processes. The Shannon probability of a time series is the likelihood that the value of the time series will increase in the next time interval. The Shannon probability is measured using the average, avg, and root mean square, rms, of the normalized increments of the time series. Using the avg to compute the Shannon probability, P:
sqrt (avg) + 1
P = -------------- ............................(1.71)
2
However, there is an error associated with the measurement of avg do to the size of the data set, N, (ie., the number of records in the time series,) used in the calculation of avg. The confidence level, c, is the likelihood that this error is less than some error level, e. Over the many time intervals represented in the time series, the error will be greater than the error level, e, (1 - c) * 100 percent of the time-requiring that the Shannon probability, P, be reduced by a factor of c to accommodate the measurement error:
sqrt (avg - e) + 1
Pc = ------------------ .......................(1.72)
2
where the error level, e, and the confidence level, c, are calculated using statistical estimates, and the product P times c is the effective Shannon probability that should be used in the calculation of optimal wagering strategies. The error, e, expressed in terms of the standard deviation of the measurement error do to an insufficient data set size, esigma, is:
e
esigma = --- sqrt (N) .........................(1.73)
rms
where N is the data set size = number of records. From this, the confidence level can be calculated from the cumulative sum, (ie., integration) of the normal distribution, ie.:
Note that the equation:
sqrt (avg - e) + 1
Pc = ------------------ .......................(1.74)
2
will require an iterated solution since the cumulative normal distribution is transcendental. For convenience, let F(esigma) be the function that given esigma, returns c, (ie., performs the table operation, above,) then:
sqrt (avg - e) + 1
P * F(esigma) = ------------------
2
rms * esigma
sqrt [avg - ------------] + 1
sqrt (N)
= ----------------------------- .(1.75)
2
Then:
sqrt (avg) + 1
--------------- * F(esigma) =
2
rms * esigma
sqrt [avg - ------------] + 1
sqrt (N)
----------------------------- .............(1.76)
2
or:
(sqrt (avg) + 1) * F(esigma) =
rms * esigma
sqrt [avg - ------------] + 1 .............(1.77)
sqrt (N)
Letting a decision variable, decision, be the iteration error created by this equation not being balanced:
rms * esigma
decision = sqrt [avg - ------------] + 1
sqrt (N)
- (sqrt (avg) + 1) * F(esigma) .....(1.78)
which can be iterated to find F(esigma), which is the confidence level, c. There are two radicals that have to be protected from numerical floating point exceptions. The sqrt (avg) can be protected by requiring that avg >= 0, (and returning a confidence level of 0.5, or possibly zero, in this instance-a negative avg is not an interesting solution for the case at hand.) The other radical:
rms * esigma
sqrt [avg - ------------] .....................(1.79)
sqrt (N)
and substituting:
e
esigma = --- sqrt (N) .........................(1.80)
rms
which is:
e
rms * --- sqrt (N)
rms
sqrt [avg - ------------------] ...............(1.81)
sqrt (N)
and reducing:
sqrt [avg - e] ................................(1.82)
requiring that:
avg >= e ......................................(1.83)
Note that if e > avg, then Pc < 0.5, which is not an interesting solution for the case at hand. This would require:
avg
esigma <= --- sqrt (N) ........................(1.84)
rms
Obviously, the search algorithm must be prohibited from searching for a solution in this space. (ie., testing for a solution in this space.) The solution is to limit the search of the confidence array to values that are equal to or less than:
avg
--- sqrt (N) ..................................(1.85)
rms
which can be accomplished by setting integer variable, top, usually set to sigma_limit - 1, to this value. Note that from Equation (1.72):
sqrt (avg - e) + 1
Pc = ------------------
2
and solving for avg - e, the effective value of avg compensated for accuracy of measurement by statistical estimation:
2
avg - e = ((2 * P * c) - 1) ..................(1.86)
and substituting into Equation (1.71):
sqrt (avg) + 1
P = --------------
2
2
avg - e = (((sqrt (avg) + 1) * c) - 1) .......(1.87)
and defining the effective value of avg as avgeff:
avgeff = avg - e ..............................(1.88)
From Equations (1.16) and (1.17) it can be seen that if optimality exists, ie., f = 2P - 1, or:
2
avg = rms ....................................(1.89)
or:
rmseff = sqrt (avgeff) ........................(1.90)
As an example of this algorithm, if the Shannon probability, P, is 0.52, corresponding to an avg of 0.0016, and an rms of 0.04, then the confidence level, c, would be 0.987108, or the error level, e, would be 0.000893, for a data set size, N, of 10000. Likewise, if P is 0.6, corresponding to an rms of 0.2, and an avg of 0.04, then the confidence level, c, would be 0.922759, or the error level, e, would be 0.028484, for a data set size of 100. The Shannon probability of a time series is the likelihood that the value of the time series will increase in the next time interval. The Shannon probability is measured using the average, avg, and root mean square, rms, of the normalized increments of the time series. Using both the avg and the rms to compute the Shannon probability, P:
avg
--- + 1
rms
P = ------- ...................................(1.91)
2
However, there is an error associated with both the measurement of avg and rms do to the size of the data set, N, (ie., the number of records in the time series,) used in the calculation of avg and rms. The confidence level, c, is the likelihood that this error is less than some error level, e. Over the many time intervals represented in the time series, the error will be greater than the error level, e, (1 - c) * 100 percent of the time-requiring that the Shannon probability, P, be reduced by a factor of c to accommodate the measurement error:
avg - ea
-------- + 1
rms + er
P * ca * cr = ------------ ....................(1.92)
2
where the error level, ea, and the confidence level, ca, are calculated using statistical estimates, for avg, and the error level, er, and the confidence level, cr, are calculated using statistical estimates for rms, and the product P * ca * cr is the effective Shannon probability that should be used in the calculation of optimal wagering strategies, (which is the product of the Shannon probability, P, times the superposition of the two confidence levels, ca, and cr, ie., P * ca * cr = Pc, eg., the assumption is made that the error in avg and the error in rms are independent.) The error, er, expressed in terms of the standard deviation of the measurement error do to an insufficient data set size, esigmar, is:
er
esigmar = --- sqrt (2N) .......................(1.93)
rms
where N is the data set size = number of records. From this, the confidence level can be calculated from the cumulative sum, (ie., integration) of the normal distribution, ie.:
Note that the equation:
avg
-------- + 1
rms + er
P * cr = ------------ .........................(1.94)
2
will require an iterated solution since the cumulative normal distribution is transcendental. For convenience, let F(esigmar) be the function that given esigmar, returns cr, (ie., performs the table operation, above,) then:
avg
-------- + 1
rms + er
P * F(esigmar) = ------------ =
2
avg
------------------- + 1
esigmar * rms
rms + -------------
sqrt (2N)
----------------------- ......(1.95)
2
Then:
avg
--- + 1
rms
------- * F(esigmar) =
2
avg
------------------- + 1
esigmar * rms
rms + -------------
sqrt (2N)
----------------------- ................(1.96)
2
or:
avg
(--- + 1) * F(esigmar) =
rms
avg
------------------- + 1 ................(1.97)
esigmar * rms
rms + -------------
sqrt (2N)
Letting a decision variable, decision, be the iteration error created by this equation not being balanced:
avg
decision = ------------------- + 1
esigmar * rms
rms + -------------
sqrt (2N)
avg
- (--- + 1) * F(esigmar) ..........(1.98)
rms
which can be iterated to find F(esigmar), which is the confidence level, cr. The error, ea, expressed in terms of the standard deviation of the measurement error do to an insufficient data set size, esigmaa, is:
ea
esigmaa = --- sqrt (N) ........................(1.99)
rms
where N is the data set size = number of records. From this, the confidence level can be calculated from the cumulative sum, (ie., integration) of the normal distribution, ie.:
Note that the equation:
avg - ea
-------- + 1
rms
P * ca = ------------ ........................(1.100)
2
will require an iterated solution since the cumulative normal distribution is transcendental. For convenience, let F(esigmaa) be the function that given esigmaa, returns ca, (ie., performs the table operation, above,) then:
avg - ea
-------- + 1
rms
P * F(esigmaa) = ------------ =
2
esigmaa * rms
avg - -------------
sqrt (N)
------------------- + 1
rms
----------------------- .....(1.101)
2
Then:
avg
--- + 1
rms
------- * F(esigmaa) =
2
esigmaa * rms
avg - -------------
sqrt (N)
------------------- + 1
rms
----------------------- ...............(1.102)
2
or:
avg
(--- + 1) * F(esigmaa) =
rms
esigmaa * rms
avg - -------------
sqrt (N)
------------------- + 1 ...............(1.103)
rms
Letting a decision variable, decision, be the iteration error created by this equation not being balanced:
esigmaa * rms
avg - -------------
sqrt (N)
decision = ------------------- + 1
rms
avg
- (--- + 1) * F(esigmaa) .................(1.104)
rms
which can be iterated to find F(esigmaa), which is the confidence level, ca. Note that from Equation (1.94):
avg
-------- + 1
rms + er
P * cr = ------------
2
and solving for rms + er, the effective value of rms compensated for accuracy of measurement by statistical estimation:
avg
rms + er = ---------------- ..................(1.105)
(2 * P * cr) - 1
and substituting into Equation (1.100):
avg
--- + 1
rms
P = -------
2
avg
rms + er = -------------------- ..............(1.106)
avg
((--- + 1) * cr) - 1
rms
and defining the effective value of avg as rmseff:
rmseff = rms +/- er ..........................(1.107)
Note that from Equation (1.100):
avg - ea
-------- + 1
rms
P * ca = ------------
2
and solving for avg - ea, the effective value of avg compensated for accuracy of measurement by statistical estimation:
avg - ea = ((2 * P * ca) - 1) * rms ..........(1.108)
and substituting into Equation (1.91):
avg
--- + 1
rms
P = -------
2
avg
avg - ea = (((--- + 1) * ca) - 1) * rms ......(1.109)
rms
and defining the effective value of avg as avgeff:
avgeff = avg - ea ............................(1.110)
As an example of this algorithm, if the Shannon probability, P, is 0.51, corresponding to an rms of 0.02, then the confidence level, c, would be 0.983847, or the error level in avg, ea, would be 0.000306, and the error level in rms, er, would be 0.001254, for a data set size, N, of 20000. Likewise, if P is 0.6, corresponding to an rms of 0.2 then the confidence level, c, would be 0.947154, or the error level in avg, ea, would be 0.010750, and the error level in rms, er, would be 0.010644, for a data set size of 10. As a final discussion to this section, consider the time series for an equity. Suppose that the data set size is finite, and avg and rms have both been measured, and have been found to both be positive. The question that needs to be resolved concerns the confidence, not only in these measurements, but the actual process that produced the time series. For example, suppose, although there was no knowledge of the fact, that the time series was actually produced by a Brownian motion fractal mechanism, with a Shannon probability of exactly 0.5. We would expect a "growth" phenomena for extended time intervals [Sch91, pp. 152], in the time series, (in point of fact, we would expect the cumulative distribution of the length of such intervals to be proportional to erf (1 / sqrt (t)).) Note that, inadvertently, such a time series would potentially justify investment. What the methodology outlined in this section does is to preclude such scenarios by effectively lowering the Shannon probability to accommodate such issues. In such scenarios, the lowered Shannon probability will cause data sets with larger sizes to be "favored," unless the avg and rms of a smaller data set size are "strong" enough in relation to the Shannon probabilities of the other equities in the market. Note that if the data set sizes of all equities in the market are small, none will be favored, since they would all be lowered by the same amount, (if they were all statistically similar.) To reiterate, in the equation avg = rms * (2P - 1), the Shannon probability, P, can be compensated by the size of the data set, ie., Peff, and used in the equation avgeff = rms * (2Peff - 1), where rms is the measured value of the root mean square of the normalized increments, and avgeff is the effective, or compensated value, of the average of the normalized increments. DATA SET DURATION CONSIDERATIONSAn additional accuracy issue, besides data set size, is the time interval over which the data was obtained. There is some possibility that the data set was taken during an extended run length, either negative or positive, and the Shannon probability will have to be compensated to accommodate this measurement error. The chances that a run length will exceed time, t, is [Sch91, pp. 160]:
1 - erf (1 / sqrt (t)) .......................(1.111)
or the Shannon probability, P, will have to be compensated by a factor of:
erf (1 / sqrt (t)) ...........................(1.112)
giving a compensated Shannon probability, Pcomp:
Pcomp = Peff * (1 - erf (1 / sqrt (t))).......(1.113)
Fortunately, since confidence levels are calculated from the normal probability function, the same lookup table used for confidence calculations (ie., the cumulative of a normal distribution,) can be used to calculate the associated error function. To use the value of the normal probability function to calculate the error function, erf (N), proceed as follows; since erf (X / sqrt (2)) represents the error function associated with the normal curve:
or:
erf (N) = 2 * (normal (t * sqrt (2)) - 0.5) ..(1.114)
PORTFOLIO OPTIMIZATIONLet K be the number of equities in the equity portfolio, and assume that the capital is invested equally in each of the equities, (ie., if V is the capital value of the portfolio, then the amount invested in each equity is V / K.) The portfolio value, over time, would be a time series with a root mean square value of the normalized increments, rmsp, and an average value of the normalized increments, avgp. Obviously, it would be advantageous to optimize the portfolio growth. From Equation (1.16):
rmsp + 1
Pp = -------- ................................(1.115)
2
where the root mean square value of the normalized increments of the portfolio value, rmsp, is the root mean square sum of the root mean square values of the normalized increments of each individual equity:
1 2 1 2
rmsp = sqrt ((-rms ) + (-rms ) + ...
K 1 K 2
1 2
... + (-rms ) ) .......................(1.116)
K K
or:
1 2 2 2
rmsp = - sqrt (rms + rms + ... + rms ) ....(1.117)
K 1 2 K
and Pp is the Shannon probability (ie., the likelyhood,) that the value of the portfolio time series will increase in the next time interval. Note that Equation (1.16) presumes that the portfolio's time series will be optimal, ie., rmsp = sqrt (avgp). This is probably not the case, since rmsp will always be less than the individual values of rms for the equities. Additionally, note that assuming the distribution of capital, V / K, invested in each equity to be identical may not be optimal. It is not clear if there is a formal optimization for the distribution, and, perhaps, the application of simulated annealing, linear/mathematical programming, or genetic algorithms to the distribution problem may be of some benefit. Again, letting K be the number of equities in the equity portfolio, and assuming that the capital is invested equally in each of the equities, (ie., if V is the capital value of the portfolio, then the amount invested in each equity is V / K.) The portfolio value, over time, would be a time series with a root mean square value of the normalized increments, rmsp, and an average value of the normalized increments, avgp. Obviously, it would be advantageous to optimize the portfolio growth. From Equation (1.17):
sqrt (avgp) + 1
Pp = --------------- .........................(1.118)
2
where the average value of the normalized increments of the portfolio value, avgp, is the sum of the average values of the normalized increments of each individual equity:
1 1 1
avgp = - avg + - avg + ... + - avg .........(1.119)
K 1 K 2 K K
or:
1
avgp = - (avg + avg + ... + avg ) ..........(1.120)
K 1 2 K
and Pp is the Shannon probability (ie., the likelyhood,) that the value of the portfolio time series will increase in the next time interval. Note that Equation (1.17) presumes that the portfolio's time series will be optimal, ie., rmsp = sqrt (avgp). This is probably not the case, since rmsp will always be less than the individual values of rms for the equities. Additionally, note that assuming the distribution of capital, V / K, invested in each equity to be identical may not be optimal. It is not clear if there is a formal optimization for the distribution, and, perhaps, the applications of simulated annealing or genetic algorithms to the distribution problem may be of some benefit. Again, letting K be the number of equities in the equity portfolio, and assuming that the capital is invested equally in each of the equities, (ie., if V is the capital value of the portfolio, then the amount invested in each equity is V / K.) The portfolio value, over time, would be a time series with a root mean square value of the normalized increments, rmsp, and an average value of the normalized increments, avgp. Obviously, it would be advantageous to optimize the portfolio growth. From Equation (1.14):
avgp
---- + 1
rmsp
Pp = -------- ................................(1.121)
2
where the average value of the normalized increments of the portfolio value, avgp, is the sum of the average values of the normalized increments of each individual equity, and rmsp is the root mean square sum of the root mean square values of the normalized increments of each individual equity:
1 1 1
avgp = - avg + - avg + ... + - avg .........(1.122)
K 1 K 2 K K
and:
1 2 1 2
rmsp = sqrt ((-rms ) + (-rms ) + ...
K 1 K 2
1 2
... + (-rms ) ) .......................(1.123)
K K
or:
1
avgp = - (avg + avg + ... + avg ............(1.124)
K 1 2 K
and:
1 2 2 2
rmsp = - sqrt (rms + rms + ... + rms ) ....(1.125)
K 1 2 K
and dividing:
(avg + avg + ... + avg )
avgp 1 2 K
---- = -------------------------------- ......(1.126)
rmsp 2 2 2
sqrt (rms + rms + ... + rms )
1 2 K
and Pp is the Shannon probability (ie., the likelyhood,) that the value of the portfolio time series will increase in the next time interval. The portfolio's average exponential rate of growth, Gp, would be, from Equation (1.37):
Gp = Pp ln (1 + rmsp) +
(1 - Pp) ln (1 - rmsp) ..................(1.127)
where the Shannon probability of the portfolio, Pp, is determined by one of the Equations, (1.115), (1.118), or (1.121). Note that assuming the distribution of capital, V / K, invested in each equity to be identical may not be optimal. It is not clear if there is a formal optimization for the distribution, and, perhaps, the applications of simulated annealing or genetic algorithms to the distribution problem may be of some benefit. Additionally, note that Equation (1.121) should be used for portfolio management, as opposed to Equations (1.115) and (1.118), which are not applicable, (Equations (1.115) and (1.118) are monotonically decreasing on K, the number of equities held concurrently.) Interestingly, plots of Equation (1.127) using Equations (1.121) and (1.127) to calculate the Shannon probability, Pp, of the portfolio, with the number of equities held, K, as a parameter for various values of avg and rms, tends to support the prevailing concept that the best number of equities to hold is approximately 10. There is little advantage in holding more, and a distinct disadvantage in holding less[12]. MEAN REVERTING DYNAMICSIt can be shown that the number of expected equity value "high and low" transitions scales with the square root of time, meaning that the cumulative distribution of the probability of an equity's "high or low" exceeding a given time interval is proportional to the reciprocal of the square root of the time interval, (or, conversely, that the probability of an equity's "high or low" exceeding a given time interval is proportional to the reciprocal of the time interval raised to the power 3/2 [Sch91, pp. 153]. What this means is that a histogram of the "zero free" run-lengths of an equity's price would have a 1 / (l^3/2) characteristic, where l is the length of time an equity's price was above or below "average.") This can be exploited for a short term trading strategy, which is also called "noise trading." The rationale proceeds as follows. Let l be the run length, (ie., the number of time intervals,) that an equity's value has been above or below average, then the probability that it will continue to do so in the next time interval will be:
Pt = erf (1 / sqrt (l + 1))...................(1.128)
where Pt is the "transient" probability. Naturally, it would be desirable to buy low and sell high. So, if an equity's price is below average, then the probability of an upward movement is given by Equation (1.128). If an equity's price is above average, then, then the probability that it will continue the trend is:
Pt = 1 - erf (1 / sqrt (l + 1)) ..............(1.129)
Equations (1.128) and (1.129) can be used to find the optimal time to trade one equity for another. Note that equation (1.37) can be used to find whether an equity's current price is above, or below average:
G = P ln (1 + rms) + (1 - P) ln (1 - rms)
by exponentiating both sides of the equation, and subtracting the value from the current price of the equity. Note that there is a heuristic involved in this procedure. The original derivation [Sch91, pp. 152], assumed a fixed increment Brownian motion fractal, (ie., V (n + 1) = V (n) + F (n)), which is different than Equation (1.3), V (n + 1) = V (n) (1 + F (n)). However, simulations of Equation (1.3) tend to indicate that a histogram of the "zero free" run-lengths of an equity's price would have a 1 / (l^3/2) characteristic, where l is the length of time an equity's price was above or below "average." Note that in both formulas, with identical statistical processes, the values would, intuitively, be above, or below, average in much the same way. Additionally, note that in the case of a fixed increment Brownian motion fractal, the average is known-zero, by definition. However, in this procedure, the average is measured, and this can introduce errors, since the average itself is fluctuating slightly, do to a finite data set size. Note, also, that mean reverting functionality was implemented on the infrastructure available in the program, ie., the measurement of avg and rms to determine the average growth of an equity. There are probably more expeditious implementations, for example, using a single or multi pole filter as described in APPENDIX 1 to measure the average growth of an equity. PERSISTENCEThe above derivations assume random walk fractal characteristics as a first order approximation to equity prices. Although adequate for many types of analysis, empirical evidence indicates that instead of a 50% chance of consecutive like movements, (as in a random walk fractal,) many equity prices have slightly greater than a 50% chance, which in principle, is exploitable. The traditional method of determination of persistence is by metrics on the Hurst exponent, [Pet91, pp. 61], [Cas94, pp. 253], [Fed88, pp. 149], and [Sch91, pp. 129]. As an alternative method, the number of consecutive like movements in an equity's price can be tallied, and the coefficient of the resultant distribution determined. For a simple random walk fractal, it will be the combinatorics, ie., 0.5, or 50%, of finding two like movements. Persistence means that there is greater than 50%, and is a metric on what is called "price momentum". Note that this is an exploitable attribute in "noise trading". For example, if the Hurst exponent is greater than 0.5, then both up movements, and down movements, tend to cluster, ie., such strategies as buying after two consecutive down movements, and selling after two consecutive up movements, may be viable. OTHER PROVISIONSFor simulation, the equities are represented, one per time unit. However, in the "real world," an equity can be represented multiple times in the same time unit, or not at all. This issue is addressed by:
The advantage of this scheme is that, since fractal time series are self-similar, it does not affect the wagering operations of the equities in relation to one another. |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
APPENDIX 1Approximating Statistical Estimates to a Time Series with a Single Pole Filter Note: The prototype to this program implemented statistical estimates with a single pole filter. The documentation for the implementation was moved to this Appendix. Although the approximation is marginal, reasonably good results can be obtained with this technique. Additionally, the time constants for the filters are adjustable, and, at least in principle, provide a means of adaptive computation to control the operational dynamics of the program. One of the implications of considering equity prices to have fractal characteristics, ie., random walk or Brownian motion, is that future prices can not be predicted from past equity price performance. The Shannon probability of a equity price time series is the likelihood that a equity price will increase in the next time interval. It is typically 0.51, on a day to day bases, (although, occasionally, it will be as high as 0.6) What this means, for a typical equity, is that 51% of the time, a equity's price will increase, and 49% of the time it will decrease-and there is no possibility of determining which will occur-only the probability. However, another implication of considering equity prices to have fractal characteristics is that there are statistical optimizations to maximize portfolio performance. The Shannon probability, P, is related to the optimal volatility of a equity's price, (measured as the root mean square of the normalized increments of the equity's price time series,) rms, by rms = 2P - 1. Also, the optimized average of the normalized increments is equal to the square of the rms. Unfortunately, the measurements of avg and rms must be made over a long period of time, to construct a very large data set for analytical purposes do to the necessary accuracy requirements. Statistical estimation techniques are usually employed to quantitatively determine the size of the data set for a given analytical accuracy. The calculation of the Shannon probability, P, from the average and root mean square of the normalized increments, avg and rms, respectively, will require require specialized filtering, (to "weight" the most recent instantaneous Shannon probability more than the least recent,) and statistical estimation (to determine the accuracy of the measurement of the Shannon probability.) This measurement would be based on the normalized increments, as derived in Equation (1.6):
V(t) - V(t - 1)
---------------
V(t - 1)
which, when averaged over a "sufficiently large" number of increments, is the mean of the normalized increments, avg. The term "sufficiently large" must be analyzed quantitatively. For example, Table I is the statistical estimate for a Shannon probability, P, of a time series, vs, the number of records required, based on a mean of the normalized increments = 0.04, (ie., a Shannon probability of 0.6 that is optimal, ie., rms = (2P - 1) * avg):
where avg is the average of the normalized increments, e is the error estimate in avg, c is the confidence level of the error estimate, and n is the number of records required for that confidence level in that error estimate. What Table I means is that if a step function, from zero to 0.04, (corresponding to a Shannon probability of 0.6,) is applied to the system, then after 27 records, we would be 70% confident that the error level was not greater than 0.0396, or avg was not lower than 0.0004, which corresponds to an effective Shannon probability of 0.51. Note that if many iterations of this example of 27 records were performed, then 30% of the time, the average of the time series, avg, would be less than 0.0004, and 70% greater than 0.0004. This means that the Shannon probability, 0.6, would have to be reduced by a factor of 0.85 to accommodate the error created by an insufficient data set size to get the effective Shannon probability of 0.51. Since half the time the error would be greater than 0.0004, and half less, the confidence level would be 1 - ((1 - 0.85) * 2) = 0.7, meaning that if we measured a Shannon probability of 0.6 on only 27 records, we would have to use an effective Shannon probability of 0.51, corresponding to an avg of 0.0004. For 33 records, we would use an avg of 0.0016, corresponding to a Shannon probability of 0.52, and so on. Following like reasoning, Table II is the statistical estimate for a Shannon probability, P, of a time series, vs, the number of records required, based on a root mean square of the normalized increments = 0.2, (ie., a Shannon probability of 0.6 that is optimal, ie., rms = (2P - 1) * avg):
where rms is the average of the normalized increments, e is the error estimate in rms, c is the confidence level of the error estimate, and n is the number of records required for that confidence level in that error estimate. What Table II means is that if a step function, from zero to 0.2, (corresponding to a Shannon probability of 0.6,) is applied to the system, then after 1 records, we would be 70% confident that the error level was not greater than 0.18, or rms was not lower than 0.02, which corresponds to an effective Shannon probability of 0.51. Note that if many iterations of this example of 1 records were performed, then 30% of the time, the root mean square of the time series, rms, would be less than 0.01, and 70% greater than 0.02. This means that the Shannon probability, 0.6, would have to be reduced by a factor of 0.85 to accommodate the error created by an insufficient data set size to get the effective Shannon probability of 0.51. Since half the time the error would be greater than 0.02, and half less, the confidence level would be 1 - ((1 - 0.85) * 2) = 0.7, meaning that if we measured a Shannon probability of 0.6 on only 1 record, we would have to use an effective Shannon probability of 0.51, corresponding to an rms of 0.02. For 2 records, we would use an rms of 0.06, corresponding to a Shannon probability of 0.53, and so on. And curve fitting to Tables I and II:
where the pole frequency, fp, is calculated by:
avg
ln (1 - ----)
0.04
fp = - ------------ ..........................(1.130)
2 PI n
which was derived from the exponential formula for a single pole filter, vo = vi ( 1 - e^(-t / rc)), where the pole is at 1 / (2 PI rc). The average of the necessary poles is 0.000354700, although an order of magnitude smaller could be used, as could 50% larger.
where the pole frequency, fp, is calculated by:
rms
ln (1 - ---)
0.2
fp = - ------------ ..........................(1.131)
2 PI n
which was derived from the exponential formula for a single pole filter, vo = vi ( 1 - e^(-t / rc)), where the pole is at 1 / (2 PI rc). The average of the necessary poles is 0.019251589, although an order of magnitude smaller could be used, as could 50% larger. Tables I, II, III, and IV represent an equity with a Shannon probability of 0.6, which is about the maximum that will be seen in the equity markets. Tables V and VI represent similar reasoning, but with a Shannon probability of 0.51, which is at the low end of the probability spectrum for equity markets:
where the pole frequency, fp, is calculated by:
avg
ln (1 - ------)
0.0004
fp = - --------------- .......................(1.132)
2 PI n
which was derived from the exponential formula for a single pole filter, vo = vi ( 1 - e^(-t / rc)), where the pole is at 1 / (2 PI rc). The average of the necessary poles is 0.000001282, although an order of magnitude smaller could be used, as could 70% larger.
where the pole frequency, fp, is calculated by:
rms
ln (1 - ----)
0.02
fp = - ------------ ..........................(1.133)
2 PI n
which was derived from the exponential formula for a single pole filter, vo = vi ( 1 - e^(-t / rc)), where the pole is at 1 / (2 PI rc). The average of the necessary poles is 0.006974618, although an order of magnitude smaller could be used, as could 60% larger. Table V presents real issues, in that metrics for equities with low Shannon probabilities may not be attainable with adequate precision to formulate consistent wagering strategies. (For example, 524307 business days is a little over two millenia-the required size of the data set for day trading.) Another issue is that the pole frequency changes with magnitude of the Shannon probability, as shown by comparison of Tables III, V, and IV, VI, respectively. There is some possibility that adaptive filter techniques could be implemented by dynamically change the constants in the statistical estimation filters to correspond to the instantaneous measured Shannon probability. The equations are defined, below. Another alternative is to work only with the root mean square values of the normalized increments, since the pole frequency is not as sensitive to the Shannon probability, and can function on a much smaller data set size for a given accuracy in the statistical estimate. This may be an attractive alternative if all that is desired is to rank equities by growth, (ie., pick the top 10,) since, for a given data set size, a larger Shannon probability will be chosen over a smaller. However, this would imply that the equities are known to be optimal, ie., rms = 2P + 1, which, although it is nearly true for most equities, is not true for all equities. There is some possibility that optimality can be verified by metrics:
2
if avg < rms
then rms = f is too large in Equation (1.12)
2
else if avg > rms
then rms = f is too small in Equation (1.12)
2
else avg = rms
and the equities time series is optimal, ie.,
rms = f = 2P - 1 from Equation (1.36)
These metrics would require identical statistical estimate filters for both the average and the root mean squared filters, ie., the square of rms would have the same filter pole as avg, which would be at 0.000001282, and would be conservative for Shannon probabilities above 0.51. The Shannon probability can be calculated by several methods using Equations (1.6) and (1.14). Equation (1.14):
avg
--- + 1
rms
P = -------
2
has two other useful alternative solutions if it is assumed that the equity time series is optimal, ie., rms = 2P - 1, and by substitution into Equation (1.14):
rms + 1
P = ------- ..................................(1.134)
2
and:
sqrt (avg) + 1
P = -------------- ...........................(1.135)
2
Note that in Equation (1.14) the confidence levels listed in Tables I and II should be multiplied together, and a new table made for the quotient of the average and the root mean square of the normalized increments, avg and rms respectively. However, with a two order of magnitude difference in the pole frequencies for avg and rms, the response time of the statistical estimate approximation is dominated by the avg pole. The decision criteria will be based on variations of the Shannon probability, P, and the average and root mean square of the normalized increments, avg and rms, respectively. Note that from Equation (1.14), avg = rms (2P - 1), which can be optimized/maximized. P can be calculated from Equations (1.14), (1.61), or (1.62). The measurement of the average, avg, and root mean square, rms, of the normalized increments can use different filter parameters than the root mean square of multiplier, ie., there can be an rms that uses different filter parameters, than the RMS in the equation, avg = RMS (2P - 1). By substitution, Equation (1.14) will have a decision criteria of the largest value of RMS * avg / rms, Equation (1.61) will have a decision criteria of the largest value of RMS * rms, and Equation (1.62) will have a decision criteria of RMS * sqrt (avg), or avg. These interpretations offer an alternative to the rather sluggish filters shown in Tables I, III, and V, since there can be two sets of filters, one to perform a statistical estimate approximation to the Shannon probability, and the other to perform a statistical estimate on rms, which can be several orders of magnitude faster than the filters used for the Shannon probability, enhancing dynamic operation. As a review of the methodology used to construct Tables I, II, III, IV, V, and VI, the size of the data set was obtained using the tsstatest(1) program, which can be approximated by a single pole low pass recursive discreet time filter [Con78], with the pole frequency at 0.000053 times the time series sampling frequency, for the average of the normalized increments of the time series, avg. (The rationale behind this value is that if we consider an equity with a measured Shannon probability of 0.51-a typical value-and we wish to include an uncertainty in the precision of this value based on the size of the data set, then we must decrease the Shannon probability by a factor of 0.960784314. This number comes from the fact that a Shannon probability, P', would be (0.5 / 0.51) * P = 0.980392157 * P = 0.51 * 0.980392157 = 0.5, a Shannon probability below which, no wager should be made, (as an absolute lower limit.) But if such a scenario is set up as an experiment that was performed many times, it would be expected that half the time, the measured value Shannon probability would be greater than 0.51, and half less, than the "real" value of the Shannon probability. So the Shannon probability must be reduced by a factor of c = 1 - 2(1 - 0.980392157) = 0.960784314. This value is the confidence level in the statistical estimate of the measurement error of the average of the normalized increments, avg, which for a Shannon probability of 0.51 is 0.0004, since the root mean square, rms, of the normalized increments of a time series with a Shannon probability of 0.51 is 0.02, and, if the time series is optimal, where avg = (2P - 1) * rms, then avg = 0.0004. So, we now have the error level, 0.0004, and the required confidence level, 0.960784314, and the number of required records, ie., the data set size, would be 9773, from the tsstatest(1) program. From this it can be calculated that the response to a unit step function, of a filter with the pole frequency at 0.000053 at the 9773'th record is 0.960784314. Note that at the single pole filter will provide an approximation to the statistical estimate that is low for values below this; for example, by comparison, at 2083 records it will give a confidence level of 0.5, and the tsstatest(1) program will give a confidence level of 0.5 at 1046 records-a sizeable error, but in a conservative direction.) Likewise, for the rms, with a Shannon probability of 0.51, the rms would be 0.02. For an error level of 0.02, below which no wagers should be made, and a confidence level, c, of c = 1 - 2(1 - (0.5 / 0.51)) = 0.960784314, the data set size would be between 3 and 4 records, or a pole frequency of 0.13 to 0.18. These two values, an rms pole of 0.15, and an avg pole of 0.000053 probably represent the highest bandwidth that is attainable. The advantage of the discreet time recursive single pole filter approximation to a statistical estimate is that it requires only 3 lines of code in the implementation-two for initialization, and one in the calculation construct. A "running average" methodology would offer far greater accuracy as an approximation to the statistical estimate, however the memory requirements for the average could be prohibitive if many equities were being tracked concurrently, (see Table V,) and computational resource requirements for circular buffer operation could possible be an issue. The other alternative would be to perform a true statistical estimate, however the architecture of a recursive algorithm implementation may be formidable. The single pole low pass filter is implemented from the following discrete time equation:
v = I * k2 + v * k1 ....................(1.136)
t + 1 t
where I is the value of the current sample in the time series, v are the value of the output time series, and k1 and k2 are constants determined from the following equations:
-2 * p * pi
k1 = e ............................(1.137)
and:
k2 = 1 - k1 ..................................(1.138)
where p is a constant that determines the frequency of the pole-a value of unity places the pole at the sample frequency of the time series. |