From: John Conover <email@example.com>
Subject: A leap of faith with fractal analysis ...
Date: Fri, 16 Jan 1998 19:26:12 -0800
"... all this is civilization, the Sumerian God of Wisdom tells Inanna of Uruk, who will take it back to her city, and thence give it to the world. And, if you wish its ... [enigmatic] ... benefits, he goes on, you must take all its qualities, ...[including its disadvantages] ... without argument. All these things I will give you, holy Inannna, but once taken, there can be no dispute, and they can not be given back." -Unknown poet; Uruk, Sumeria, circa 3000BC. (From the epic of Gilgamesh, transcribed by Sin-leqiuninni around 1300BC, interpreted by the anthropologist, Michael Wood.) The Gilgamesh is the tale of the invention of civilization by the ancient Sumerians, somewhat before the fourth millenia, BC. Interestingly, it contains a theme that will be echoed by poets through the ages when describing the enigma of civilization-the formula for a fractal. As with any application of abstraction, the quality of the abstraction has to be evaluated. Historically, at least in mathematics, this was done by determining the scalability of a theory-in short, how far can its precision be extrapolated. In the more robust theories, for example the quantum mechanics, scalability is preserved over many orders of magnitude. Fractal analysis is usually used to describe the evolution of a complex system that operates under uncertainty. Equity values over time, industrial markets, and complex (eg., iterated game-theoretic,) economic phenomena are examples. An argument can be made that if equity values are a complex system that operates under uncertainty, and so are industrial markets, then it would be reasonable to assume that the GDP, also, is a complex system that operates under uncertainty. This argument is based on the fact that fractals, which describe such things, are self-similar. (What this means is that a fractal is made up of smaller fractals, which in turn, are made up of even smaller fractals, and so on, without end.) And, if that is true, then we should be able to argue that the same is true for all the GDP's in the history of civilization. Such an extrapolation should be viewed with scepticism. Although we have very precise data for the financial markets in this century, (in point of fact, the US equity market tickers are some of the most precise data we have all of science,) we don't really have good data on the ecology/GDP of the cultural history-and the proposal is to make a "leap of faith" over nine orders of magnitude! (That is, see if we can present arguments that relate the US equity market ticker, by the second, to the history of civilization, by the century.) It should be regarded as sloppy science to do things like this, but since I have received several questions on the subject, I'll attempt it-on the qualification that the analysis be viewed with a significant degree of scepticism. Here is the proposal. We need to take the data from the US Equity markets, by the second, and extrapolate that data to by the day, and then analyze the fractal probabilities of that data. Then, we need to take the US GDP data, by the year, and extrapolate that data to by the day, and do likewise. Then we need to look at the historical record of cultures/civilizations, and scheme up a way of scaling that data, by the day. We need a metric to compare how accurate our extrapolated probabilities are. For this, I propose that we use the Shannon probability, based on the day as a time unit. (The Shannon probability is the simply likelihood that the value for the next day in a data set will increase.) The reason I propose this is because we do not have accurate data on the GDP 's of cultures in the historical record-but we do have reasonably good data on how long cultures persisted, (ie., the run length of cultures; the ancient Sumerians were pretty meticulous with their accounting, as were the Egyptians, the Romans, the Greeks, the Mayans in Central America, the Chinese Dynasties, as were the European countries since the Renaissance, etc.) Granted, the historical perspective is subject to interpretation, but its the only data we have. We ask only that the interpretations be axiomatically consistent. Here is the strategy. We will look at the distribution of run-lengths, (ie., how long cultures/civilizations persisted,) and see if it fits a known distribution, within reason. If the distribution assembles itself into a reciprocal of the square root of time, (time being measured in centuries,) then this will give some, but not a lot, of credibility that civilization/culture is a fractal system that operates under uncertainty. Since the US is in its second century, we can use the probability that the run-length will continue in the second century to compute the Shannon probability for the entire century, extrapolate it to the day, and compare it with the extrapolate data for the US equity markets and GDP, for this century. The argument here is that if civilizations/cultures are fractal systems, then they should start, and for the most part, grow for two centuries, on average, then decay back to zero for two centuries, on average. During the first two centuries, since the GDP/ecology is expanding/growing, the Shannon probability will be greater than one half. In the last two centuries, on average, the Shannon probability will decrease, and be less than one half-being exactly 0.5, on average, over the a average of four centuries. Do we see a distribution of the run lengths of civilizations that is a reciprocal of the square root of time? The data set is limited, (about 400 samples, far too small for analytical comfort,) but we do see something that appears to be similar to what we expected, (The ancient Egyptians won the duration lottery-about 30 centuries-which would have a theoretical probability of about 18.3%. Note that we should have found more, but our data set only extends only over a time interval of 60 centuries-about twice the duration of the ancient Egyptian civilization.) The Romans were next, at 8 centuries, with most lasting one century, or less. The average is about 3.6 centuries-which is close to the 4 we expected. (Note, this type of distribution could not be made by a system that operated on simpler statistical mechanisms; a pure Gaussian distribution doesn't fit, since virtually all of the distribution would be beyond the left three sigma limit of the curve, instead of at 4 centuries; a random mechanism would have given a "splatter" that have produced an equal probability of a civilization lasting one century, or thirty. Not compelling arguments, but interesting, nonetheless.) So, it would appear that, although our data set size is too small to justify any optimism, that, at least there is some, limited, circumstantial evidence that culture/civilization evolves as a complex system that operates under uncertainty. Performing our "leap of faith," and calculating the Shannon probability for the ecological growth in second century of a "typical" culture/civilization, by assuming that the average growth in the second century, will be reflected, on average, to all the constituent fractals that make up the cultural/civilization fractal: G(t) = sqrt (t) where t is time, measured in centuries-and taking the derivative with respect to time to get the marginal growth, M: dG(t) 1 ----- = ------------ dt 2 * sqrt (t) and calculating this value at 1.5 centuries, (since we will be comparing this value to the value for equity markets obtained in the last half of the second century in the US,) and 2 centuries: dG(t) 1 ----- = -------------- = 0.4082483 dt 2 * sqrt (1.5) dG(t) 1 ----- = ------------ = 0.3535534 dt 2 * sqrt (2) and average the two to get M = 0.3819008. This means that there should have been a 38% growth, on the average, in the second century of cultures that lasted longer than two centuries. I now need to solve for the Shannon probability, by the century, that would give this growth, ie., solve the equation: ln (1 + M) = 1 + P ln (P) + (1 - P) ln (1 - P) 2 2 2 which gives P = 0.87859, for the second century of an average culture/civilization. I now need to convert this value from a century value, to a daily value. Knowing that: R + 1 P = ----- 2 where R is the root mean square of the century fluctuations: 0.75718 + 1 P = ----------- 2 and, using since the daily fluctuations add root mean square to get the century fluctuations, and there are 25,700 business days in a century, (because I will be comparing it to equity data, taken by the business day,) to get the root mean square of the daily fluctuations, r: r = 0.75718 / sqrt (25700) = 0.004723159 and the daily Shannon probability, p, would be: r + 1 p = ----- = 0.5023616 2 For the analysis of the US GDP, the data came from http://www.doc.gov/BudgetFY97/index.html, and is by year, 1940 to 1995, inclusive. Again, the data set is far too small. There is an issue since the equity market numbers are not adjusted for inflation, (and it is not clear what term "duration of civilization, adjusted for inflation" would mean,) so I will derive the fractal statistics for the US GDP in both real and non-adjusted dollars. The average of the marginal increments of the US GDP is 0.082307, and the root mean square of the marginal increments is 0.095762. From this, the non-adjusted Shannon probability, by year, can be calculated: 0.082307 -------- + 1 0.095762 P = ------------ = 0.9297477 2 and, since the daily fluctuations add root mean square to get the annual fluctuations, and there are 257 business days in a year, (because I will be comparing it to equity data, taken by the business day) I need to calculate the Shannon probability of the non-inflation adjusted US GDP by day, p: 0.082307 -------- 0.095762 + 1 ---------- sqrt (257) 0.05799607 + 1 p = -------------- = -------------- = 0.5268069 2 2 Since there are no reliable inflation numbers, (including the CPI,) by the year, I will use the deflation numbers provided by the CBO, (which are a running exponential,) and strike a curve through the inflation adjusted GDP, calculate the growth, to derive the Shannon probability for the inflation adjusted US GDP. In 1940 the inflation adjusted US GDP was 831.735 billion. In 1995, it was 5,438.699 billion-a factor of growth of 6.53898 in 55 years. Solving: 55 (x) = 6.53898 or: 55 * ln (x) = ln (6.53898) and: ln (6.53898) ------------ 55 x = e = 1.034731 and solving the equation: ln (x) = 1 + P ln (P) + (1 - P) ln (1 - P) 2 2 2 to get the Shannon probability of the inflation adjusted US GDP, by the year, which is P = 0.6299. Again, using the fact that the daily fluctuations add root mean square to get the annual fluctuations, and there are 257 business days in a year, (because I will be comparing it to equity data, taken by the business day) I need to calculate the Shannon probability of the inflation adjusted US GDP by day, p: R + 1 P = 0.6299 = ----- 2 where R is the root mean square of the annual fluctuations, or: R = 0.2598 and dividing R by the square root of 257 to get the root mean square of the daily fluctuations in the inflation adjusted US GDP, r: r = 0.01620588 and the daily Shannon probability, p, would be: r + 1 0.01620588 + 1 p = ----- = -------------- = 0.5081029 2 2 For the analysis of the US equity markets, the data came from the NYSE, by second, and was combined for all stocks, integrated and sampled to directly convert the second ticker data to daily ticker data, for the inclusive annual time interval, 1966 to 1996. The average of the marginal increments of the NYSE is 0.000290, and the 0.00029 ------- + 1 0.00851 0.03407756 + 1 p = ----------- = -------------- = 0.5170388 2 2 Here is how the Shannon probabilities, extrapolated over nine orders of magnitude, compare: non-inflation NYSE inflation adjusted duration of adjusted US GDP US GDP civilizations 0.5268069 0.5170388 0.5081029 0.5023616 or about 0.5135776 +/- 2.33% Which, depending on one's point of view, are reasonably close. We now have to ask the question, "how reliable are these probabilities?" It turns out that traditional statistical estimation techniques let us down in fractal analysis, (we are hunting for the probability of a probability that controls a system's evolution over a time interval.) The assumption was that we were investigating the evolution of a complex system that operates under uncertainty-ie., a fractal. But there are whole families of fractals, and the paradigm assumption was that we were dealing with the simplest fractal of all, fractional Brownian motion. How valid was that assumption? What is the chance we were misled and it is really some other kind of fractal? These are complicated questions, and depend on the magnitude of the probabilities under analysis. The analysis of a system that is controlled by larger Shannon probabilities can get by with smaller data sets. Smaller Shannon probabilities require larger data set sizes, (for example, the analysis of fractional Brownian motion with a Shannon probability of exactly 50% requires an infinite data set size.) It is not trivial to develop a methodology to evaluate how reliable our analysis is, but it can be done. Without beleaguering the issue, here is how are reliability stacks up for the metrics used in this analysis, where S is the data set size, and P is the measured Shannon probability: non-inflation NYSE inflation adjusted duration of adjusted US GDP US GDP civilizations S = 55 S = 46,260,00 S = 55 S = 1600 P = 0.9297477 P = 0.5170388 P = 0.6299 P = 0.87854 P' = 0.791092 P' = 0.515843 P' = 0.530797 P' = 0.841083 x = 85.08674% x = 99.76872% x = 84.26687% x = 95.73645% Here, P' is the Shannon probability, after being adjusted for a finite data set size, and x is the ratio of P' to P. Not surprisingly, the mass of data provided by the NYSE provides the most reliable probability. Also, not surprisingly, the GDP data is to be considered the most unreliable, with the historical data being somewhere in between, (don't forget that we still have to qualify the historical data as being subject to interpretation.) Our reliability estimate on these probabilities can only be regarded as poor to fair, (which was a qualification that was made in the beginning of the analysis.) However, we can state that there is some evidence, albeit circumstantial, that we can extrapolate, or scale, fractal analysis over nine orders of magnitude in the study of ecology. The evidence, however, can not be regarded as compelling. John -- John Conover, firstname.lastname@example.org, http://www.johncon.com/