From: John Conover <john@email.johncon.com>

Subject: Are the US Equity Markets Under Valued?

Date: 19 Aug 2002 08:39:21 -0000

Determination of the value of equity markets is difficult. But we
can gain some insight into the issues using the techniques offered in
the sections Quantitative
Analysis of Non-Linear High Entropy Economic Systems II and Quantitative
Analysis of Non-Linear High Entropy Economic Systems III, from the
Mathematical
Analysis & Numerical Methods series at the NdustriX
site. In the following analysis, the historical time series of the
DJIA, NASDAQ, and S&P500 indices through July 26, 2002 was
obtained from Yahoo!'s database of
equity Historical Prices,
(ticker symbols *^DJI*, *^IXIC*, and, *^SPC*,
respectively,) in *csv* format-which was converted to a Unix
database format using the

program. (The converted DJIA time series,
*csv2tsinvest*

, started on
January 2, 1900, the NASDAQ,
*djia1900-2002*

, on October 11,
1984, and January 3, 1928 for the S&P500, file name,
*nasdaq1984-2002*

.)*sp1928-2002*

The strategy of the analysis will be to convert the indices' time series into their Brownian motion/random walk fractal equivalents, and work with the Brownian motion statistics-since normal/Gaussian frequency distributions, (which are characteristic of Brownian motion,) are much more expedient to work with than the indices' characteristic log-normal frequency distributions. A short analysis of the limited precision available, (do to data set sizes,) of the statistics will be offered, followed by a Kalman filtering technique that will provide a methodology for the best possible accuracy for the current value of the markets, and statistics on how good the best possible accuracy is, which will be demonstrated in the conclusion.

Using standard statistical estimation techniques, (with a confidence level of at least one standard deviation, and an accuracy to within one standard deviation):

**
***tsfraction* *djia1900-2002* | *tsstatest* -c 0.841344746068542954 -e 0.00003680801891209803
For a mean of 0.000232, with a confidence level of 0.841345
that the error did not exceed 0.000037, 177081 samples would be required.
(With 28047 samples, the estimated error is 0.000092 = 39.899222 percent.)
For a standard deviation of 0.010988, with a confidence level of 0.841345
that the error did not exceed 0.000037, 88541 samples would be required.
(With 28047 samples, the estimated error is 0.000065 = 0.595169 percent.)
*tsfraction* *sp1928-2002* | *tsstatest* -c 0.841344746068542954 -e 0.00004125036602217883
For a mean of 0.000260, with a confidence level of 0.841345
that the error did not exceed 0.000041, 149462 samples would be required.
(With 19802 samples, the estimated error is 0.000113 = 43.652147 percent.)
For a standard deviation of 0.011313, with a confidence level of 0.841345
that the error did not exceed 0.000041, 74731 samples would be required.
(With 19802 samples, the estimated error is 0.000080 = 0.708319 percent.)
*tsfraction* *nasdaq1984-2002* | *tsstatest* -c 0.841344746068542954 -e 0.00007393334833205898
For a mean of 0.000466, with a confidence level of 0.841345
that the error did not exceed 0.000074, 73223 samples would be required.
(With 4489 samples, the estimated error is 0.000299 = 64.043947 percent.)
For a standard deviation of 0.014193, with a confidence level of 0.841345
that the error did not exceed 0.000074, 36612 samples would be required.
(With 4489 samples, the estimated error is 0.000211 = 1.487677 percent.)

Where the expected accuracy of the average,

, and deviation,
**avg**

, of the marginal increments of the
indices' value, (for example, the DJIA's marginal increments are,
**rms**

and **avg = 0.000232**

,) are shown. These two values are metrics
that determine the daily increase in value of the indices, (from the
Important
Formulas section of Quantitative
Analysis of Non-Linear High Entropy Economic Systems I,) and,
obviously, there is a problem with using the gain in an indices' value
as a metric-for example, although the DJIA has 28048 samples
represented, (over a century of daily closes,) the accuracy of the
measurement of **rms =
0.010988**

is only within 40% of
the measured value for one standard deviation of the time!**avg**

Bottom line, any methodology that depends on measuring the increase
in gain of the marginal increments, (and thus

,) in an indices' value will lead to
inconsistent results-we can not find the median value of the indices
using any method that depends on **avg**

using the limited data sets at hand, (and this goes for intuition and
interpretation of graphs, too.) However, that the deviation of the
marginal increments, **avg**

, converges to
respectable accuracies, quite quickly-which is exploitable.**rms**

Using the methodology described in Quantitative
Analysis of Non-Linear High Entropy Economic Systems III, and, Equation
(3.1), where it is shown that the accuracy of the metrics of the
median value, (which is the fair market value at the end of the time
series,) of the Brownian motion/random walk fractal equivalent of an
index will converge with a normal/Gaussian frequency distribution of
the error in the median value, (the convergence-the deviation of the
frequency distribution of the error-is, actually,

, for a data set size of
**rms *
erf (1 / sqrt (n))**

, which is about **n**

for **rms * (1
/ sqrt (n))**

, so we are working with the
**n >>
1**

, instead of
**rms**

). Larger data sets will have a
smaller deviation around the mean, (which is the fair market value at
the end of the data set,) and smaller data sets will have a larger
deviation. It is the median value, and the distribution of its
measurement error, measured for all possible data set sizes that we
want to work with.**avg**

The following Unix Shell script, although not efficient, will suffice:

**
#!/bin/sh
#
rm -f ***djia.jan.02.1900-jul.26.2002.histogram* \
*sp.jan.03.1928-jul.26.2002.histogram* \
*nasdaq.oct.11.1984-jul.26.2002.histogram*
#
i=28048
#
while [ ${i} -gt 1 ]
do
tail -$i *djia1900-2002* | *tsmath* -l | *tslsq* -o | \
tail -1 >> *djia.jan.02.1900-jul.26.2002.histogram*
i=`expr $i - 1`
done
#
i=19308
#
while [ ${i} -gt 1 ]
do
tail -$i *sp1928-2002* | *tsmath* -l | *tslsq* -o | \
tail -1 >> *sp.jan.03.1928-jul.26.2002.histogram*
i=`expr $i - 1`
done
#
i=4490
#
while [ ${i} -gt 1 ]
do
tail -$i *nasdaq1984-2002* | *tsmath* -l | *tslsq* -o | \
tail -1 >> *nasdaq.oct.11.1984-jul.26.2002.histogram*
i=`expr $i - 1`
done

Where the

construct converts the log-normal characteristics
of the indices to their Brownian motion/random walk fractal
equivalents-there were 28048 trading days represented in the DJIA time
series, 19308 in the S&P, and, 4490 in the NASDAQ, and the script
*tsmath*
-l | *tslsq*
-o*walks through* each time series, calculating the least squares
fit to the median, (fair market value,) for

, **n**

,
**n - 1**

... **n - 2**

elements in the time series,
saving the final value of the median, (fair market value,) for each
iteration.**1**

Figure I is a plot of the measured median, (i.e., fair market,)
value for the DJIA, S&P, and NASDAQ, as a function of data set
size, (the X-Axis was rescaled by subtracting the number of elements
in the time series from each X-Axis value for intuitive presentation,
aligning all graphs on the right, with time extending into the past to
the left.) For example, using a data set for the DJIA that ran from
15,000 days in the past, (January 20, 1944,) to the present, the
median value on July 26, 2002 would be measured as

, or, the DJIA
would be calculated as being over valued by about 32%. Likewise, using
a data set that ran from 5,000 days in the past, (October 1, 1982,) to
July 26, 2002, the DJIA's median value would be measured as
**e^0.279987 = 1.32311261176**

, or under
valued by about 34%. (The reason for the values around 0.8 for the
DJIA near time -30,000, is the Great Depression, where the DJIA lost
about 90% of its value, about a once-in-a millennia event, biasing the
data to smaller growth values for the entire century.)**e^-0.415470 = 0.660029994**

We now want to look at the frequency distribution of the measured median values in Figure I:

**
***tsnormal* -s 17 -t *djia.jan.02.1900-jul.26.2002.histogram* > *djia.normal*
*tsnormal* -s 17 -t -f *djia.jan.02.1900-jul.26.2002.histogram* > *djia.histogram*
*tsnormal* -s 17 -t *sp.jan.03.1928-jul.26.2002.histogram* > *sp.normal*
*tsnormal* -s 17 -t -f *sp.jan.03.1928-jul.26.2002.histogram* > *sp.histogram*
*tsnormal* -s 17 -t *nasdaq.oct.11.1984-jul.26.2002.histogram* > *nasdaq.normal*
*tsnormal* -s 17 -t *nasdaq.oct.11.1984-jul.26.2002.histogram* > *nasdaq.histogram*

And, plotting:

Figure II is a plot of the frequency distribution of the measured median, (i.e., fair market,) values in Figure I, for the DJIA, S&P, and NASDAQ, as a function of data set size-centered on the indices' measured median value on July 26, 2002 for comparison. Calculating the average, (i.e., mean = median,) and deviation:

**
***tsavg* -p *djia.jan.02.1900-jul.26.2002.histogram*
0.199951
*tsrms* -p *djia.jan.02.1900-jul.26.2002.histogram*
0.451294
*tsavg* -p *sp.jan.03.1928-jul.26.2002.histogram*
-0.225497
*tsrms* -p *sp.jan.03.1928-jul.26.2002.histogram*
0.335909
*tsavg* -p *nasdaq.oct.11.1984-jul.26.2002.histogram*
-0.712235
*tsrms* -p *nasdaq.oct.11.1984-jul.26.2002.histogram*
0.775733

(The median of the DJIA is 0.259990, -0.111612 for the S&P, and -0.858068 for the NASDAQ.)

On July 26, 2002, (converting the Brownian motion/random walk fractal equivalents back to their log-normal characteristics by exponentiating):

The DJIA's median value, (i.e., market value,) was

, or about 22% over valued, and the high side deviation was**e^0.199951 = 1.22134291089**

and the low side,**e^0.199951 + 0.451294 = 1.9236895833**

, or the chances are one standard deviation, (about 16%,) that the DJIA was over valued by more than 92%, or under valued by more than 12%.**e^0.199951 - 0.451294 = 0.777789855**The S&P's median value was

, or about 20% under valued, with a high side deviation of**e^-0.225497 = 0.798119455**

and the low side,**e^-0.225497 + 0.335909 = 1.11673807178**

, or the chances are one standard deviation, or about 16%, that the S&P was over valued by at least 12%, or under valued by at least 43%.**e^-0.225497 - 0.335909 = 0.570406508**The NASDAQ's median value was

, or about 51% under valued, with a high side deviation of**e^-0.712235 = 0.4905466**

and the low side,**e^-0.712235 + 0.775733 = 0.063498**

, or the chances are one standard deviation, or about 16%, that the NASDAQ was over valued by at least 6%, or under valued by at least 77%.**e^-0.712235 - 0.775733 = 0.225831078**

So, as for the best assessment that can be made on whether the US
equity markets were under valued on July 26, 2002, the answer is that
they were probably close to fair market value, with both the DJIA and
S&P within about 20% of being fair, meaning their median value,
(the DJIA being slightly over valued, the S&P slightly under
valued, by about 20%,) with the NASDAQ somewhat lower valued than the
S&P at about 50% of fair value, (with the qualification that the
limited data set size of the NASDAQ means that the number was more
like *numerology* than science.)

The DJIA bottomed, during the Great Depression, on July 8, 1932. As it turns out, using this date as the start date in the time series for determining the fair market value of the DJIA and S&P works fairly well as representative of the conclusions, above:

**
tail -18453 ***djia1900-2002* | *tsmath* -l | *tslsq* -o | *tsmath* -e > *djia.jul.08.1932-jul.26.2002*
tail -18464 *sp1928-2002* | *tsmath* -l | *tslsq* -o | *tsmath* -e > *sp.jul.08.1932-jul.26.2002*

and plotting:

Figure III is a plot of the normalized value, (meaning the
exponential curvature has been removed-or detrended,) for the DJIA and
S&P indices, from July 8, 1932, to July 26, 2002. Values greater
than unity mean the market is expanding faster than its median value,
(i.e., in a *bubble*,) and is over valued, and less than unity,
under valued. For example, around trading day 8,000 past July 8, 1932,
(around January 11, 1961,) the DJIA and S&P were running about 20%
over valued. Of interest is the *bubble* from around trading
day 16,500, (about October 19, 1994.) This is the so-called
*dot-com bubble*. Note that it was not necessarily unique in
the last 70 years-the *bubble* coming out of the Great
Depression, (on trading days 0-2,000, or about July 8, 1932 to March
27, 1939,) was almost as significant, (both had magnitudes, where the
market was over valued by a factor of slightly more than two,) and the
market value of the run up prior to the Crash of 1929 was over valued
by almost a factor of two, also.

So, the claim that the *dot-com* era was the greatest wealth
generator in the last century is a myth. It was only one of a few,
(but was not the greatest-the period between trading day 6,500 and
10,000 holds that honor, about January 27, 1955 to January 29, 1969;
the market was only about 20% over valued, but the *bubble*
lasted fourteen years. Typical *bubbles* last only about four
years-as did the *dot-com bubble*.) Of passing interest is the
negative *bubbles* from trading day 2,000 through 6,500 and
11,000 through 15,000, (March 27, 1939 through January 27, 1955 and
January 17, 1973 through November 14, 1988,) where the markets were
about 40% under valued. (The period from trading day 14,500 through
16,000 is interesting, too-the market was at its fair market median
for almost six years, November 21, 1986 though October 28, 1992.)

At any rate, the intuitionist can conjure up a rationale for all
the *bubbles* of the last seventy years, based on the
prevailing dogma, (i.e., depending on who is telling the story.)

Which leads me to a closing remark about such things. Suppose that one was to assume that the data set from October 11, 1984 to July 26, 2002, (just under 18 calendar years,) was adaquate to determine what the US markets were doing:

**
#!/bin/sh
#
tail -18453 ***djia1900-2002* | *tsmath* -l | *tslsq* -o | tail -4490 | *tsmath* -e > *djia.jul.08.1932-jul.26.2002*
tail -18464 *sp1928-2002* | *tsmath* -l | *tslsq* -o | tail -4490 | *tsmath* -e > *sp.jul.08.1932-jul.26.2002*
#
tail -4490 *djia1900-2002* | *tsmath* -l | *tslsq* -o | *tsmath* -e > *djia.oct.11.1984-jul.26.2002*
tail -4490 *sp1928-2002* | *tsmath* -l | *tslsq* -o | *tsmath* -e > *sp.oct.11.1984-jul.26.2002*
tail -4490 *nasdaq1984-2002* | *tsmath* -l | *tslsq* -o | *tsmath* -e > *nasdaq.oct.11.1984-jul.26.2002*

And plotting:

Figure IV is the last 4490 trading days, (i.e., trading days between October 11, 1984 and July 26, 2002,) of Figure III, and for comparison, imposed on graphs made by using the exact same methodology, but using only data sets for the DJIA, S&P, and NASDAQ, of the 4490 trading days, since October 11, 1984. Note how easy it is to jump to the conclusion that the markets are in terrible shape, and under valued by almost 50%.

Its always easy to find statistical significance where there isn't any.

-- John Conover, john@email.johncon.com, http://www.johncon.com/

Copyright © 2002 John Conover, john@email.johncon.com. All Rights Reserved. Last modified: Mon Jul 29 12:29:27 PDT 2002 $Id: 020819134019.29290.html,v 1.0 2004/10/13 07:15:03 conover Exp $