Re: Poll on market efficiency

From: John Conover <>
Subject: Re: Poll on market efficiency
Date: 5 Aug 2000 03:41:26 -0000

Hi Jim. Regarding my reply of yesterday regarding your question on
how/why entropy and market efficiency are related, I got some free
time today, and did a simulation that might lend some insight.

The simulation and analytical tools are Open Source, (from and,)
so you can compile them, and verify the results. For the sake of
simplicity, only the EMH model was used in the simulation.

Here is what I did:

    1) Using the tsinvestsim[1] program, with no persistence, and a
    51% chance of moving up on any given day, (and a 49% of moving
    down,) with a daily volatility of 2%, (I choose these values
    because they are "typical" of stocks on the US exchanges,) for a
    million days, yielded a stock price time series, that measured an
    average gain per day of 0.04%, a volatility of 2%, which
    translates to a daily gain of 0.02% per day. The stock has maximal
    gain, and lowering OR raising the risk, (risk = volatility,)
    results in less growth in price. This is a baseline stock for
    comparison[2]. There was no persistence, as measured by the
    tsrootmeanscale program[3].

    2) By using the tsinvest[4] program, it can be verified that it
    can do no better than the original time series in the growth of a
    stock's value over time.

    3) By iterating the tsinvestsim program parameters, but with
    persistence[5], to find new values that give identical statistics
    to 1), (above, and verifying as such, as in [1], [2], and [3],)
    the values are verified as being the same, for all practical
    purposes, (and the persistence, as measured by the tsrootmeanscale
    program,) is, as specified.

    4) But by using the tsinvest program, as before, it can be seen
    that it can do substantially better than the original time series
    in the growth of a stock's value over time.  Using the same
    proceedure as in 1) and 2), the output time series from the
    tsinvest program has an average gain per day of 0.24%, and a
    volatility of 2%, which translates to a daily gain of 0.2% per
    day[6]. Additionally, there was NO persistence in the output time
    series as measured by the tsrootmeanscale program, (i.e., the
    entropy has been raised to infinity, meaning it is 100%
    efficient, at least in the sense of the EMH.)

If we define efficiency as the ratio of annual gains, then, (with 253
trading days per year,) the inefficiency of the original time series
would be 1.0002^253 / 1.002^253 = 1.052 / 1.66 = 63%.

The 5% persistence, (e.g., lowering the time series entropy,) also
lowered the stock price efficiency by about a third, (in this very
simple case, assuming the paradigm of the EMH, which may or may not be
representative of real market conditions.)


[1] The tsinvestsim program is part of the tsinvest program suite from It uses a random number generator,
(the one supplied in the Linux 2.2.12 kernel, which is not the best,
but adequate for the needs here.) All the program does is produce a
time series of a stock's price by implementing the EMH, which, in a
nutshell, is a compound interest algorithm-with variable interest
rates, (which can be negative, or positive, depending on price
movements,) but must be a random process that has a Gaussian/Normal
distribution, (for consistency with the paradigm of the EMH.) The mode
or mean of the distribution can be offset such that there is a long
term exponential gain in a stock's price, which seems consistent with
the empiricals of stocks on the US exchanges over the last
century. Persistence can be included in the simulation by adding the
last movement in the time series, multiplied by an appropriate value,
such that when added to the current movement, there is a better than
50/50 chance that the resulting current movement will be the same as
the last. Bottom line, the program is a stock price simulator, with
EMH characteristics, and allows the variables, (volatility,
exponential growth, persistence, etc.,) to be altered.

[2] The record used for tsinvestsim was "my_name, p = 0.51", and the
characteristics were measured by using the tsfraction program, (which
just subtracts the current value of the stock, from the last value of
the stock, and divides this by the last value, ie., it is the marginal
revenue per day-which is the exact opposite of the way tsinvestsim
constructed the times series,) and the output of the tsfraction
program piped to both tsavg, and tsrms, which computes the average,
and root mean square of the marginal returns.

[3] The tsrootmeanscale program is not sophisticated-it just catalogs
like sequential movements in a time series, (many use an FFT to do the
same thing; the program's only redeeming value is that it is
conceptually simple.) All it does is, when it gets two like movements,
it increments the value in the first element of an array-three, it
increments the second, and so on. It then divides the values in the
array by the total increments. If the time series under consideration
is a Brownian motion fractal, then the sequence in the array would be
0.5, 0.5, 0.5 ..., but if there is a persistence, of say a 55% chance
of what happened the previous day happening again today, then the
series would be 0.55, 0.55, 0.55 ...; and if there is short term
persistence, (i.e., Markovian,) then it would look something like
0.55, 0.53, 0.51, 0.50. (Again, 0.55 was chosen because it is
"typical" of stocks in the US exchanges.)

[4] The tsinvest program is just a monolithic program that compiles
the statistics as in [1], [2], [3], and [5], on a stock price time
series, attempts to time the market, (there are other options, but for
the sake here, only the -d5 option is used-which instructs the program
to attempt to time the market based on persistence measurements, using
the EMH paradigm.)

[5] The record used for tsinvestsim was "my_name, , p = 0.5088, f =
0.02, h = 0.55", which gives a 55% chance of what happened today
happening again tomorrow, the same volatility, and the same gain, as
in 1). The persistence is short term Markovian, which will be shown to
be inefficient, even though the measured characteristics, (other than
the persistence,) are are not statistically different than in 1),
above, (I chose them to be exact, but there are round off, and data
set size considerations-even at a million records, that get in the

[6] The program was only invested for about half the time, and the
coprocessor overflowed at record 812,983-it was invested in 507,488 of
those days. What I needed was a very large data set size, (see the
tsshannoneffective program as to how large these data sets have to
be,) and so what I did was pick a value-a million-records that I knew
would overflow the coprocessor late in the simulation, and then stop
the simulation the record before. The time series from the tsinvest
program has "holes" in it when it was not invested, so I removed those
records; otherwise, I would have about a third of the records with
zero gain, and zero risk, distorting the statistics. Removing the
records has about the same effect as giving the program many more
stocks to chose from, that are all identical.

jim blair writes:
> <> wrote in message
> > FWIW, efficiency in the sense of the EMH, (depending on who is telling
> > the story, of course,) gets larger as the system's entropy gets
> > larger, (the EMH assumes infinite entropy, i.e., 100% efficiency, as a
> > first order approximation; .....etc.
> I have thought over your clam of a relationship between "efficiency" and
> entropy (=disorder), but fail to see any connection.  If anything, I would
> expect a reverse relationship (if any).  Greater order connected to greater
> efficiency?
> At any rate, one problem with entropy is the need to define a "zero level".
> That is, only changes in entropy can be measured.
> In chemistry, the 3rd Law establishes (defines) the zero level of entropy.
> But what is it in economics/market/or whatever it is we are trying to relate
> to?


John Conover,,

Copyright © 2000 John Conover, All Rights Reserved.
Last modified: Fri Aug 4 21:42:38 PDT 2000 $Id: 000804213922.4718.html,v 1.0 2001/11/17 23:05:50 conover Exp $
Valid HTML 4.0!