we don't know enough about the subject to even know if there's
an error, much less gently point it out.

In article <n1uqp3pb156lfjetp2dvtap1djc7k99k79@4ax.com>,
Hatunen <hatunen@cox.net> wrote:

we don't know enough about the subject to even know if there's
an error, much less gently point it out.

Why do you assume everybody here is as ignorant as you are?

rog...@lpbroadband.net writes:
Hi all;

I'm looking for input on this.

A number of predictors use bullseye predictive schemes, notably JOB
and Petra. This involves partial credit for quakes which fall outside
the primary prediction parameters by specified amounts.

OK, I haven't seen the form of these predictions, so that makes concrete
suggestions a bit tricky. Could you perhaps post some examples?

I've tried several ways of dealing with this. I have insisted on
perfect hits with Petra and I've used multiple probabilities with JOB
but neither method satisfies anyone. It's difficult to deal with
because accurate probability is essential to determining significance
but dealing with 4 ranges with 3 parameters is a LOT of possible
answers.

From this it sounds like they want variation in all possible parameters?
That sounds like you should give up and require exact predictions. The
other solutions, given the combinatorics don't seem very palatable.

No, it's quite burdensome.

I'm thinking that the proper way to handle this might be to consider
such a prediction to be multiple predictions, one for each possible
combination, since the predictor isn't specifying which combination to
expect. That would be devastating to their significance, as even a
perfect hit would also involve multiple misses as well. But it's
mathematically sound as far as I can tell.

Well, then perhaps that is the proper method. One would expect having a
wider prediction in time, space and size would devastate any expected
significance. It seems to be having the desired effect.

Although, the perfect hit causes "misses" seems to be at least
intuitively distasteful. Do the predictors specify how much partial
credit they want for events outside the range? Do they want 1/2 credit
if the magitude is off by no more than 1.0?

JOB wants a 10% reduction for each of 3 ranges which are 10% larger.

If so, then perhaps a possible methodology would be to compute some
awful combined probability. Using a simplified example, for a
prediction in area A, during time window T, with magnitude in the range
[M_l, M_h]:

Compute probability P_exact using those parameters.
Compute probability P_near using A, T, [M_l - 1, M_h + 1].

Combine the probabilities by using P_exact + W*(P_near - P_exact) where
W is the weight assigned to a near miss. The number in parentheses
would be the excess probability added by relaxing the boundaries.

Of course, this would have to be done over all combinations, so the
combinatorics of doing this start to look a bit horrendous, but perhaps
not any more so than if you treat them as multiple independent
hypotheses.

It amounts to 27 different possibilities for JOB to be calculated for

In any case, since the particular dimension on which a near miss might
be claimed is not specified in advance, perhaps it would be enough to
just perform two computations:
1. The exact hit probability
2. The fully relaxed hit probability

That seems simple and also appealing, because it encapsulates the exact
and the approximate parts of the relationship. Again, a weighting
factor can be applied based on how much credit is claimed for near
misses.

The best solution is to make only specific predictions and accept only
perfect hits but that's not something I can control. It's up to the
predictors to quit wanting credit for "near enough" results.
Especially when "near enough" is quite a way away.

I agree with the best solution.

On the other hand, perhaps showing the devastating effect on
significance of including the near-misses may be what is needed to
demonstrate why it isn't a good idea to do the experiment that way.

Predictors are generally impervious to constructive criticism, which

Any opinions?

I think the grasping for near misses to be included is just a sign of
the self-delusion that the scientific method, controlled experimentation
and the search for statistical significance is supposed to counteract.
By allowing ex-post-facto excuses or inclusions, one is just encouraging
unscientific behavior, and it is perhaps wisest (although perhaps
quixotic) to fight against that.

Lord knows I've tried!

Where near misses should figure in, is in the process of guiding
predictors to try to refine whatever methods they have for generating
the predictions. They should not be viewed as a way of getting credit
for misses, but rather as learning experiences from which to try to
improve in the next round of experimentation.

Agreed. I wrote an evaluation program called BESTFIT which finds the 3

Which actually brings up the next question: When the prediction method
changes, one really should start an entirely new set of experiments to
determine significance. I'm not sure that is being done.

No, it isn't.

--
Thomas A. Russ, USC/Information Sciences Institute

Why shouldn't I?

Yes it is. That's why I need good probability numbers. But my problem
is with people who want partial credit for a 5 when they predict a 6
to 7.

Hi all;

I'm looking for input on this.

A number of predictors use bullseye predictive schemes, notably JOB
and Petra. This involves partial credit for quakes which fall outside
the primary prediction parameters by specified amounts.

I've tried several ways of dealing with this. I have insisted on
perfect hits with Petra and I've used multiple probabilities with JOB
but neither method satisfies anyone. It's difficult to deal with
because accurate probability is essential to determining significance
but dealing with 4 ranges with 3 parameters is a LOT of possible
answers.

I'm thinking that the proper way to handle this might be to consider
such a prediction to be multiple predictions, one for each possible
combination, since the predictor isn't specifying which combination to
expect. That would be devastating to their significance, as even a
perfect hit would also involve multiple misses as well. But it's
mathematically sound as far as I can tell.

The best solution is to make only specific predictions and accept only
perfect hits but that's not something I can control. It's up to the
predictors to quit wanting credit for "near enough" results.
Especially when "near enough" is quite a way away.

Any opinions?

On Mon, 28 Jan 2008 18:52:58 -0000, ellis@no.spam () wrote:

In article <n1uqp3pb156lfjetp2dvtap1djc7k99k79@4ax.com>,
Hatunen <hatunen@cox.net> wrote:

we don't know enough about the subject to even know if there's
an error, much less gently point it out.

Why do you assume everybody here is as ignorant as you are?

Why shouldn't I?

rogerh@lpbroadband.net writes:

Yes it is. That's why I need good probability numbers. But my problem
is with people who want partial credit for a 5 when they predict a 6
to 7.

Well, unfortunately, I don't think you can come up with anything
reasonable that could handle such ex-post facto claims. They should
just be treated as misses. Especially since those partial credit scores
are only claimed where there is a near miss -- in other words, quite
selectively.

Why not allow amateur predictors the same latitude? Still require time,
place, and magnitude but add the requirement of confidence intervals. So
a range would be predicted as M5±.5, 90%, ±1 10% If an earthquake occurs
meeting the other two criteria and is M5, that gives a 90% score for
magnitude ... but if it happened with M4.4 or M5.6, that would give only
a 10% score.

IOW, draw a bullseye around the location and draw a two-dimensional
bullseye around the time and magnitude, and score for accuracy. Thus "I
was close" gets 1%.

And, of course, the confidence intervals would have to be tight enough
to clearly invalidate a miss that was no better than chance. Perhaps
even assign negative points for hits claimed that are too damned far
away. ("Well, it happened a week later, on the other side of the planet,
so I claim a hit!" would lose ten points.)

--

Timberwoof <me at timberwoof dot com>http://www.timberwoof.com
"When you post sewage, don't blame others for
emptying chamber pots in your direction." ‹Chris L.

Hatunen <hatu...@cox.net> wrote in news:vu5vp3t4a4rfvehuph691cr7mr1dqev7vl@
4ax.com:

Well, of course. But close only counts in horseshoes and
handgrenades, as they say.

And nuclear war? I've always added that one to the saying.

What's needed is a clear explanation of the predictor's method,
though, before any further critique can be made.

Agreed. Like I said, both predictor and evaluator must be on
the same game table playing by the same rules.

Brian
--http://www.skywise711.com- Lasers, Seismology, Astronomy, Skepticism
Seismic FAQ:http://www.skywise711.com/SeismicFAQ/SeismicFAQ.html
Quake "predictions":http://www.skywise711.com/quakes/EQDB/index.html
Sed quis custodiet ipsos Custodes?

tar@sevak.isi.edu (Thomas A. Russ) wrote in
news:ymi4pcxxxrv.fsf@blackcat.isi.edu:

rogerh@lpbroadband.net writes:

Yes it is. That's why I need good probability numbers. But my problem
is with people who want partial credit for a 5 when they predict a 6
to 7.

Well, unfortunately, I don't think you can come up with anything
reasonable that could handle such ex-post facto claims. They should
just be treated as misses. Especially since those partial credit scores
are only claimed where there is a near miss -- in other words, quite
selectively.

Good point. The method of scoring "near hits" has to be agreed
upon in advance. Since we know this won't happen, it's almost
a moot point.

I have in the past given thought to a way of scoring close calls
and in all my ideas, both parties woudl have to agree to the
method.

We can't even get them to accept black & white hit & miss. Grey
zones would be like arguing politics with a 1 year old.

Simply make a prediction as to range, like "between 6.0 and 7.0",
or even "6.0 or greater" and then ruthlessly exclude anything
outside the range.

Hatunen <hatunen@cox.net> wrote in
news:bppup39asup0fnqefnfsnq8p60hpf2gjno@4ax.com:

Simply make a prediction as to range, like "between 6.0 and 7.0",
or even "6.0 or greater" and then ruthlessly exclude anything
outside the range.

And what happens when a M5.9 or M7.1 quake strikes within the time
and location of the perdiction?

Hatunen <hatunen@cox.net> wrote in
news:bppup39asup0fnqefnfsnq8p60hpf2gjno@4ax.com:

Simply make a prediction as to range, like "between 6.0 and 7.0",
or even "6.0 or greater" and then ruthlessly exclude anything
outside the range.

And what happens when a M5.9 or M7.1 quake strikes within the time
and location of the perdiction?

In article <13pur86stknrka9@corp.supernews.com>,
Skywise <into@oblivion.nothing.com> wrote:

Hatunen <hatunen@cox.net> wrote in
news:bppup39asup0fnqefnfsnq8p60hpf2gjno@4ax.com:

Simply make a prediction as to range, like "between 6.0 and 7.0",
or even "6.0 or greater" and then ruthlessly exclude anything
outside the range.

And what happens when a M5.9 or M7.1 quake strikes within the time
and location of the perdiction?

When rocket scientists present the expected landing areas of their space
craft, they give them in terms of probabilities: 90% within this
ellipse, 99% within this larger one, 99.99% on the planet somewhere...
Likewise, when astronomers predict impacts of asteroids on Mars, they
give likelihoods within target areas.