Main Page | Report this Page
 
   
Science Forum Index  »  Geology - Earthquakes Forum  »  Musings on evaluation
Page 1 of 4    Goto page 1, 2, 3, 4  Next
Author Message
Guest
Posted: Fri Jan 25, 2008 9:07 am
Hi all;

I'm looking for input on this.

A number of predictors use bullseye predictive schemes, notably JOB
and Petra. This involves partial credit for quakes which fall outside
the primary prediction parameters by specified amounts.

I've tried several ways of dealing with this. I have insisted on
perfect hits with Petra and I've used multiple probabilities with JOB
but neither method satisfies anyone. It's difficult to deal with
because accurate probability is essential to determining significance
but dealing with 4 ranges with 3 parameters is a LOT of possible
answers.

I'm thinking that the proper way to handle this might be to consider
such a prediction to be multiple predictions, one for each possible
combination, since the predictor isn't specifying which combination to
expect. That would be devastating to their significance, as even a
perfect hit would also involve multiple misses as well. But it's
mathematically sound as far as I can tell.

The best solution is to make only specific predictions and accept only
perfect hits but that's not something I can control. It's up to the
predictors to quit wanting credit for "near enough" results.
Especially when "near enough" is quite a way away.

Any opinions?

Roger
Guest
Posted: Fri Jan 25, 2008 3:36 pm
On Jan 25, 4:59 pm, "Mike Williams" <miklw...@pacbell.net> wrote:
Quote:
"Hatunen" <hatu...@cox.net> wrote in message

news:cdnkp3d7b7jcmvcd9dvauag2h6rqe0b1uk@4ax.com...



On Fri, 25 Jan 2008 11:07:07 -0800 (PST), rog...@lpbroadband.net
wrote:

Hi all;

I'm looking for input on this.

A number of predictors use bullseye predictive schemes, notably JOB
and Petra. This involves partial credit for quakes which fall outside
the primary prediction parameters by specified amounts.

I've tried several ways of dealing with this. I have insisted on
perfect hits with Petra and I've used multiple probabilities with JOB
but neither method satisfies anyone. It's difficult to deal with
because accurate probability is essential to determining significance
but dealing with 4 ranges with 3 parameters is a LOT of possible
answers.

I'm thinking that the proper way to handle this might be to consider
such a prediction to be multiple predictions, one for each possible
combination, since the predictor isn't specifying which combination to
expect. That would be devastating to their significance, as even a
perfect hit would also involve multiple misses as well. But it's
mathematically sound as far as I can tell.

The best solution is to make only specific predictions and accept only
perfect hits but that's not something I can control. It's up to the
predictors to quit wanting credit for "near enough" results.
Especially when "near enough" is quite a way away.

Any opinions?

Yeah. They cover this stuff in university statistics courses. You
ought to try taking some of them.

Unfortunately, statistics courses don't deal with logarithmic
data like earthquake M or Richter sizes so calculating standard
deviations won't work real well. Of course you could change them
to non-logartihmic numbers. That might give a better idea of what
you're actually facing.

--
************* DAVE HATUNEN (hatu...@cox.net) *************
* Tucson Arizona, out where the cacti grow *
* My typos & mispellings are intentional copyright traps *

Dave, I think Roger is well-qualified to be doing the statistical analysis.
If I'm not mistaken, he spent a career doing something like that for the
USGS.

Roger? Could you, once again, re-state your relevant background for us?

Mike Williams
Arroyo Grande, CA U.S.

Mike;

Academically, a BA in Psychologyy but I had started out in Geology so
had enough credits to qualify as a geophysicist with the government.

I spent 34 years, most of it with the Seismology group in first NOAA
and then USGS when they took us over. As part of that I created and
ran the Prediction Monitoring and Evaluation Program which accepted
predictions from all sources and evaluated their success rates. The
project was terminated prematurely so I never got to publish a final
report but after looking at several hundred predictors, none managed
to pass the test.

My current programs are similar to the PMEP program but use better
statistical methods adopted from Dr Alan Jones. I was also able to
persuade him to create the method to be used on predictions with
different probabilities.

Roger
Guest
Posted: Fri Jan 25, 2008 3:39 pm
On Jan 25, 3:10 pm, Hatunen <hatu...@cox.net> wrote:
Quote:
On Fri, 25 Jan 2008 11:07:07 -0800 (PST), rog...@lpbroadband.net
wrote:



Hi all;

I'm looking for input on this.

A number of predictors use bullseye predictive schemes, notably JOB
and Petra. This involves partial credit for quakes which fall outside
the primary prediction parameters by specified amounts.

I've tried several ways of dealing with this. I have insisted on
perfect hits with Petra and I've used multiple probabilities with JOB
but neither method satisfies anyone. It's difficult to deal with
because accurate probability is essential to determining significance
but dealing with 4 ranges with 3 parameters is a LOT of possible
answers.

I'm thinking that the proper way to handle this might be to consider
such a prediction to be multiple predictions, one for each possible
combination, since the predictor isn't specifying which combination to
expect. That would be devastating to their significance, as even a
perfect hit would also involve multiple misses as well. But it's
mathematically sound as far as I can tell.

The best solution is to make only specific predictions and accept only
perfect hits but that's not something I can control. It's up to the
predictors to quit wanting credit for "near enough" results.
Especially when "near enough" is quite a way away.

Any opinions?

Yeah. They cover this stuff in university statistics courses. You
ought to try taking some of them.

Unfortunately, statistics courses don't deal with logarithmic
data like earthquake M or Richter sizes so calculating standard
deviations won't work real well. Of course you could change them
to non-logartihmic numbers. That might give a better idea of what
you're actually facing.

--
************* DAVE HATUNEN (hatu...@cox.net) *************
* Tucson Arizona, out where the cacti grow *
* My typos & mispellings are intentional copyright traps *

dave;

Doesn't matter. The logarithmic nature of magnitude has no bearing on
the analysis which is concerned only with hit or miss results.

Roger
Mike Williams
Posted: Fri Jan 25, 2008 5:06 pm
Guest
<rogerh@lpbroadband.net> wrote in message
news:b1f0def3-2d0f-4eed-86be-551b05145ced@d4g2000prg.googlegroups.com...
Quote:
Hi all;

I'm looking for input on this.

A number of predictors use bullseye predictive schemes, notably JOB
and Petra. This involves partial credit for quakes which fall outside
the primary prediction parameters by specified amounts.

I've tried several ways of dealing with this. I have insisted on
perfect hits with Petra and I've used multiple probabilities with JOB
but neither method satisfies anyone. It's difficult to deal with
because accurate probability is essential to determining significance
but dealing with 4 ranges with 3 parameters is a LOT of possible
answers.

I'm thinking that the proper way to handle this might be to consider
such a prediction to be multiple predictions, one for each possible
combination, since the predictor isn't specifying which combination to
expect. That would be devastating to their significance, as even a
perfect hit would also involve multiple misses as well. But it's
mathematically sound as far as I can tell.

The best solution is to make only specific predictions and accept only
perfect hits but that's not something I can control. It's up to the
predictors to quit wanting credit for "near enough" results.
Especially when "near enough" is quite a way away.

Any opinions?

Roger

First, my opinion is only worth so much. I am not a statistician, but do
have some minor math chops.

As I'm fairly sure you realize, Roger, the requests of the predictors in
this regard are simply a nuisance, and reflect the predictors'
misunderstanding of how probability works (I'm not at all sure J.O.B. has
actually made such a request). In return for the much more complicated
calculations involved, the resulting probabilities are a total wash. Your
efforts to comply with such wishes are admirable. As you point out,
complying with their wishes would increase their number of "hits" with a
corresponding decrease in those hits' net value.

Your "best solution" does, indeed, appear to be the best in my humble
opinion, and does not in any way penalize or work to the disadvantage of the
predictors. OTOH, your proposed solution, of treating the "bull's-eye
predictive schemes" as multiple, separate predictions seems to be a
reasonable one.

Mike Williams
Arroyo Grande, CA U.S.
Guest
Posted: Fri Jan 25, 2008 6:07 pm
On Jan 25, 7:02 pm, Hatunen <hatu...@cox.net> wrote:
Quote:
On Fri, 25 Jan 2008 17:36:28 -0800 (PST), rog...@lpbroadband.net
wrote:



On Jan 25, 4:59 pm, "Mike Williams" <miklw...@pacbell.net> wrote:
"Hatunen" <hatu...@cox.net> wrote in message
Yeah. They cover this stuff in university statistics courses. You
ought to try taking some of them.

Unfortunately, statistics courses don't deal with logarithmic
data like earthquake M or Richter sizes so calculating standard
deviations won't work real well. Of course you could change them
to non-logartihmic numbers. That might give a better idea of what
you're actually facing.

Dave, I think Roger is well-qualified to be doing the statistical analysis.
If I'm not mistaken, he spent a career doing something like that for the
USGS.

Roger? Could you, once again, re-state your relevant background for us?

Mike;

Academically, a BA in Psychologyy but I had started out in Geology so
had enough credits to qualify as a geophysicist with the government.

I spent 34 years, most of it with the Seismology group in first NOAA
and then USGS when they took us over. As part of that I created and
ran the Prediction Monitoring and Evaluation Program which accepted
predictions from all sources and evaluated their success rates. The
project was terminated prematurely so I never got to publish a final
report but after looking at several hundred predictors, none managed
to pass the test.

My current programs are similar to the PMEP program but use better
statistical methods adopted from Dr Alan Jones. I was also able to
persuade him to create the method to be used on predictions with
different probabilities.

Then why are you asking these questions here in this group?
You're obviously the expert.

--
************* DAVE HATUNEN (hatu...@cox.net) *************
* Tucson Arizona, out where the cacti grow *
* My typos & mispellings are intentional copyright traps *

Dave;

Just thinking out loud, so to speak.

Roger
Hatunen
Posted: Fri Jan 25, 2008 6:10 pm
Guest
On Fri, 25 Jan 2008 11:07:07 -0800 (PST), rogerh@lpbroadband.net
wrote:

Quote:
Hi all;

I'm looking for input on this.

A number of predictors use bullseye predictive schemes, notably JOB
and Petra. This involves partial credit for quakes which fall outside
the primary prediction parameters by specified amounts.

I've tried several ways of dealing with this. I have insisted on
perfect hits with Petra and I've used multiple probabilities with JOB
but neither method satisfies anyone. It's difficult to deal with
because accurate probability is essential to determining significance
but dealing with 4 ranges with 3 parameters is a LOT of possible
answers.

I'm thinking that the proper way to handle this might be to consider
such a prediction to be multiple predictions, one for each possible
combination, since the predictor isn't specifying which combination to
expect. That would be devastating to their significance, as even a
perfect hit would also involve multiple misses as well. But it's
mathematically sound as far as I can tell.

The best solution is to make only specific predictions and accept only
perfect hits but that's not something I can control. It's up to the
predictors to quit wanting credit for "near enough" results.
Especially when "near enough" is quite a way away.

Any opinions?

Yeah. They cover this stuff in university statistics courses. You
ought to try taking some of them.

Unfortunately, statistics courses don't deal with logarithmic
data like earthquake M or Richter sizes so calculating standard
deviations won't work real well. Of course you could change them
to non-logartihmic numbers. That might give a better idea of what
you're actually facing.

--
************* DAVE HATUNEN (hatunen@cox.net) *************
* Tucson Arizona, out where the cacti grow *
* My typos & mispellings are intentional copyright traps *
Guest
Posted: Fri Jan 25, 2008 6:13 pm
On Jan 25, 7:33 pm, Hatunen <hatu...@cox.net> wrote:
Quote:
On Fri, 25 Jan 2008 17:39:25 -0800 (PST), rog...@lpbroadband.net
wrote:

Unfortunately, statistics courses don't deal with logarithmic
data like earthquake M or Richter sizes so calculating standard
deviations won't work real well. Of course you could change them
to non-logartihmic numbers. That might give a better idea of what
you're actually facing.

Doesn't matter. The logarithmic nature of magnitude has no bearing on
the analysis which is concerned only with hit or miss results.

It matters if a hit can be an order of maghnitude outside the
main predicted number. To say that a quake of magnitude 7 will
occur and then allowing a quake of magnitude 6 to fulfill the
prophecy is to be deceived by the "small" difference of "only"
one.

You need to discuss that with the predictors. They typically specify
ranges for all parameters.

Quote:

But yes, if you set up very clear hit or miss criteria and don't
fudge them post facto it doesn't much matter. Nevertheless,
predicting a quake between, say, 4.0 and 7.0 is kind of silly.


Yes it is. That's why I need good probability numbers. But my problem
is with people who want partial credit for a 5 when they predict a 6
to 7.

Roger
Quote:

--
************* DAVE HATUNEN (hatu...@cox.net) *************
* Tucson Arizona, out where the cacti grow *
* My typos & mispellings are intentional copyright traps *
Mike Williams
Posted: Fri Jan 25, 2008 7:59 pm
Guest
"Hatunen" <hatunen@cox.net> wrote in message
news:cdnkp3d7b7jcmvcd9dvauag2h6rqe0b1uk@4ax.com...
Quote:
On Fri, 25 Jan 2008 11:07:07 -0800 (PST), rogerh@lpbroadband.net
wrote:

Hi all;

I'm looking for input on this.

A number of predictors use bullseye predictive schemes, notably JOB
and Petra. This involves partial credit for quakes which fall outside
the primary prediction parameters by specified amounts.

I've tried several ways of dealing with this. I have insisted on
perfect hits with Petra and I've used multiple probabilities with JOB
but neither method satisfies anyone. It's difficult to deal with
because accurate probability is essential to determining significance
but dealing with 4 ranges with 3 parameters is a LOT of possible
answers.

I'm thinking that the proper way to handle this might be to consider
such a prediction to be multiple predictions, one for each possible
combination, since the predictor isn't specifying which combination to
expect. That would be devastating to their significance, as even a
perfect hit would also involve multiple misses as well. But it's
mathematically sound as far as I can tell.

The best solution is to make only specific predictions and accept only
perfect hits but that's not something I can control. It's up to the
predictors to quit wanting credit for "near enough" results.
Especially when "near enough" is quite a way away.

Any opinions?

Yeah. They cover this stuff in university statistics courses. You
ought to try taking some of them.

Unfortunately, statistics courses don't deal with logarithmic
data like earthquake M or Richter sizes so calculating standard
deviations won't work real well. Of course you could change them
to non-logartihmic numbers. That might give a better idea of what
you're actually facing.

--
************* DAVE HATUNEN (hatunen@cox.net) *************
* Tucson Arizona, out where the cacti grow *
* My typos & mispellings are intentional copyright traps *

Dave, I think Roger is well-qualified to be doing the statistical analysis.
If I'm not mistaken, he spent a career doing something like that for the
USGS.

Roger? Could you, once again, re-state your relevant background for us?

Mike Williams
Arroyo Grande, CA U.S.
Hatunen
Posted: Fri Jan 25, 2008 10:02 pm
Guest
On Fri, 25 Jan 2008 17:36:28 -0800 (PST), rogerh@lpbroadband.net
wrote:

Quote:
On Jan 25, 4:59 pm, "Mike Williams" <miklw...@pacbell.net> wrote:
"Hatunen" <hatu...@cox.net> wrote in message

Yeah. They cover this stuff in university statistics courses. You
ought to try taking some of them.

Unfortunately, statistics courses don't deal with logarithmic
data like earthquake M or Richter sizes so calculating standard
deviations won't work real well. Of course you could change them
to non-logartihmic numbers. That might give a better idea of what
you're actually facing.

Dave, I think Roger is well-qualified to be doing the statistical analysis.
If I'm not mistaken, he spent a career doing something like that for the
USGS.

Roger? Could you, once again, re-state your relevant background for us?

Mike;

Academically, a BA in Psychologyy but I had started out in Geology so
had enough credits to qualify as a geophysicist with the government.

I spent 34 years, most of it with the Seismology group in first NOAA
and then USGS when they took us over. As part of that I created and
ran the Prediction Monitoring and Evaluation Program which accepted
predictions from all sources and evaluated their success rates. The
project was terminated prematurely so I never got to publish a final
report but after looking at several hundred predictors, none managed
to pass the test.

My current programs are similar to the PMEP program but use better
statistical methods adopted from Dr Alan Jones. I was also able to
persuade him to create the method to be used on predictions with
different probabilities.

Then why are you asking these questions here in this group?
You're obviously the expert.

--
************* DAVE HATUNEN (hatunen@cox.net) *************
* Tucson Arizona, out where the cacti grow *
* My typos & mispellings are intentional copyright traps *
Hatunen
Posted: Fri Jan 25, 2008 10:33 pm
Guest
On Fri, 25 Jan 2008 17:39:25 -0800 (PST), rogerh@lpbroadband.net
wrote:


Quote:
Unfortunately, statistics courses don't deal with logarithmic
data like earthquake M or Richter sizes so calculating standard
deviations won't work real well. Of course you could change them
to non-logartihmic numbers. That might give a better idea of what
you're actually facing.

Doesn't matter. The logarithmic nature of magnitude has no bearing on
the analysis which is concerned only with hit or miss results.

It matters if a hit can be an order of maghnitude outside the
main predicted number. To say that a quake of magnitude 7 will
occur and then allowing a quake of magnitude 6 to fulfill the
prophecy is to be deceived by the "small" difference of "only"
one.

But yes, if you set up very clear hit or miss criteria and don't
fudge them post facto it doesn't much matter. Nevertheless,
predicting a quake between, say, 4.0 and 7.0 is kind of silly.

--
************* DAVE HATUNEN (hatunen@cox.net) *************
* Tucson Arizona, out where the cacti grow *
* My typos & mispellings are intentional copyright traps *
Mike Williams
Posted: Fri Jan 25, 2008 10:45 pm
Guest
"Hatunen" <hatunen@cox.net> wrote in message
news:d35lp3143qds67mb7lt9t6cjglalondflv@4ax.com...
Quote:
On Fri, 25 Jan 2008 17:36:28 -0800 (PST), rogerh@lpbroadband.net
wrote:

On Jan 25, 4:59 pm, "Mike Williams" <miklw...@pacbell.net> wrote:
"Hatunen" <hatu...@cox.net> wrote in message

Yeah. They cover this stuff in university statistics courses. You
ought to try taking some of them.

Unfortunately, statistics courses don't deal with logarithmic
data like earthquake M or Richter sizes so calculating standard
deviations won't work real well. Of course you could change them
to non-logartihmic numbers. That might give a better idea of what
you're actually facing.

Dave, I think Roger is well-qualified to be doing the statistical
analysis.
If I'm not mistaken, he spent a career doing something like that for the
USGS.

Roger? Could you, once again, re-state your relevant background for us?

Mike;

Academically, a BA in Psychologyy but I had started out in Geology so
had enough credits to qualify as a geophysicist with the government.

I spent 34 years, most of it with the Seismology group in first NOAA
and then USGS when they took us over. As part of that I created and
ran the Prediction Monitoring and Evaluation Program which accepted
predictions from all sources and evaluated their success rates. The
project was terminated prematurely so I never got to publish a final
report but after looking at several hundred predictors, none managed
to pass the test.

My current programs are similar to the PMEP program but use better
statistical methods adopted from Dr Alan Jones. I was also able to
persuade him to create the method to be used on predictions with
different probabilities.

Then why are you asking these questions here in this group?
You're obviously the expert.

--
************* DAVE HATUNEN (hatunen@cox.net) *************
* Tucson Arizona, out where the cacti grow *
* My typos & mispellings are intentional copyright traps *

Jeez, Dave, what's with the gratuitous crotchetiness? Roger is doing us a
service doing his analysis, and has done so in other venues for a long time
now.

Also, you wrote: <<statistics courses don't deal with logarithmic
Quote:
data like earthquake M or Richter sizes so calculating standard
deviations won't work real well.

Roger's analysis should not be affected at all by the logarithmic nature of
magnitudes. Why do you think it would? A predictor predicts an event
(typically with a minimum magnitude), and Roger can quite accurately
determine the likelihood of that same event happening without any reference
to logarithms.

Mike Williams
Arroyo Grande, CA U.S.
Skywise
Posted: Sat Jan 26, 2008 12:26 am
Guest
Hatunen <hatunen@cox.net> wrote in news:d35lp3143qds67mb7lt9t6cjglalondflv@
4ax.com:

Quote:
Then why are you asking these questions here in this group?
You're obviously the expert.

Do experts have all the answers? No they don't. You need to check
your definition of "expert".

Brian
--
http://www.skywise711.com - Lasers, Seismology, Astronomy, Skepticism
Seismic FAQ: http://www.skywise711.com/SeismicFAQ/SeismicFAQ.html
Quake "predictions": http://www.skywise711.com/quakes/EQDB/index.html
Sed quis custodiet ipsos Custodes?
Skywise
Posted: Sat Jan 26, 2008 12:40 am
Guest
Hatunen <hatunen@cox.net> wrote in news:hv6lp39rcngdadm4cokcr7d6s8usdm0sah@
4ax.com:

Quote:
It matters if a hit can be an order of maghnitude outside the
main predicted number. To say that a quake of magnitude 7 will
occur and then allowing a quake of magnitude 6 to fulfill the
prophecy is to be deceived by the "small" difference of "only"
one.

Since the prediction and the evaluation are all made using
the same logarithmic scale, it matters not. I think you need
to check your definition of "logarithm" as well.


Quote:
But yes, if you set up very clear hit or miss criteria and don't
fudge them post facto it doesn't much matter. Nevertheless,
predicting a quake between, say, 4.0 and 7.0 is kind of silly.

Maybe we could start demanding that they make their magnitude
predictions in ergs?

Brian
--
http://www.skywise711.com - Lasers, Seismology, Astronomy, Skepticism
Seismic FAQ: http://www.skywise711.com/SeismicFAQ/SeismicFAQ.html
Quake "predictions": http://www.skywise711.com/quakes/EQDB/index.html
Sed quis custodiet ipsos Custodes?
Hatunen
Posted: Sat Jan 26, 2008 1:04 am
Guest
On Sat, 26 Jan 2008 04:26:09 -0000, Skywise
<into@oblivion.nothing.com> wrote:

Quote:
Hatunen <hatunen@cox.net> wrote in news:d35lp3143qds67mb7lt9t6cjglalondflv@
4ax.com:

Then why are you asking these questions here in this group?
You're obviously the expert.

Do experts have all the answers? No they don't. You need to check
your definition of "expert".

Here???

It is to laugh...

--
************* DAVE HATUNEN (hatunen@cox.net) *************
* Tucson Arizona, out where the cacti grow *
* My typos & mispellings are intentional copyright traps *
Skywise
Posted: Sat Jan 26, 2008 1:21 am
Guest
Hatunen <hatunen@cox.net> wrote in news:8qflp3h31pnlfrc7uabf0t1p087j2e35ul@
4ax.com:

Quote:
On Sat, 26 Jan 2008 04:26:09 -0000, Skywise
into@oblivion.nothing.com> wrote:

Hatunen <hatunen@cox.net> wrote in news:d35lp3143qds67mb7lt9t6cjglalondflv@
4ax.com:

Then why are you asking these questions here in this group?
You're obviously the expert.

Do experts have all the answers? No they don't. You need to check
your definition of "expert".

Here???

It is to laugh...

Mea Culpa. I thought you were asking a real question.

Instead it appears you are one of those who asks questions for
which you already know the answer. You know, the type who asks
the question only to argue with the answers given because it
does not fit within their preconceived notions of what the
answer is supposed to be....because they actually already have
all the answers.

YOU are the expert.

Brian
--
http://www.skywise711.com - Lasers, Seismology, Astronomy, Skepticism
Seismic FAQ: http://www.skywise711.com/SeismicFAQ/SeismicFAQ.html
Quake "predictions": http://www.skywise711.com/quakes/EQDB/index.html
Sed quis custodiet ipsos Custodes?
 
Page 1 of 4    Goto page 1, 2, 3, 4  Next   All times are GMT - 5 Hours
The time now is Thu Dec 04, 2008 7:43 pm