 |
|
| Science Forum Index » Space - Consult Forum » significance test - explanatory rhetoric needed... |
|
Page 1 of 1 |
|
| Author |
Message |
| ... |
 Posted: Sat Sep 19, 2009 6:14 pm |
|
|
|
Guest
|
I am not a statistician, but I do have experience in application.
My question is about a particular case. Some statistically unlearned
are dismissing its statistically significant result as due to a sample
being too small - thereby overlooking the whole point of a
significance test, which (as I understand things) trades off sample
size against magnitude of effect.
The real data are 'sensitive'. The following dummy data have labels
and numbers changed, but are modelled on the actual data:
Place 1 Place 2
Diseased 400 3
Not diseased 600000 500
Monte Carlo testing results in a p <0.0001 of getting a proportion of
diseased at Place 2 as great as, or greater, than the observed count
of 3. It seems to me that there is a high probability that something
has happened at Place 2, and a substantive explanation should be
sought.
Persuasion of the unlearned will be a matter of rhetoric, not
statistics. I need some citable source, to add to my credibility in
arguing the case that some explanation should be sought for Place 2,
in spite of it being a small sample when compared with Place 1.
I can go through various texts, but I wonder whether anybody here can
recommend a published source that cogently explains this particular
issue to non-statisticians. |
|
|
| Back to top |
|
|
|
| Joe Bloe... |
| Posted: Sun Sep 20, 2009 5:56 am |
|
|
|
Guest
|
Bruce Weaver <bweaver at (no spam) lakeheadu.ca> wrote:
[quote:b8fc238ae5]I can imagine someone questioning a failure to find statistical
significance because the sample size was too small. But I don't
understand how or why one would dismiss a statistically significant
result because the sample size was too small.
[/quote:b8fc238ae5]
Suppose there are 1000 hypotheses and there is "high certainty" that
only one of them represents a non-zero effect (Ha), and 999 are true
nulls (H0). I do all 1000 tests, sort results by p-value and ask, how
likely it is that the smallest one is a Ha. Probability that the
smallest P represents Ha increases with sample size, and I believe
this can be quantified by specifying assumptions for the effect
size. Someone would say though why jump through hoops with p-values to
get something bayesian in the first place. |
|
|
| Back to top |
|
|
|
| Bruce Weaver... |
| Posted: Sun Sep 20, 2009 9:14 am |
|
|
|
Guest
|
charlie6 at (no spam) sellfmore.com wrote:
[quote:9dca9e1d3d]I am not a statistician, but I do have experience in application.
My question is about a particular case. Some statistically unlearned
are dismissing its statistically significant result as due to a sample
being too small - thereby overlooking the whole point of a
significance test, which (as I understand things) trades off sample
size against magnitude of effect.
[/quote:9dca9e1d3d]
This sounds backward to me. I can imagine someone questioning a
failure to find statistical significance because the sample size
was too small. But I don't understand how or why one would
dismiss a statistically significant result because the sample size
was too small.
[quote:9dca9e1d3d]
The real data are 'sensitive'. The following dummy data have labels
and numbers changed, but are modelled on the actual data:
Place 1 Place 2
Diseased 400 3
Not diseased 600000 500
Monte Carlo testing results in a p <0.0001 of getting a proportion of
diseased at Place 2 as great as, or greater, than the observed count
of 3. It seems to me that there is a high probability that something
has happened at Place 2, and a substantive explanation should be
sought.
Persuasion of the unlearned will be a matter of rhetoric, not
statistics. I need some citable source, to add to my credibility in
arguing the case that some explanation should be sought for Place 2,
in spite of it being a small sample when compared with Place 1.
I can go through various texts, but I wonder whether anybody here can
recommend a published source that cogently explains this particular
issue to non-statisticians.
[/quote:9dca9e1d3d]
I'm still struggling to understand what the issue is. I think you
are anticipating people being bothered by the imbalance in the two
sample sizes, is that right?
--
Bruce Weaver
bweaver at (no spam) lakeheadu.ca
http://sites.google.com/a/lakeheadu.ca/bweaver/
"When all else fails, RTFM." |
|
|
| Back to top |
|
|
|
| J. Horikx... |
| Posted: Sun Sep 20, 2009 5:34 pm |
|
|
|
Guest
|
charlie6 at (no spam) sellfmore.com reageerde als volgt:
[quote:6fdde2bfab] Place 1 Place 2
Diseased 400 3
Not diseased 600000 500
Monte Carlo testing results in a p <0.0001 of getting a proportion of
diseased at Place 2 as great as, or greater, than the observed count
of 3. It seems to me that there is a high probability that something
has happened at Place 2, and a substantive explanation should be
sought.
Persuasion of the unlearned will be a matter of rhetoric, not
statistics.
[/quote:6fdde2bfab]
I do not see what "the unlearned" have to do with it. It could be
that Place-1 is a middle-great city that has got her first flue
patient 4 days ago and that Place-2 got her first flu patient 6
days ago.
I can't see the exact nature of the _statistical_ problem here.
JH |
|
|
| Back to top |
|
|
|
| Joe Bloe... |
| Posted: Mon Sep 21, 2009 5:30 am |
|
|
|
Guest
|
Rich Ulrich <rich.ulrich at (no spam) comcast.net> wrote:
[quote:4dc8f6203d]On Sun, 20 Sep 2009 15:56:13 +0000 (UTC), Joe Bloe <Joe.Bloe at (no spam) nospam.invalid> wrote:
Bruce Weaver <bweaver at (no spam) lakeheadu.ca> wrote:
I can imagine someone questioning a failure to find statistical
significance because the sample size was too small. But I don't
understand how or why one would dismiss a statistically significant
result because the sample size was too small.
Suppose there are 1000 hypotheses and there is "high certainty" that
only one of them represents a non-zero effect (Ha), and 999 are true
nulls (H0). I do all 1000 tests, sort results by p-value and ask, how
likely it is that the smallest one is a Ha. Probability that the
smallest P represents Ha increases with sample size, and I believe
this can be quantified by specifying assumptions for the effect
size. Someone would say though why jump through hoops with p-values to
get something bayesian in the first place.
Ah, no I don't think that this is the case at all, that
"probability ... increases with sample size."
[/quote:4dc8f6203d]
In the setup that I considered it is in fact the case.
[quote:4dc8f6203d]You *seem* to me to be confusing several pieces of advice,
none of which have anything to do with "multiple tests".
[/quote:4dc8f6203d]
To say that 1 in 1000 is a false null is simply a way of saying
something about a prior for H0. There is no way to quantify the
"probability" above without introducing such a prior.
You seem to think the the original poster (charlie_something)
and me (Joe Bloe) are the same person.
In his setup there seem to be a test where association with disease
is tested at multiple "places", perhaps these are positions
in the genome. If one is doing that at all, there must be a reasonable
information that the disease has a genetic component. "I know one
of you did it, but I don't know which one". As you increase sample size,
the guilty one bubbles up to the top. It is a matter of writing
down the Bayes rule (which would involve the "noncentrality" for the
"guilty" test, and that will increase with N). People averse to the
word Bayes might consider the distribution of ranks for the tests
that correspond to Ha instead, and that would give the same conclusions.
The confounding issue you mentioned is a good point, still the way
of looking at the problem the way I described is a good start. |
|
|
| Back to top |
|
|
|
| ... |
| Posted: Tue Sep 22, 2009 5:41 pm |
|
|
|
Guest
|
| Thanks to all for useful advice on this topic. |
|
|
| Back to top |
|
|
|
| Rich Ulrich... |
| Posted: Wed Sep 23, 2009 10:20 pm |
|
|
|
Guest
|
On Mon, 21 Sep 2009 15:30:13 +0000 (UTC), Joe Bloe
<Joe.Bloe at (no spam) nospam.invalid> wrote:
[quote:4587680c6d]Rich Ulrich <rich.ulrich at (no spam) comcast.net> wrote:
On Sun, 20 Sep 2009 15:56:13 +0000 (UTC), Joe Bloe <Joe.Bloe at (no spam) nospam.invalid> wrote:
Bruce Weaver <bweaver at (no spam) lakeheadu.ca> wrote:
I can imagine someone questioning a failure to find statistical
significance because the sample size was too small. But I don't
understand how or why one would dismiss a statistically significant
result because the sample size was too small.
Joe
Suppose there are 1000 hypotheses and there is "high certainty" that
only one of them represents a non-zero effect (Ha), and 999 are true
nulls (H0). I do all 1000 tests, sort results by p-value and ask, how
likely it is that the smallest one is a Ha. Probability that the
smallest P represents Ha increases with sample size, and I believe
this can be quantified by specifying assumptions for the effect
size. Someone would say though why jump through hoops with p-values to
get something bayesian in the first place.
RU
Ah, no I don't think that this is the case at all, that
"probability ... increases with sample size."
Joe
In the setup that I considered it is in fact the case.
[/quote:4587680c6d]
Okay - I can see how it can be the case - you are not
at all mixing sample sizes.
So, if I get it, you are looking at the implications of
the fact that there is a broader CI around the effect size
represented by a particular p when the N is smaller.
But I don't see where you go with that.
RU>
[quote:4587680c6d]You *seem* to me to be confusing several pieces of advice,
none of which have anything to do with "multiple tests".
Joe
To say that 1 in 1000 is a false null is simply a way of saying
something about a prior for H0. There is no way to quantify the
"probability" above without introducing such a prior.
[/quote:4587680c6d]
I don't see relevance. I think I don't see what you mean
by saying, "quantify the 'probability'" since I don't see where
I would get a different simple result.
[quote:4587680c6d]
You seem to think the the original poster (charlie_something)
and me (Joe Bloe) are the same person.
[/quote:4587680c6d]
True. Sorry.
[quote:4587680c6d]
In his setup there seem to be a test where association with disease
is tested at multiple "places", perhaps these are positions
in the genome. If one is doing that at all, there must be a reasonable
information that the disease has a genetic component. "I know one
of you did it, but I don't know which one". As you increase sample size,
the guilty one bubbles up to the top. It is a matter of writing
down the Bayes rule (which would involve the "noncentrality" for the
"guilty" test, and that will increase with N). People averse to the
word Bayes might consider the distribution of ranks for the tests
that correspond to Ha instead, and that would give the same conclusions.
[/quote:4587680c6d]
Okay, I agree that bigger sample sizes are better than smaller ones.
However, I can point out that "bubbling up" might take some time.
If there are *far* too many tests (and the problem as stated said
nothing about genes or more than one test), then the "accidental"
effects, for small or moderate N, can be larger than the expected
and predicted "effect".
In a way, as a scientist, seeing an effect that is "too large" is
something that ought to be discomfiting. I know that I have read
papers when I concluded that the authors were being a bit
dishonest because, in the absence of decent power, they touted
the "significant result", where I thought they should have warned
about "other causes" or accident.
[In particular: I once reviewed a paper which blamed "life events"
like divorce, etc., for "short survivorship" from HIV virus. The N
was small, but effect was huge. Ignored was the apparent
confound, that "low T-cell count" was registered as a Life Event
and not used as a relevant covariate.]
[quote:4587680c6d]
The confounding issue you mentioned is a good point, still the way
of looking at the problem the way I described is a good start.[/quote:4587680c6d] |
|
|
| Back to top |
|
|
|
|
|
All times are GMT - 5 Hours
The time now is Sat Dec 05, 2009 4:07 pm
|
|