Thanks to everybody for the explanations and the suggestions.
I understand all the math now.
About the experiment: I think we will start with a simple one,
with a pre-test and a post-test, without a control group, just to
see if the effect size is large as we expect (d = -0.7 or so).
Then we can look forward to make a full experiment, with
a control group (more than 10 subjects in each group).
Fabrizio Coppola

scientia@ipotesi.net wrote:
First of all, thanks to everybody who answered.
Later I will post something about the topics explained
by Bruce Weaver and Marc Schwartz.
+++++++++++++++++++++++++++++++++++++

A final question: I see that tables of Student's t distribution
never report alpha beyond 0.001: is this considered extreme?

The particular cutoff can be, to some extent, community standard specific.
These days, most folks use software (such as R[1], which is what I use)
and these will usually calculate p values to 15 or 16 significant digits.
However, save for special circumstances, most will report extreme values
as:

p < 0.0001

Four decimal places is common.

HTH,
From a historical view (post R.A. Fisher), the 4 decimal approach to p

"Marc Schwartz" <marc_schwartz@comcast.net> wrote in message
news:BNOdna-gfKynlv7YnZ2dnUVZ_ridnZ2d@comcast.com...
scientia@ipotesi.net wrote:
First of all, thanks to everybody who answered.
Later I will post something about the topics explained
by Bruce Weaver and Marc Schwartz.
+++++++++++++++++++++++++++++++++++++

++++++++++++++++++++++++++++++++++++++++
A final question: I see that tables of Student's t distribution
never report alpha beyond 0.001: is this considered extreme?

The particular cutoff can be, to some extent, community standard specific.
These days, most folks use software (such as R[1], which is what I use)
and these will usually calculate p values to 15 or 16 significant digits.
However, save for special circumstances, most will report extreme values
as:

p < 0.0001

Four decimal places is common.

HTH,
From a historical view (post R.A. Fisher), the 4 decimal approach to p
values is what has been used. However this creates an unsolvable problem.
Does this mean then at all of the millions of software routines developed
since 1950 in all concevable computer languages and instruction objects to
calculate p values only needs to be accurate to 4 decimals? There are still
people out there that are writting new distribution p value routines (such
as Alfonso). How then can we evaluate stat programs for correctness when the
only criteria is 4 decimal places.?

Or is this issue of correctness moot?

Subject Initial After Differences
1 50 38 12
2 39 30 9
3 36 28 8
4 52 41 11
5 54 44 10
6 40 31 9
7 35 27 8
8 49 40 9
9 55 45 10
10 53 42 11
Avgs 46.3 36.6 9.7

Old Mac User wrote:
Subject Initial After Differences
1 50 38 12
2 39 30 9
3 36 28 8
4 52 41 11
5 54 44 10
6 40 31 9
7 35 27 8
8 49 40 9
9 55 45 10
10 53 42 11
Avgs 46.3 36.6 9.7

If I had been shown this "data" and asked about my
assessment of the "before and after" effect, I would
have suspected that the data was "manufacturered"
by someone who had never seen any real data to fake
some results to make a publication, as has often been
done.

I don't need to check any arithmetic. I don't need to check
any statistical significance. One glance at the ALL positive
(and LARGE) differences made it unmistakable that the
result is "significant" in the sense of OBVIOUSLY there.

I don't see the logic here: you seem to be suggesting that the data was

Reef Fish wrote:
Old Mac User wrote:
Subject Initial After Differences
1 50 38 12
2 39 30 9
3 36 28 8
4 52 41 11
5 54 44 10
6 40 31 9
7 35 27 8
8 49 40 9
9 55 45 10
10 53 42 11
Avgs 46.3 36.6 9.7

If I had been shown this "data" and asked about my
assessment of the "before and after" effect, I would
have suspected that the data was "manufacturered"
by someone who had never seen any real data to fake
some results to make a publication, as has often been
done.

I don't need to check any arithmetic. I don't need to check
any statistical significance. One glance at the ALL positive
(and LARGE) differences made it unmistakable that the
result is "significant" in the sense of OBVIOUSLY there.

I don't see the logic here: you seem to be suggesting that the data was
faked purely because all of the differences were in the same direction.
Do I understand you correctly?

snip

Anon. wrote:
Reef Fish wrote:
Old Mac User wrote:
Subject Initial After Differences
1 50 38 12
2 39 30 9
3 36 28 8
4 52 41 11
5 54 44 10
6 40 31 9
7 35 27 8
8 49 40 9
9 55 45 10
10 53 42 11
Avgs 46.3 36.6 9.7
If I had been shown this "data" and asked about my
assessment of the "before and after" effect, I would
have suspected that the data was "manufacturered"
by someone who had never seen any real data to fake
some results to make a publication, as has often been
done.

I don't need to check any arithmetic. I don't need to check
any statistical significance. One glance at the ALL positive
(and LARGE) differences made it unmistakable that the
result is "significant" in the sense of OBVIOUSLY there.

I don't see the logic here: you seem to be suggesting that the data was
faked purely because all of the differences were in the same direction.
Do I understand you correctly?

snip

You completely MISUNDERSTOOD my point which I elaborated
in the rest of my post, as you always misunderstood everything
I have ever posted.

There was no sign or indication, from what the OP said or others
said, about the data being faked.

My point was that the CONCLUSION of "significant effect" was SO
OBVIOUS from the data, that it borders something that people
MIGHT think the data was faked. It was that "extremely obviously
significant" in the statistical sense that hardly required any
pedantic support.

The point was that everyone was seeking SUPPORT like a drunk
seek a lamp post, by statistical significance or p-value and other
academic mumbo jumbo that they seemed to have overlooked the
perfectly OBVIOUS result that the result was significant.

Ah, OK, but it wasn't clear: it would have helped if you make your main

Reef Fish wrote:
Anon. wrote:
Reef Fish wrote:
Old Mac User wrote:
Subject Initial After Differences
1 50 38 12
2 39 30 9
3 36 28 8
4 52 41 11
5 54 44 10
6 40 31 9
7 35 27 8
8 49 40 9
9 55 45 10
10 53 42 11
Avgs 46.3 36.6 9.7
If I had been shown this "data" and asked about my
assessment of the "before and after" effect, I would
have suspected that the data was "manufacturered"
by someone who had never seen any real data to fake
some results to make a publication, as has often been
done.

I don't need to check any arithmetic. I don't need to check
any statistical significance. One glance at the ALL positive
(and LARGE) differences made it unmistakable that the
result is "significant" in the sense of OBVIOUSLY there.

I don't see the logic here: you seem to be suggesting that the data was
faked purely because all of the differences were in the same direction.
Do I understand you correctly?

snip

You completely MISUNDERSTOOD my point which I elaborated
in the rest of my post, as you always misunderstood everything
I have ever posted.

There was no sign or indication, from what the OP said or others
said, about the data being faked.

My point was that the CONCLUSION of "significant effect" was SO
OBVIOUS from the data, that it borders something that people
MIGHT think the data was faked. It was that "extremely obviously
significant" in the statistical sense that hardly required any
pedantic support.

The point was that everyone was seeking SUPPORT like a drunk
seek a lamp post, by statistical significance or p-value and other
academic mumbo jumbo that they seemed to have overlooked the
perfectly OBVIOUS result that the result was significant.

Ah, OK, but it wasn't clear: it would have helped if you make your main
point clear at the start, not several paragraphs in.

You argument reads
as "I think the data might be fake, because of the low p-value".

Incidentally, the OP was asking about the effect size, not a test of the
null hypothesis anyway!

[snip]

Anon. wrote:
Reef Fish wrote:
Anon. wrote:
Reef Fish wrote:
Old Mac User wrote:
Subject Initial After Differences
1 50 38 12
2 39 30 9
3 36 28 8
4 52 41 11
5 54 44 10
6 40 31 9
7 35 27 8
8 49 40 9
9 55 45 10
10 53 42 11
Avgs 46.3 36.6 9.7
If I had been shown this "data" and asked about my
assessment of the "before and after" effect, I would
have suspected that the data was "manufacturered"
by someone who had never seen any real data to fake
some results to make a publication, as has often been
done.

I don't need to check any arithmetic. I don't need to check
any statistical significance. One glance at the ALL positive
(and LARGE) differences made it unmistakable that the
result is "significant" in the sense of OBVIOUSLY there.

I don't see the logic here: you seem to be suggesting that the data was
faked purely because all of the differences were in the same direction.
Do I understand you correctly?

snip
You completely MISUNDERSTOOD my point which I elaborated
in the rest of my post, as you always misunderstood everything
I have ever posted.

There was no sign or indication, from what the OP said or others
said, about the data being faked.

My point was that the CONCLUSION of "significant effect" was SO
OBVIOUS from the data, that it borders something that people
MIGHT think the data was faked. It was that "extremely obviously
significant" in the statistical sense that hardly required any
pedantic support.

The point was that everyone was seeking SUPPORT like a drunk
seek a lamp post, by statistical significance or p-value and other
academic mumbo jumbo that they seemed to have overlooked the
perfectly OBVIOUS result that the result was significant.

Ah, OK, but it wasn't clear: it would have helped if you make your main
point clear at the start, not several paragraphs in.

It was clear in my first sentence.

A sentence which makes no mention of significance tests, p-values etc.

You argument reads
as "I think the data might be fake, because of the low p-value".

Why would I need any p-value to know that the effect was OBVIOUS?

Indeed: I totally agree.

Incidentally, the OP was asking about the effect size, not a test of the
null hypothesis anyway!

No, Anon Bob. You're the one who ALWAYS came late. NEVER
read any part of the thread, and immediately jumped into where I
entered and immediately made your NOISE showing that you hadn't
read anything and didn't understand anything I said.

I certainly read the title, and the original post, in which the OP wrote

Old Mac User wrote:
Subject Initial After Differences
1 50 38 12
2 39 30 9
3 36 28 8
4 52 41 11
5 54 44 10
6 40 31 9
7 35 27 8
8 49 40 9
9 55 45 10
10 53 42 11
Avgs 46.3 36.6 9.7

If I had been shown this "data" and asked about my
assessment of the "before and after" effect, I would
have suspected that the data was "manufacturered"
by someone who had never seen any real data to fake
some results to make a publication, as has often been
done.

I don't need to check any arithmetic. I don't need to check
any statistical significance. One glance at the ALL positive
(and LARGE) differences made it unmistakable that the
result is "significant" in the sense of OBVIOUSLY there.

Anything else that is done, is merely pedantic, for THIS
data set and this one only.

While there are worthwhile discussions IN GENERAL,
without specific data and CONTEXT of the physics
experiment, about tests and controls, etc., this is the
case that any discussion of a "significant" test or p-
value is of values ONLY to the drunks looking at the
lamp post for support.

What attracted me to this thread in the first place was the
mention of "Cohen's effect size", and then one of the Cohen
Quacks in these groups, Richard Ulrich.

I have yet to see one single idea of Cohen that is worth
consideration for his artificially contrived measures and
terms outside of the mainstream of statistics.

In this particular case, once the DATA is given, the rest of
the discussant (except perhaps Old Mac User) who
wanted to see the DATA in the first place and merely
completed the routine arithmetic to come to the obvious
conclusion, the rest of the discussants seemed to be
BLIND to the OBVIOUS, poking all around in search of
some objects for "support" as a drunk seeks support to
keep from rolling off his feet.

What DIFFERENCE does it make what p-value that
correspond, to two decimals or 4 decimals?

The fact that all 10 differences are positive has one
chance in 2^10 or 1024, even if there is no parametric
information to get support on some T-lamppost or
Z-lamppost.

OMU's t value of 22.85 for 9 d.f. merely gives a

pedantic and worthless p-value of less than .000000001.

"So what? " as that question had been asked in other
contexts in this thread, but irrelevant to the actual DATA
and experiment.

Does anyone really NEED that number before they can
see the VERY OBVIOUS, from the raw data, without
any calculation, except perhaps my mental note of 1
in 1024?

-- Reef Fish Bob.

Reef Fish...

I basically agree with you. My only reason for doing the paired t-test
was to illustrate to the OP the difference between "paired" and "not
paired" and the 1-column t-test.

Plainly, with 10 out of 10 in one
direction, there is no justification for doing these calculations
merely to cite the value of a t-test or its p-value.

The first thing that struck me (concerning these data) is the rather
small amount of variation in the differences. If this were really data
from one of the "hard sciences" (physics, etc.) then I might expect
that. But now that we've learned these are hypothetical data from
experiments on humans, the modest amount of variation in the
differences looks artificial.
If I had know the source of these data (hypothetical or "real") I would
have raised a question about the small variation in the difference.

Regardless of "all the above", I'm now astonished that this thread has
turned into "discussions" about the number of significant figures in
reported p-values. Are those "discussants" programmed to claim that
(say) p = 0.049 is "not signficiant" and p = 0.051 is "significant"?

Sidebar comment: I have major problems with using expressions such as
"significant" and "not significant". I would write a publishable paper
on the reasons for this. I'll cite just one. I sometimes serve as an
expert witness in certain legal actions concerned with the validity of
patents. When I see a patent that proclaims that certain data is
"significant" or "not significant" then I know there's a good chance we
can give the patent owner great pain and probably render the patent
invalid. There are other reasons, but this is one card I delight in
playing.

I am horrified at how so many cite outcomes from "Cohen's Test" and a
multiplicity of others. It would be helpful if they would say "I have
this much data structure in the following manner... coming from work
done in (chemistry, human experiments, etc.) and I'm trying to
determine whether changing (independent variables) actually made a
difference in (fill in the blanks). What test or tests are appropriate
here?"

Every discipline (if they can be called that) seems to have their own
homebrewed "tests"... most trying to accomplish the same as other
homebrewed tests from other disciplines.
The characteristics of many of these data (Type I Error Rates, power,
etc.) are unknown.
In fact, the essential characteristics of many of these tests are
actually very poor when compared to well-established tests found in
generic statistical textbooks.

So the educations types have their own recipes. The psychologists have
their own. When a novice finds one of these in software or in a filthy
gutter, they just assume they have found a little miracle. This is a
sad state of affairs. OMU

Old Mac User wrote:
Subject Initial After Differences
1 50 38 12
2 39 30 9
3 36 28 8
4 52 41 11
5 54 44 10
6 40 31 9
7 35 27 8
8 49 40 9
9 55 45 10
10 53 42 11
Avgs 46.3 36.6 9.7

If I had been shown this "data" and asked about my
assessment of the "before and after" effect, I would
have suspected that the data was "manufacturered"
by someone who had never seen any real data to fake
some results to make a publication, as has often been
done.

I don't need to check any arithmetic. I don't need to check
any statistical significance. One glance at the ALL positive
(and LARGE) differences made it unmistakable that the
result is "significant" in the sense of OBVIOUSLY there.

Anything else that is done, is merely pedantic, for THIS
data set and this one only.

While there are worthwhile discussions IN GENERAL,
without specific data and CONTEXT of the physics
experiment, about tests and controls, etc., this is the
case that any discussion of a "significant" test or p-
value is of values ONLY to the drunks looking at the
lamp post for support.

What attracted me to this thread in the first place was the
mention of "Cohen's effect size", and then one of the Cohen
Quacks in these groups, Richard Ulrich.

I have yet to see one single idea of Cohen that is worth
consideration for his artificially contrived measures and
terms outside of the mainstream of statistics.

In this particular case, once the DATA is given, the rest of
the discussant (except perhaps Old Mac User) who
wanted to see the DATA in the first place and merely
completed the routine arithmetic to come to the obvious
conclusion, the rest of the discussants seemed to be
BLIND to the OBVIOUS, poking all around in search of
some objects for "support" as a drunk seeks support to
keep from rolling off his feet.

What DIFFERENCE does it make what p-value that
correspond, to two decimals or 4 decimals?

The fact that all 10 differences are positive has one
chance in 2^10 or 1024, even if there is no parametric
information to get support on some T-lamppost or
Z-lamppost.

OMU's t value of 22.85 for 9 d.f. merely gives a

pedantic and worthless p-value of less than .000000001.

"So what? " as that question had been asked in other
contexts in this thread, but irrelevant to the actual DATA
and experiment.

Does anyone really NEED that number before they can
see the VERY OBVIOUS, from the raw data, without
any calculation, except perhaps my mental note of 1
in 1024?

-- Reef Fish Bob.

I don't need to check any arithmetic. I don't need to check
any statistical significance. One glance at the ALL positive
(and LARGE) differences made it unmistakable that the
result is "significant" in the sense of OBVIOUSLY there.
++++++++++++++++

Anything else that is done, is merely pedantic, for THIS
data set and this one only.

While there are worthwhile discussions IN GENERAL,
without specific data and CONTEXT of the physics
experiment, about tests and controls, etc., this is the
case that any discussion of a "significant" test or p-
value is of values ONLY to the drunks looking at the
lamp post for support.
++++++++++++++++++++++++++

What attracted me to this thread in the first place was the
mention of "Cohen's effect size", and then one of the Cohen
Quacks in these groups, Richard Ulrich.

I have yet to see one single idea of Cohen that is worth
consideration for his artificially contrived measures and
terms outside of the mainstream of statistics.
++++++++++++++++++++++++++++++++++++++

In this particular case, once the DATA is given, the rest of
the discussant (except perhaps Old Mac User) who
wanted to see the DATA in the first place and merely
completed the routine arithmetic to come to the obvious
conclusion, the rest of the discussants seemed to be
BLIND to the OBVIOUS, poking all around in search of
some objects for "support" as a drunk seeks support to
keep from rolling off his feet.

What DIFFERENCE does it make what p-value that
correspond, to two decimals or 4 decimals?
Depends on what the standards are for the "publication" and publication

The fact that all 10 differences are positive has one
chance in 2^10 or 1024, even if there is no parametric
information to get support on some T-lamppost or
Z-lamppost.

OMU's t value of 22.85 for 9 d.f. merely gives a

pedantic and worthless p-value of less than .000000001.

"So what? " as that question had been asked in other
contexts in this thread, but irrelevant to the actual DATA
and experiment.

Does anyone really NEED that number before they can
see the VERY OBVIOUS, from the raw data, without
any calculation, except perhaps my mental note of 1
in 1024?

-- Reef Fish Bob.

Reef Fish...

I basically agree with you. My only reason for doing the paired t-test

Regardless of "all the above", I'm now astonished that this thread has
turned into "discussions" about the number of significant figures in
reported p-values. Are those "discussants" programmed to claim that
(say) p = 0.049 is "not signficiant" and p = 0.051 is "significant"?

Sidebar comment: I have major problems with using expressions such as
"significant" and "not significant". I would write a publishable paper
on the reasons for this. I'll cite just one. I sometimes serve as an
expert witness in certain legal actions concerned with the validity of
patents. When I see a patent that proclaims that certain data is
"significant" or "not significant" then I know there's a good chance we
can give the patent owner great pain and probably render the patent
invalid. There are other reasons, but this is one card I delight in
playing.

I am horrified at how so many cite outcomes from "Cohen's Test" and a
multiplicity of others. It would be helpful if they would say "I have

this much data structure in the following manner... coming from work
done in (chemistry, human experiments, etc.) and I'm trying to
determine whether changing (independent variables) actually made a
difference in (fill in the blanks). What test or tests are appropriate
here?"

Every discipline (if they can be called that) seems to have their own
homebrewed "tests"... most trying to accomplish the same as other
homebrewed tests from other disciplines.
The characteristics of many of these data (Type I Error Rates, power,
etc.) are unknown.
In fact, the essential characteristics of many of these tests are
actually very poor when compared to well-established tests found in
generic statistical textbooks.

So the educations types have their own recipes. The psychologists have
their own. When a novice finds one of these in software or in a filthy
gutter, they just assume they have found a little miracle. This is a
sad state of affairs. OMU

On 27 Nov 2006 06:22:02 -0800, "Old Mac User"
chendrixstats@yahoo.com> wrote:

Reef Fish...

I basically agree with you. My only reason for doing the paired t-test

[snip]

Regardless of "all the above", I'm now astonished that this thread has
turned into "discussions" about the number of significant figures in
reported p-values. Are those "discussants" programmed to claim that
(say) p = 0.049 is "not signficiant" and p = 0.051 is "significant"?

I'm surprised that anyone is "astonished" that a thread has
wandered to issues irrelevant to the original post. Your post
takes the wandering even further astray, and that is fine.

No one claimed anything about the exact 5% test, but my
own post warned that p-values generally are not typically
that precise. Further, I expect that there are many readers
who did not know the information that I passed along, to the
effect that p-levels (from parametric tests) smaller than 0.001
are increasingly bad estimates, merely on narrow statistical grounds.
(After the statistical grounds, there are "inferential" grounds.
That's a bigger topic.)


Sidebar comment: I have major problems with using expressions such as
"significant" and "not significant". I would write a publishable paper
on the reasons for this. I'll cite just one. I sometimes serve as an
expert witness in certain legal actions concerned with the validity of
patents. When I see a patent that proclaims that certain data is
"significant" or "not significant" then I know there's a good chance we
can give the patent owner great pain and probably render the patent
invalid. There are other reasons, but this is one card I delight in
playing.

I am horrified at how so many cite outcomes from "Cohen's Test" and a
multiplicity of others. It would be helpful if they would say "I have

This slap at amateurs would be more impressive if there were
such a test as "Cohen's test." Googling the net gives me only 289
hits, and the first 20 are about Bernie Cohen and Baron-Cohen.
Neither of them is the Jacob Cohen who advocated "effect sizes"
as an alternative to reporting p-levels. (Which the original poster
confused with t-tests, in ways my post clarified. As the OP has
recently validated.)

this much data structure in the following manner... coming from work
done in (chemistry, human experiments, etc.) and I'm trying to
determine whether changing (independent variables) actually made a
difference in (fill in the blanks). What test or tests are appropriate
here?"

Or, beyond testing, "How *big* is the effect, after you are sure
that there is one?"


Every discipline (if they can be called that) seems to have their own
homebrewed "tests"... most trying to accomplish the same as other
homebrewed tests from other disciplines.
The characteristics of many of these data (Type I Error Rates, power,
etc.) are unknown.
In fact, the essential characteristics of many of these tests are
actually very poor when compared to well-established tests found in
generic statistical textbooks.

So the educations types have their own recipes. The psychologists have
their own. When a novice finds one of these in software or in a filthy
gutter, they just assume they have found a little miracle. This is a
sad state of affairs. OMU

I've seen posts that document how bad *industry* has been
with its "Six Sigma" testing, and so on, but I don't remember
*widespread* errors by psychologists or "education types" that
are comparable by being inherently wrong. Oh, Reef Fish has
posted claims to that end, but Reef Fish knows so little about
social sciences that he probably believed that there's a popular
"Cohen's test."

I *do* see folks doing a poor job of applying tests that
are well-established, but that is not the problem that you
described -- "their own recipes" where they assume "they
have found a little miracle". Is that a flight of rhetoric, or,
maybe, do you have a more concrete example?

[snip, rest]

--
Rich Ulrich, wpilib@pitt.edu
http://www.pitt.edu/~wpilib/index.html