Effect size with no control group?

Main Page | Report this Page

Science Forum Index » Space - Consult Forum » Effect size with no control group?

Page 1 of 3 Goto page 1, 2, 3 Next

Author

Message

Guest

Posted: Mon Nov 20, 2006 1:01 pm

Being a physicist, I know some statistics but not very much,
and I tend to oversimplify calculations when statistics is
involved. So I'm posting a few questions to check if I am doing
too wrong.

Suppose I want to estimate the effect of a treatment, with
no control group. There are n subjects, and only 2 tests:
an initial test before the treatment, and a final test after
the treatment, that's all. I believe that reliable results can
be obtained even in this case, but it seems that this case is
unusual, since I have not found anything like this in books,
articles, or simple web pages.

This is what I would do.
I know that Cohen's effect-size is
d = (MeanAfter - MeanBefore) / SQR((StDevAfter^2 + StDevBefore^2) /2)
However, rather than d, I would consider important the value
z = d * SQR(n) , that is roughly the z ot t score
to be used in a z-test or t-test to compare the two means
(as if there were two different groups of n subjects, rather
than a single group measured before and after the treatment).
So, provided that z is large (as I expect), I can easily refuse
the null hypothesis (that nothing has changed).
Can't I obtain a reliable result in this way?
I can't see how this can be wrong.

However, suppose that a control group is strictly needed
(for reasons that I don't understand at this time).
Can I use 3 times the same group, at 3 different times?
For example, imagine that the treatment duration is 1 month.
I would do the first test 1 month before starting the treatment;
the second test, just before starting the treatment;
the third test, at the end of the treatment (that is 1 month
after starting).
Test 1 would act as "control group" to compare with Test 2.
Of course, effect would be evaluated by comparing Test 2 and 3.
What is wrong in this?

Thanks

Fabrizio Coppola
Italy

Guest

Posted: Mon Nov 20, 2006 2:05 pm

scientia@ipotesi.net wrote:

Quote:

I know that Cohen's effect-size is
d = (MeanAfter - MeanBefore) / SQR((StDevAfter^2 + StDevBefore^2) /2)
However, rather than d, I would consider important the value
z = d * SQR(n)

Sorry: it would not be SQR(n), but SQR(n/2)
Of course I suppose that n1 = n2, and consider (n-1) ~ n

Quote:

... that is roughly the z or t-score
to be used in a z-test or t-test to compare the two means
(as if there were two different groups of n subjects, rather
than a single group measured before and after the treatment).

I mean that it does make sense to me to compare (Mean2 - Mean1)
with the "standard error" (as it is usually made in t-test of two
means)
rather than with the full standard deviation (as it is made with Cohen
effect size).

In any case, provided that what I wrote is roughly correct,
my question is: what is wrong is making such a research
with no control group, but only with a pre-test and a post-test
on the same group?

Thanks again

Fabrizio Coppola
Italy

Old Mac User

Posted: Mon Nov 20, 2006 2:52 pm

Guest

FC...

You said that each of n subjects were evaluated "before treatment" and
also "after treatment". This describes "paired data". The method for
analyzing paired data via a one-column t-test is straightforward.

The method you have described (Cohen's) is either the sames as or
"like" the ordinary two-column t-test. If there is variation among
the n subjects (almost surely true) then... without knowing more... I'd
go directly to the one-column (paired data) t-test and would not use
the two-column t-test. The reason for this can be explained. But
before attemptint to explain it, I have a request of you. Please post
a sample of your data the following format

Subject Initial After
1 xx yy
2 xx yy
3 xx yy
etc. for perhaps 10 subjects. xx and yy indicates numerical data...
the "test data".

Quote:

From this I will calculate the paired data one-column t-test and do
this so you can see how it's done. Then you can extend it to all of

your data and/or to other such data. In doing this I will explain why I
suggested treating this as paired data.

In doing this I as assuming that the xx and yy data for (say) Sugject 1
is "paired". That is, that these values did indeed come from the
Initial and After measurements for Subject 1.
If I have this right, then I see no reason for needing a control group.
In fact, it's likely best to do this work without a control group for
reasons that also need to be explained.

If for any reason you wish to, feel free to contact me via e-mail.

Take care... OMU

scientia@ipotesi.net wrote:

Quote:

Guest

Posted: Mon Nov 20, 2006 3:16 pm

Thanks for your help!
Here are sample data:

Subject Initial After
1 50 38
2 39 30
3 36 28
4 52 41
5 54 44
6 40 31
7 35 27
8 49 40
9 55 45
10 53 42

Old Mac User wrote:

Quote:

before attemptin to explain it, I have a request of you. Please post
a sample of your data the following format

Subject Initial After
1 xx yy
2 xx yy
3 xx yy

In doing this I as assuming that the xx and yy data for (say) Subject 1
is "paired". That is, that these values did indeed come from the
Initial and After measurements for Subject 1.

This is the case, unless we misunderstand each other's words
(that I don't think)

Quote:

If I have this right, then I see no reason for needing a control group.

I see there is always a control group in researches like this.
I don't know why. And I either don't understand why there is.

Quote:

In fact, it's likely best to do this work without a control group for
reasons that also need to be explained.

Thanks for all

Fabrizio Coppola
Italy

Bruce Weaver

Posted: Mon Nov 20, 2006 3:40 pm

Guest

scientia@ipotesi.net wrote:

Quote:

If I have this right, then I see no reason for needing a control group.

I see there is always a control group in researches like this.
I don't know why. And I either don't understand why there is.

In fact, it's likely best to do this work without a control group for
reasons that also need to be explained.

When dealing with data collected on humans (or other animals), you could
ask if the difference between pre- and post-treatment means is due to
the treatment, to the passage of time (e.g., maturation), to a placebo
effect, or to some combination of the three? Without a control group
(that also has the two measures but without the treatment), you wouldn't
know for certain. But given that you are doing physics, these kinds of
issues are probably not a concern. (I say probably, because who knows
what strange things are going on out there at the fringes of theoretical
physics!)

--
Bruce Weaver
bweaver@lakeheadu.ca
www.angelfire.com/wv/bwhomedir

Marc Schwartz

Posted: Mon Nov 20, 2006 4:00 pm

Guest

scientia@ipotesi.net wrote:

Quote:

I see there is always a control group in researches like this.
I don't know why. And I either don't understand why there is.

Because you have to be able to answer the question: "So What?"

To put it into terms a physicist should be able to understand, as
Einstein noted, "It's all relative".

Presuming that we are talking about human clinical trials here, since
you used the words "treatment" and "subjects", take an example for a
double blinded, randomized controlled clinical trial. In such a trial,
neither the subjects nor the investigators know what treatment the
subjects are getting (hence double blinded). The subjects, after
meeting inclusion and exclusion criteria, are randomized to each
treatment to minimize selection bias and the entire trial is conducted
under the guidance of a rigorously defined protocol. The protocol is
designed by both clinical subject matter experts and statisticians.

At the conclusion of the study:

Your experimental group, using your treatment of interest, is observed
to have a reduction of 30% from baseline for some measured
characteristic.

Your control group, using a placebo treatment, is observed to have a
reduction of 15% from baseline for the same measured characteristic. By
the way, such a change in a placebo group is not uncommon.

Can you still claim a benefit for your experimental treatment of a 30%
reduction from baseline?

I would recommend that you begin with a basic review of the following:

http://en.wikipedia.org/wiki/Placebo_effect

and:

http://en.wikipedia.org/wiki/Clinical_trial

Note that increasingly, even for equivalence studies (ie. comparing a
generic drug to an approved branded drug), regulatory agencies are
requiring a third control arm to demonstrate efficacy. Simply
demonstrating equivalence to the current gold standard is insufficient
to demonstrate efficacy relative to no treatment at all.

HTH,

Marc Schwartz

Old Mac User

Posted: Mon Nov 20, 2006 5:06 pm

Guest

I'm working against a deadline today.
Will do a paired data analysis of this and will post that analysis here
tonight. This data is exactly what I expected. Thanks. OMU

scientia@ipotesi.net wrote:

Quote:

Thanks for your help!
Here are sample data:

Subject Initial After
1 50 38
2 39 30
3 36 28
4 52 41
5 54 44
6 40 31
7 35 27
8 49 40
9 55 45
10 53 42

Old Mac User wrote:

before attemptin to explain it, I have a request of you. Please post
a sample of your data the following format

Subject Initial After
1 xx yy
2 xx yy
3 xx yy

In doing this I as assuming that the xx and yy data for (say) Subject 1
is "paired". That is, that these values did indeed come from the
Initial and After measurements for Subject 1.

This is the case, unless we misunderstand each other's words
(that I don't think)

If I have this right, then I see no reason for needing a control group.

I see there is always a control group in researches like this.
I don't know why. And I either don't understand why there is.

In fact, it's likely best to do this work without a control group for
reasons that also need to be explained.

Thanks for all

Fabrizio Coppola
Italy

Old Mac User

Posted: Mon Nov 20, 2006 6:22 pm

Guest

Subject Initial After Differences
1 50 38 12
2 39 30 9
3 36 28 8
4 52 41 11
5 54 44 10
6 40 31 9
7 35 27 8
8 49 40 9
9 55 45 10
10 53 42 11
Avgs 46.3 36.6 9.7

Diff of averages is 46.3 - 36.6 = 9.7
Average of Differences = 9.7

Note: This is just a check on arithmetic.

I really need the column of differences,
the average of that column,
and the standard deviation of that column, Sd

Calculate the std dev. of the column of differences:

Sd = 1.337

Now for the one-column t-test.

t = Avg of Diffs/[Sd*Sqrt(1/n)]

t = 9.7/[1.337*Sqrt(1/10)] = 9.7/[1.337 * 0.316)] = 22.85

The Avg of Differences is very large relative to the variation in
the differences. Very large, indeed...!!! This t-ratio is exceedingly
large relative to the reference values of t (in a t-table) with
9 degrees of freedom. To use some statistical jargon
(which I detest) "this is significant at the 0.001 level" and even
beyond that level.

Why 9df? Because the variation in the column of differences was
calculated with just 9 df.

Quote:

From this I conclude... the treatment (initial - after) causes a real
and enduring change in the test results. This difference of averages

is not due to chance.

So why did we use a paired test (one-column t-test) instead of the
usual two-column t-test?

If we use the two-column t-test the variation (the pooled standard
deviation) will be calculated from variation in the columns "Initial"
and "After". Most of that variation comes from variation among the
subjects... and variation among the subjects is very large. So
large, in fact, that the two-column t-ratio will be "numbed" (greatly
depressed) and we may not be able to proclaim what is obviously
a real difference. (After all, the After tests were lower than the
Initial
tests 10 times out of ten!!!). Just for your information, the
two-column
t-ratio is only 2.93 with 18 degrees of freedom (9 from each column).
True, this is sufficient to claim there is a real difference. But not
really
consistent with noting that there is a difference in the same direction
10 times out of 10. In some instances the two-column t-test will
fail to detect a difference even when, by inspection of the data, such
a
difference is obviously present.

The one-column (paired data) test is appropriate when (1) the data
are paired (as they are in this instance... 38 goes with the 50,
30 goes with the 39... changing the order of the numbers in one column
would change the meaning and interpretation of the data. And... (2)
there is notable variation among the subjects... which there certainly
is
in this instance.

Why did I say that comparing against a standard might not be a good
idea?
Mainly because, if you compare "After" against a standard
again-and-again
then all of your data is based on just one source (or material, sor
whatever
constitutes "subjects". In this case all of your conclusions are based
on
work done with just one "subject", and will not be based broadly on
"subjects in general". If there is a problem with the "standard" then
your
data can be misleading.

As others have noted, it is true that when working with human subjects
there are other issues that must be considered. Particularly when
working with
ethical drugs or medical devices and when preparing claims that must be
approved by government agencies. Moreover, in a lot of "medical work"
comparisons are made of "no treatment vs a treatment" to demonstrate
that
there is some merit in the treatment. In many (if not most) instances
there
should also be a valid comparison of "our treatment vs an established
and accepted treatment"... hence three columns of data to be
considered.

I hope this helps. If you have questions, comments, etc. please let me
know.

Just fyi I'm a chemical engineer and also a statistician.
But I do work with medical devices and certain ethical drugs as well.
OMU

scientia@ipotesi.net wrote:

Quote:

Guest

Posted: Tue Nov 21, 2006 8:20 am

First of all, thanks to everybody who answered.
Later I will post something about the topics explained
by Bruce Weaver and Marc Schwartz.

About the math, thanks to Old Mac User.
I understand now what you mean with two-column and
one-column t-test: without your explanations, I would
have done a two-column z-test (*).

So, my result would have been z = 3 or so
(instead of t = 22).
I felt that it had to be much more significant than z = 3
but I was not able to understand how to calculate
the real t. Your explanation on the differences is very
clear and finally answers my doubts.

I imagine that Cohen's "Effect size" d can be calculated
even in this case, by comparing the average 9.7 to the
standard deviation 1.337, rather than to the standard
error 1.337/sqr(n).
So d = 9.7/1.337 = 7.26 (that is huge).
Or is Cohen's d not used in cases like this?

What happens if one of the initial subjects does not
participate in the second test?
I imagine that the value of the first test should be
used for the second test (so that S.D. of differences is
increased and significance becomes lower).
But I am not sure of this, and I don't know if there are
standard methods in this case.

A final question: I see that tables of Student's t distribution
never report alpha beyond 0.001: is this considered extreme?

Thanks again,

Fabrizio Coppola

(*) By the way, physicists always tend to use a gaussian
(even if n=10) and always consider n=n-1 .
In other words, 10 = 9 >> 30 Smile

Marc Schwartz

Posted: Tue Nov 21, 2006 10:20 am

Guest

scientia@ipotesi.net wrote:

Quote:

In a clinical trial, this would, along with all other such details, be
specified in the statistical analysis plan document. Such decisions are
made on an a priori basis, not based upon post hoc findings and
therefore biased. There are international regulatory guideline documents
that provide accepted standards here.

In addition, if a scheduled test was missed, a formal protocol deviation
would be documented.

Commonly, one would impute missing data and there are a multitude of
ways in which this can be done.

One of the better references is:

Statistical Analysis with Missing Data, Second Edition
by Roderick J. A. Little, Donald B. Rubin
Wiley-Interscience; 2nd edition (September 9, 2002)
http://www.amazon.com/Statistical-Analysis-Missing-Data-Second/dp/0471183865

Quote:

I imagine that the value of the first test should be
used for the second test (so that S.D. of differences is
increased and significance becomes lower).
But I am not sure of this, and I don't know if there are
standard methods in this case.

That particular approach would be known as LOCF or Last Observation
Carry Forward, which is shown to be a fairly biased method. There are
better ways and these are described in the aforementioned reference.

Quote:

A final question: I see that tables of Student's t distribution
never report alpha beyond 0.001: is this considered extreme?

The particular cutoff can be, to some extent, community standard
specific. These days, most folks use software (such as R[1], which is
what I use) and these will usually calculate p values to 15 or 16
significant digits. However, save for special circumstances, most will
report extreme values as:

p < 0.0001

Four decimal places is common.

HTH,

Marc

1: http://www.r-project.org/

Richard Ulrich

Posted: Tue Nov 21, 2006 10:47 pm

Guest

On 20 Nov 2006 09:01:06 -0800, scientia@ipotesi.net wrote:

Quote:

Being a physicist, I know some statistics but not very much,
and I tend to oversimplify calculations when statistics is
involved. So I'm posting a few questions to check if I am doing
too wrong.

You seem to confuse a number of questions, and not all
of them have been cleared up by previous responses....

Quote:

Suppose I want to estimate the effect of a treatment, with
no control group.

There are two distinct notions of "the effect of treatment."

One is the 'statistical test' perspective: Is there *anything*
there? That calls for a test, which becomes more powerful
with an increase of sample size. A t-test or z-test takes a
larger value with increasing sample size, for the same
difference in means between groups.

The other is the "effect size" perspective: How 'big' is
the effect, in some useful terms. That can be given *best*
if the terms of the test are familiar, such as "growing by
three inches" or "increasing IQ by 8 points." It can be
given in a more generic way by using an index such as
Cohen's d.

Quote:

There are n subjects, and only 2 tests:
an initial test before the treatment, and a final test after
the treatment, that's all. I believe that reliable results can
be obtained even in this case, but it seems that this case is
unusual, since I have not found anything like this in books,
articles, or simple web pages.

Yes, it I supposed it is usually *dismissed* in the first week
of any course in experimental design. But you should find
information, of a sort, by looking for "paired t-test."

The scientists who are satisfied with simple pre-post comparisons
are looking to confirm something that is strongly suspected, or
else they have huge differences that out-weigh any doubts
by their face validity. Either X worked - big effect - or it didn't.

Quote:

This is what I would do.
I know that Cohen's effect-size is
d = (MeanAfter - MeanBefore) / SQR((StDevAfter^2 + StDevBefore^2) /2)

This is a measure of "effect size" -- see my second paragraph.
You rightly note that it is converted to something else for testing.

Quote:

However, rather than d, I would consider important the value
z = d * SQR(n) , that is roughly the z ot t score
to be used in a z-test or t-test to compare the two means
(as if there were two different groups of n subjects, rather
than a single group measured before and after the treatment).

As it has been pointed out, you can be pretty confident of
the direction when your cases (as in your later example)
are 100% showing change in the same direction. And the
simple test on "change" is a one-sample t-test, using the
standard deviation of the change-scores and comparing to
zero.

Quote:

So, provided that z is large (as I expect), I can easily refuse
the null hypothesis (that nothing has changed).
Can't I obtain a reliable result in this way?
I can't see how this can be wrong.

And it is bad experimental design, *if* there are any other
reasons that *might* explain the difference -- aging,
learning the test, actions of the experimenter that are not
mentioned. Thus, that design in the social sciences is hardly
ever adequate.

If no one will suggest or argue for any other explanation,
then the design can be fine.

Quote:

However, suppose that a control group is strictly needed
(for reasons that I don't understand at this time).

It will be hard to set up a *proper* control group if you don't
understand the need. What is it that *might* need to be controlled
for?

Quote:

Can I use 3 times the same group, at 3 different times?
For example, imagine that the treatment duration is 1 month.
I would do the first test 1 month before starting the treatment;
the second test, just before starting the treatment;
the third test, at the end of the treatment (that is 1 month
after starting).
Test 1 would act as "control group" to compare with Test 2.
Of course, effect would be evaluated by comparing Test 2 and 3.
What is wrong in this?

This sort of design has been used, in the absence of something
better. It is fairly 'sound' if there is no discernible change
between Tests 1 and 2. That would give (some) inferential
support to your contention that nothing would ordinarily happen.
- You do have a problem if there is change is documented between
Test 1 and 2, since it violates your presumption that the tests are
fair and do not measure anything irrelevant to the experiment.

--
Rich Ulrich, wpilib@pitt.edu
http://www.pitt.edu/~wpilib/index.html

Guest

Posted: Wed Nov 22, 2006 12:57 pm

Thanks to Marc Schwartz for the "methodological" answer,
to Old Mac User for the math, and to Rich Ulrich for the
answer somewhere in between.

Here is a description of the results that I would like to
reproduce. If you visit the following web page
http://www.tm.org/discover/research/charts/e18.html
you will find the "effect size" caused by Transcendental
Meditation (a simple mental technique) in decreasing
anxiety, several weeks after taking the course.
The effect size is around -0.75 (very good).
I have some articles with more details: the data that
I posted in this newsgroup were from a specific research
in this field. The values have been obtained with the
Spielberger's STAI inventory. Best results are obtained
by "common" people, with a starting STAI value around the
overall average (estimated on US population and considered
valid for all western countries), or just over the average.
The effect size brings them well under the average, as you
can see from the effect size.
However, I suspect that they only considered those subjects
who took the final test, and ignored those people who
abandoned the trial (that would have decrease the effect size).
In any case, the same excellent results are not obtained
if the starting value of anxiety is too high (more specific
treatments are adequate in this case: this would be the field
of clinical psychology rather than a meditation technique).

To make a long story short, the course to learn TM
(Transcendental Meditation), that was around $200 in 1993,
became more and more expensive: at this time it is $2500
in the USA, €1800 in Europe, £1280 in the UK
(but since spring 2005, it is no more taught in the UK).
This price is too much, since this is a simple technique
for "common" people rather than for people who really need
psychotherapy.

I am involved, with other people, in a study about a
similar technique, that seems to achieve the same excellent
results as TM (that is stronger than other kinds of
meditation or relaxation techniques). There are no
real statisticans in our group: so, I, as a physicist,
am supposed to be an "expert" (in statistics). But, as
everyone can see, I am not expert at all.

You can understand the rest of the story.
We want to achieve some reliable results, but we can't
afford to make a big research, with a control group (even
if the TM researchers had one) or to make a double-blind
test. However, I imagine that a double-blind, placebo-
controlled trial, is strictly needed in drug testing, but
is not so important in a case like ours, where we don't
care about possible psychological influences or placebo
effects: even if they exist, they can be considered a part
of the overall effect. This is what I think.
Any comment is welcome.

To anyone who wants to know more about this new technique:
feel free to email me.

I have minor questions about the math, but I think it is
better that I post them in the sci.stat.math NG.

Fabrizio Coppola

Bruce Weaver

Posted: Wed Nov 22, 2006 3:05 pm

Guest

scientia@ipotesi.net wrote:

Quote:

---- snip the rest -----

Given the subject matter, you definitely need at least one control group
IMO. The control group must have pre and post scores too, but without
the TM intervention.

The reason I said you need *at least one* control group is that doing
nothing between pre and post may not be adequate. Ideally, you should
have another control group that does "something" between pre and post
under conditions that are similar to what the TM group experiences
(e.g., is the TM done in a group setting?), but without the critical
element(s). In short, there are lots of things to control for in this
type of study, and if you don't control for the obvious ones, you'll not
be in a very strong position to sing the praises of TM.

--
Bruce Weaver
bweaver@lakeheadu.ca
www.angelfire.com/wv/bwhomedir

Guest

Posted: Thu Nov 23, 2006 1:37 am

scientia@ipotesi.net wrote:

Quote:

I understand you suggest a single-blind study.
I imagine that a double-blind study may not be implemented, since
the instructor must be aware s/he is instructing the real technique,
rather than a dummy technique (this is not like a drug testing, that
may be done by an unaware experimenter).

This is a difficulty in all behavioral interventions. Although it's
impossible for the instructor to be blind to the intervention, you'd still
want to make sure that whomever is doing the pre and post treatment
assessment of the outcomes is as blind as possible (this is also
difficult, as the participant will sometimes inadvertently reveal what
intervention they had).

Quote:

(e.g., is the TM done in a group setting?) ...

It may be either in a group setting or individually.
The problem for us, at this time, is that we may find have a few
subjects,
that are available for a limited time, so it will be difficult for us
to make
even one control group. However, I will consider your suggestion, aimed
to get better, unassailable results.

There's no point in saving time if the design is essentially worthless,
and without a well-conceived and executed control condition, the study is
is essentially worthless.

I'd also add that because the correct test is a comparison across groups,
you'll need to determine the sample size based on what you think a
meaningful difference would be across the groups. The number you derive
in this calculation may surprise you in that it may take a fairly large
number in each group.

Mike Babyak

Guest

Posted: Thu Nov 23, 2006 4:48 am

Bruce Weaver wrote:

Quote:

Given the subject matter, you definitely need at least one control group
IMO. The control group must have pre and post scores too, but without
the TM intervention.

Quote:

(e.g., is the TM done in a group setting?) ...

It may be either in a group setting or individually.
The problem for us, at this time, is that we may find have a few
subjects,
that are available for a limited time, so it will be difficult for us
to make
even one control group. However, I will consider your suggestion, aimed
to get better, unassailable results.

By the way, this is not TM (that has been widely experimented), but
a new technique that claims to get the same results as TM at a lower
price and in a simpler way. We are a small group of non-expert, at
this time. About the real TM: nobody us learning it in the world at
this
time, due to the high price and to other problems (sectarian
tendencies,
weird projects, etc.). For example, it's incredible that the TM
organization
does not work anymore in the UK, that was the country where TM
became famous, around 40 years ago (there were no studies at this
time, so they were at the stage we are now).
For more information, feel free to email me.

Thanks for all.

Fabrizio Coppola

Page 1 of 3 Goto page 1, 2, 3 Next All times are GMT - 5 Hours The time now is Wed Dec 03, 2008 3:42 pm Science Forum Index