 |
|
| Science Forum Index » Statistics - Math Forum » Randomization test - need help interpreting results... |
|
Page 1 of 1 |
|
| Author |
Message |
| Zootal... |
 Posted: Wed Nov 04, 2009 11:31 pm |
|
|
|
Guest
|
I'm doing a randomization test on a data set, and I'm not quite sure how to
interpret the results. Suppose I have two data sets with a known sample mean
difference, and I want to do a randomization test to test the null
hypothesis that the sample means are the same, u1 = u2.
Suppose the known sample mean difference is 5, ie. u1 is 25 and u2 is 30. I
do a bunch of tests(some arbitrary large number) and I find that 40% of the
tests give mean differences that are greater than or equal to +/- my known
sample mean of 5, or the sample mean difference in 40% of the tests is >= 5
or <= -5. This should give us a two-sided p value of 0.400.
This is where I'm not sure how to proceed. Exactly what does that mean, or
in other words, how would I intrepret these results? What does this p-value
of 0.400 mean, and how would I use it to determine whether or not to reject
the null hypothesis? |
|
|
| Back to top |
|
|
|
| Ray Koopman... |
| Posted: Thu Nov 05, 2009 12:45 pm |
|
|
|
Guest
|
On Nov 4, 8:31 pm, "Zootal" <use... at (no spam) spam.zootal.nospam.com> wrote:
[quote]I'm doing a randomization test on a data set, and I'm not quite sure how to
interpret the results. Suppose I have two data sets with a known sample mean
difference, and I want to do a randomization test to test the null
hypothesis that the sample means are the same, u1 = u2.
Suppose the known sample mean difference is 5, ie. u1 is 25 and u2 is 30. I
do a bunch of tests(some arbitrary large number) and I find that 40% of the
tests give mean differences that are greater than or equal to +/- my known
sample mean of 5, or the sample mean difference in 40% of the tests is >= 5
or <= -5. This should give us a two-sided p value of 0.400.
This is where I'm not sure how to proceed. Exactly what does that mean, or
in other words, how would I intrepret these results? What does this p-value
of 0.400 mean, and how would I use it to determine whether or not to reject
the null hypothesis?
[/quote]
p = .40 means that the data are consistent with the null hypothesis,
that there are no grounds to reject it. |
|
|
| Back to top |
|
|
|
| Ray Koopman... |
| Posted: Thu Nov 05, 2009 5:49 pm |
|
|
|
Guest
|
On Nov 5, 5:49 pm, "Zootal" <use... at (no spam) spam.zootal.nospam.com> wrote:
[quote]"Ray Koopman" <koop... at (no spam) sfu.ca> wrote in message
news:33845df9-823b-4942-aa5c-2e4a18e1b351 at (no spam) f18g2000prf.googlegroups.com...
On Nov 4, 8:31 pm, "Zootal" <use... at (no spam) spam.zootal.nospam.com> wrote:
I'm doing a randomization test on a data set, and I'm not quite sure
how to interpret the results. Suppose I have two data sets with a
known sample mean difference, and I want to do a randomization test
to test the null hypothesis that the sample means are the same,
u1 = u2.
Suppose the known sample mean difference is 5, ie. u1 is 25 and u2
is 30. I do a bunch of tests(some arbitrary large number) and I find
that 40% of the tests give mean differences that are greater than or
equal to +/- my known sample mean of 5, or the sample mean difference
in 40% of the tests is >= 5 or <= -5. This should give us a two-sided
p value of 0.400.
This is where I'm not sure how to proceed. Exactly what does that
mean, or in other words, how would I intrepret these results? What
does this p-value of 0.400 mean, and how would I use it to determine
whether or not to reject the null hypothesis?
p = .40 means that the data are consistent with the null hypothesis,
that there are no grounds to reject it.
So let me take a step back and see if I understand this. I'm using
the same data, but mixing it up into different permutations (isn't it
really different conbinations?). I test those random permutations to
see how far apart those sample means are (this is what a randomization
test is). If I'm doing a two sided t-test, then unless I find my test
statistic to be greater then 97.5 of the random tests, I fail to reject
the null. In this case, my test statistic is only greater then 60%,
so I do not reject the null.
I would then conclude something like there is no evidence that the
sample means are not the same.
Does this sound right?
[/quote]
Almost. I would say that there is insufficient evidence to justify
concluding that the sample means differ. Note that that is *not*
equivalent to saying that the sample means are the same. You do not
*accept* the null hypothesis, you *retain* it as a reasonable
possibility. |
|
|
| Back to top |
|
|
|
| Zootal... |
| Posted: Thu Nov 05, 2009 8:49 pm |
|
|
|
Guest
|
"Ray Koopman" <koopman at (no spam) sfu.ca> wrote in message
news:33845df9-823b-4942-aa5c-2e4a18e1b351 at (no spam) f18g2000prf.googlegroups.com...
[quote]On Nov 4, 8:31 pm, "Zootal" <use... at (no spam) spam.zootal.nospam.com> wrote:
I'm doing a randomization test on a data set, and I'm not quite sure how
to
interpret the results. Suppose I have two data sets with a known sample
mean
difference, and I want to do a randomization test to test the null
hypothesis that the sample means are the same, u1 = u2.
Suppose the known sample mean difference is 5, ie. u1 is 25 and u2 is 30.
I
do a bunch of tests(some arbitrary large number) and I find that 40% of
the
tests give mean differences that are greater than or equal to +/- my
known
sample mean of 5, or the sample mean difference in 40% of the tests is >=
5
or <= -5. This should give us a two-sided p value of 0.400.
This is where I'm not sure how to proceed. Exactly what does that mean,
or
in other words, how would I intrepret these results? What does this
p-value
of 0.400 mean, and how would I use it to determine whether or not to
reject
the null hypothesis?
p = .40 means that the data are consistent with the null hypothesis,
that there are no grounds to reject it.
[/quote]
So let me take a step back and see if I understand this. I'm using the same
data, but mixing it up into different permutations (isn't it really
different conbinations?). I test those random permutations to see how far
apart those sample means are (this is what a randomization test is). If I'm
doing a two sided t-test, then unless I find my test statistic to be greater
then 97.5 of the random tests, I fail to reject the null. In this case, my
test statistic is only greater then 60%, so I do not reject the null.
I would then conclude something like there is no evidence that the sample
means are not the same.
Does this sound right? |
|
|
| Back to top |
|
|
|
| Luis A. Afonso... |
| Posted: Fri Nov 06, 2009 4:00 am |
|
|
|
Guest
|
*************************************
Date: Nov 4, 2009 11:31 PM
Author: Zootal
Subject: Randomization test - need help interpreting results
I'm doing a randomization test on a data set, and I'm not quite sure how to interpret the results. Suppose I have two data sets with a known sample mean difference, and I want to do a randomization test to test the null hypothesis that the sample means are the same, u1 = u2. Suppose the known sample mean difference is 5, ie. u1 is 25 and u2 is 30. I do a bunch of tests (some arbitrary large number) and I find that 40% of the tests give mean differences that are greater than or equal to +/- my known sample mean of 5, or the sample mean difference in 40% of the tests is >= 5 or <= -5. This should give us a two-sided p value of 0.400. This is where I'm not sure how to proceed. Exactly what does that mean, or in other words, how would I intrepret these results? What does this p-value of 0.400 mean, and how would I use it to determine whether or not to reject the null hypothesis?
*****************************************
My Comment
__1___As R. Koopman did note, in this thread, one should say that that is no sufficient evidence (from the data) to reject the Null Hypotheses. We are dealing with a two-tail test and so exactly u1=u2 is very unlike to really be true.
What it´s said above is the current and correct way to express the situation. Then one can keep the Null as eventually true.
However:
As I repeatable said here I do not know why one should be so cautious I the case of a one-tail test of the parameter p, one should prefer, my opinion, the accept/reject option for the test H0: p<=p0, Ha: p>p0. In fact, in this case, the test falls inside the interval (-infinity, critical value] which warrants that p is in fact less or equal to the p0 value at Alpha significance level. Otherwise the interval (critical value, +infinity) is the REJECTION interval: H0 must not be true (at Alpha level)
Similarly for H0: p>=p0, Ha: p<p0. If the test statistics happens to fall in [crit. value, +inf) one can be sure (at the significance level) that the parameter p have, at least, the value p0. Otherwise the interval (-inf., crit. value) leads to accept that p<p0.
__2__In what concerns the Permutation Test for means it is a worth one. But here is a thing to take into account: the two samples must be originated from Distributions with similar variances in order to be randomized (values interchange).
I think it was correct that, for each sample, to divide the each one values by the respective sample rank (greater value minus least) and then to use the current procedure. (Only an idea).
Luis A. Afonso |
|
|
| Back to top |
|
|
|
|
|
All times are GMT - 5 Hours
The time now is Mon Nov 30, 2009 2:55 pm
|
|