Hi all,

I have performed an experiment in which subjects could perform task A
or B. I wanted to compare if a performance variable V differs between
the tasks. The experiment was conceived as a between subject design,
with a group of subjects performing A and another group performing B.
However, a large minority of subjects spontaneously chose to perform
both tasks (although they almost always performed the desired task
first). (They were children and the tasks are fun, hence why they did
more than asked).

I could stick to my original plan and just include the assigned task
for each individual. But I would like to make use of the extra data.
But I can't do a simple paired samples analysis because for most of
the subjects one or the other task is missing.

I'm pretty sure it would be valid in a sense to use all the data and
do an ANOVA including individual as a random factor. I don't like that
solution because I was hoping to use a permutation test (because my
data is non-normal) and I don't think there is a widely used way of
including random factors in permutation tests.

What I would like to do is just use all the data and use a simple two-
sample permutation test. Some instinct is telling me this is wrong
because it is mixing up a between subjects and cross subjects design.

But when I really think about it, I can't see why doing this could
bias my test at all. OK, so the data points in the two samples would
not be completely independent - if subject S's score for V in task A
is high, perhaps it tends to be high in B also. But such effects would
hardly make it less valid if I find a difference between the two
groups.

So do you think I can do this?

Cheers,

Ben

On Apr 30, 12:24 pm, amorphia <spam.onto...@gmail.com> wrote:
Hi all,

I have performed an experiment in which subjects could perform task A
or B. I wanted to compare if a performance variable V differs between
the tasks. The experiment was conceived as a between subject design,
with a group of subjects performing A and another group performing B.
However, a large minority of subjects spontaneously chose to perform
both tasks (although they almost always performed the desired task
first). (They were children and the tasks are fun, hence why they did
more than asked).

I could stick to my original plan and just include the assigned task
for each individual. But I would like to make use of the extra data.
But I can't do a simple paired samples analysis because for most of
the subjects one or the other task is missing.

I'm pretty sure it would be valid in a sense to use all the data and
do an ANOVA including individual as a random factor. I don't like that
solution because I was hoping to use a permutation test (because my
data is non-normal) and I don't think there is a widely used way of
including random factors in permutation tests.

What I would like to do is just use all the data and use a simple two-
sample permutation test. Some instinct is telling me this is wrong
because it is mixing up a between subjects and cross subjects design.

But when I really think about it, I can't see why doing this could
bias my test at all. OK, so the data points in the two samples would
not be completely independent - if subject S's score for V in task A
is high, perhaps it tends to be high in B also. But such effects would
hardly make it less valid if I find a difference between the two
groups.

So do you think I can do this?

Cheers,

Ben

I would apply the KISS principle. I.e., I would do the intended
between-groups analysis on the assigned task data. Then I would do a
within-Ss analysis on those who did both tasks.

If I did want to also try an analysis that used all the data, I'd look
into a multilevel model approach. You didn't say what software you're
using, but in SPSS, for example, the MIXED command syntax would look
something like this, with GROUP referring to assigned task:

MIXED
Y BY group task
/FIXED = group task task*group | SSTYPE(3)
/PRINT = SOLUTION
/REPEATED = task | SUBJECT(ID) COVTYPE(UN)
.

The data are in the long format for MIXED (i.e., one row per
observation of Y, not one row per subject), so all of the available
data are used.