I have a question about Simulation Process and Independent Vs Paired
Samples.

Given two related samples, in order to enhance the sensitivity of t-
test,
1 sample t-test should be applied to the two samples differences (null
hypothesis = the differences average is statistically zero), instead
of
applying 2 sample t-test to both the samples separately (null
hypothesis =
both the samples come from the same population).
Simulations made to decide whether to modify or not a certain device,
yield paired (=related) samples (i.e.: before- and after-modification
measurements) spontaneously, however.
Sometimes it happens that the two samples differences result
significantly different from zero (i.e.: 1 sample t-test is
significant),
while both samples seem coming from the same population (i.e.: 2
sample
t-test is not significant).
In these cases, I do not want to modify the device under test because
I am
not able to reject the null hypothesis of the 2 sample t-test. The
"paired
t-test" (i.e.: 1 sample t-test applied to the two samples differences)
suggests to modifying the device under test, however.

What is the proper treatment of paired samples resulting spontaneously
from simulation?

Hanspeter wrote:
I have a question about Simulation Process and Independent Vs Paired
Samples.

Given two related samples, in order to enhance the sensitivity of t-
test,
1 sample t-test should be applied to the two samples differences (null
hypothesis = the differences average is statistically zero), instead
of
applying 2 sample t-test to both the samples separately (null
hypothesis =
both the samples come from the same population).
Simulations made to decide whether to modify or not a certain device,
yield paired (=related) samples (i.e.: before- and after-modification
measurements) spontaneously, however.
Sometimes it happens that the two samples differences result
significantly different from zero (i.e.: 1 sample t-test is
significant),
while both samples seem coming from the same population (i.e.: 2
sample
t-test is not significant).
In these cases, I do not want to modify the device under test because
I am
not able to reject the null hypothesis of the 2 sample t-test. The
"paired
t-test" (i.e.: 1 sample t-test applied to the two samples differences)
suggests to modifying the device under test, however.

What is the proper treatment of paired samples resulting spontaneously
from simulation?


If the simulation generates pairs of observations (one from each
population), and if the paired observations are correlated to each
other, then the paired-difference t-test should be more accurate. The

two sample t-test calculates the variance of the difference in means
under the assumption that the samples are mutually independent, and will
overestimate the variance of the difference in means (reducing power) if
there is a positive correlation between pairs of observations.

On Sat, 26 Apr 2008 16:49:47 -0400, Paul Rubin <rubin@msu.edu> wrote:

Hanspeter wrote:
I have a question about Simulation Process and Independent Vs Paired
Samples.

Given two related samples, in order to enhance the sensitivity of t-
test,
1 sample t-test should be applied to the two samples differences (null
hypothesis = the differences average is statistically zero), instead
of
applying 2 sample t-test to both the samples separately (null
hypothesis =
both the samples come from the same population).
Simulations made to decide whether to modify or not a certain device,
yield paired (=related) samples (i.e.: before- and after-modification
measurements) spontaneously, however.
Sometimes it happens that the two samples differences result
significantly different from zero (i.e.: 1 sample t-test is
significant),
while both samples seem coming from the same population (i.e.: 2
sample
t-test is not significant).
In these cases, I do not want to modify the device under test because
I am
not able to reject the null hypothesis of the 2 sample t-test. The
"paired
t-test" (i.e.: 1 sample t-test applied to the two samples differences)
suggests to modifying the device under test, however.

What is the proper treatment of paired samples resulting spontaneously
from simulation?

If the simulation generates pairs of observations (one from each
population), and if the paired observations are correlated to each
other, then the paired-difference t-test should be more accurate. The

I would go further, and state that if the observations
are correlated, the grouped test is *wrong* -- the
assumption of independence was not met, and the
question then is how much it matters. If you get
different results, then it apparently matters.


two sample t-test calculates the variance of the difference in means
under the assumption that the samples are mutually independent, and will
overestimate the variance of the difference in means (reducing power) if
there is a positive correlation between pairs of observations.

Similarly, a negative correlation will cause the Student's
test to underestimate the variance, and yield too much
power. (Someone once posted a related question concerning
a forced-choice design of this sort, where Left+Right < or = 10.)

I'm in the process of writing simulation software that can generate
paired values (patient histories). Basically, at some point in a
patient's lifecourse, choices can be made regarding treatment. So a
simulated patient can fork into separate simulations when such a point
is reached (and fork again if more than one treatment is involved). The
histories are identical up to the point different treatments can be
applied.

One suggestion that was made to me was to use the same random numbers
for each 'post-fork' simulation. I quickly dismissed that as it does
introduce correlations that depend on the detailed implementation of the
random variate generators.

But it doesn't (to me) seem unreasonable for
the pre-fork histories to be identical.

The simulation model basically models transitions between states, until
eventually a patient dies. If a treatment is assumed to solely prolong
the expected length of time spent in state X until transition to state
Y, then there is an argument that the with / without treatment records
should only differ in the length of time spent in state X. It has also
been argued (not by me) that the same uniform random variate should be
used to generate the length of time in state X (when using the inversion
method of random variate generation). I can easily dismiss the latter
as being unreasonable,

although there is an implicit assumption
(untestable in the real world) however I choose to generate the two
times to transition. There are practical reasons why trying to pair
records after a fork is difficult, because in most cases there are
competing risks, and the effect of a treatment might be that a patient
moves from state X to state Z, rather than state Y.

The point of pairing the records is simply to reduce the computational
time necessary to give a given degree of confidence in model
predictions; typically changes in life expectancy.

Judicious pairing
seems reasonable to me, if the question relates to the predicted effects
on a particular cohort of patients (and not otherwise).

I'm not sure how relevant this is to the OP, but increasing the number
of simulated cases will generally give as much power as you want.

duncan smith wrote:


I'm in the process of writing simulation software that can generate
paired values (patient histories). Basically, at some point in a
patient's lifecourse, choices can be made regarding treatment. So a
simulated patient can fork into separate simulations when such a point
is reached (and fork again if more than one treatment is involved).
The histories are identical up to the point different treatments can
be applied.

One suggestion that was made to me was to use the same random numbers
for each 'post-fork' simulation. I quickly dismissed that as it does
introduce correlations that depend on the detailed implementation of
the random variate generators.

Yes, that's the point; you want the correlations. With a bit of luck,
the clones of the original patient entity produce outputs that are
positively correlated, which allows you to do the ultimate comparisons
with more power for a given sample size or smaller sample size for a
given power. I don't see why you "quickly dismissed" this.

I will note that the use of common random number streams does not
guarantee that positive pairwise correlation, but sometimes it helps.

But it doesn't (to me) seem unreasonable for the pre-fork histories to
be identical.

The simulation model basically models transitions between states,
until eventually a patient dies. If a treatment is assumed to solely
prolong the expected length of time spent in state X until transition
to state Y, then there is an argument that the with / without
treatment records should only differ in the length of time spent in
state X. It has also been argued (not by me) that the same uniform
random variate should be used to generate the length of time in state
X (when using the inversion method of random variate generation). I
can easily dismiss the latter as being unreasonable,

Why? Might not the random times to transition incorporate some factors
that would be intrinsic to a patient (genetics, cardiovascular
conditioning, ...) such that a patient who is, say, tougher than average
would last longer with or without treatment than another patient who is
weaker than average? The use of common random numbers is intended to
"level the playing field" across decision alternatives.

although there is an implicit assumption (untestable in the real
world) however I choose to generate the two times to transition.
There are practical reasons why trying to pair records after a fork is
difficult, because in most cases there are competing risks, and the
effect of a treatment might be that a patient moves from state X to
state Z, rather than state Y.

The point of pairing the records is simply to reduce the computational
time necessary to give a given degree of confidence in model
predictions; typically changes in life expectancy.

Correct.

Judicious pairing seems reasonable to me, if the question relates to
the predicted effects on a particular cohort of patients (and not
otherwise).

I'm not sure how relevant this is to the OP, but increasing the number
of simulated cases will generally give as much power as you want.

Assuming you're patient (no pun intended).

/Paul

In the model I'm dealing with there are competing risks. Once a
treatment is introduced the patient might embark on a different series
of transitions before death. Reducing the risk of transition from state
X to state Y might result in a transition to state Z instead. I don't
want the time spent in state Y (without treatment) to be correlated with
the time spent in state Z (with treatment). (This violates the
assumptions of the model.) Even in the case where there's only one
possible transition I wouldn't use the same uniform random variate to
generate the transition time, because that would imply that every
patient would respond positively to the treatment (assuming it is on
average effective). That's an assumption I'm not 100% happy with. I
prefer the weaker assumption that the hazard is reduced in the same way
for each patient, but that the times to transition in the treatment /
non-treatment arms of the simulation are independent given the relevant
hazard function.

duncan smith wrote:

In the model I'm dealing with there are competing risks. Once a
treatment is introduced the patient might embark on a different series
of transitions before death. Reducing the risk of transition from
state X to state Y might result in a transition to state Z instead. I
don't want the time spent in state Y (without treatment) to be
correlated with the time spent in state Z (with treatment). (This
violates the assumptions of the model.) Even in the case where
there's only one possible transition I wouldn't use the same uniform
random variate to generate the transition time, because that would
imply that every patient would respond positively to the treatment
(assuming it is on average effective). That's an assumption I'm not
100% happy with. I prefer the weaker assumption that the hazard is
reduced in the same way for each patient, but that the times to
transition in the treatment / non-treatment arms of the simulation are
independent given the relevant hazard function.

I'll confess that I'm having a bit of trouble parsing this, since I
don't have your model to look at. Let's say that Patient X enters the
simulation and, at some point, you clone him, so that X continues on to
get one treatment and X' gets a different treatment. If there is no
medical reason to believe responses to the two treatments would be
correlated, then there's no reason to use common random numbers in
generating the treatment outcomes.

However, let's say that some characteristics of the patient (age,
severity of illness, whether the patient is left-handed) affect both
treatment outcomes in similar ways (such as increasing likelihood of
transition to the next plane, as it were). If those characteristics are
explicit parameters in the hazard function, you can attach them to the
patient as attributes and, as each patient (or clone) moves through the
model, plug them into the relevant functions. Perhaps not every common
causal factor gets picked up as an attribute, though. That's an
argument for using common random numbers (with different hazard
functions) for patient X and clone X' ... but only when the random
numbers are being used to generate outcomes where the common causal
factors would indicate common "tendencies" (meaning toward the
successful or unsuccessful side of the outcome distribution).

Disclaimer: I'm used to dealing with queueing models. There common
random numbers are frequently appropriate. If I'm comparing two
possible layouts for a fast food joint, and in simulating the first
variant I manage to generate a customer who can't make up his mind and
gums up the queue, I want a similar amount of dithering by that
customer's clone in the second variant.

/Paul