Main Page | Report this Page
 
   
Science Forum Index  »  Statistics - Education Forum  »  ICC in mixed vs. fixed design
Page 1 of 1    
Author Message
Haris
Posted: Wed Jan 31, 2007 1:13 pm
Guest
I need to estimate reliability of a measurement across time.
Normally, a regular correlation or an ICC would do, but in my case I
have:
a. more than two measurements per person and
b. unequal number of measurements per person (1-Cool.

So, the question is how comparable are ICCs from the fixed design
scenario (assuming two measurements per person and no missing data)
and ICC from a random intercept mixed model (intercept variance /
[total variance or intercept variance + residual unexplained
variance]).

There are two interpretations of the ICC in the case of a mixed
model:
1. Proportion of variance between subjects
2. Correlation between two randomly drawn observations from the
same subject.

Both of these sound acceptable as definitions of measurement
reliability, but I get very different results if I compute ICC from a
simple two-time dataset in fixed vs. random intercept models. Also,
ICC from a mixed model cannot be negative, while ICC from the fixed
model can be negative and I've also seen them greater than and less
than 1.

I don't know much about fixed model ICC as an estimate of
reliability. Any comments? Recommended reading about ICC in the
fixed design world?

Thanks.
Ray Koopman
Posted: Wed Jan 31, 2007 3:13 pm
Guest
Haris wrote:
Quote:
I need to estimate reliability of a measurement across time.
Normally, a regular correlation or an ICC would do, but in my case I
have:
a. more than two measurements per person and
b. unequal number of measurements per person (1-Cool.

So, the question is how comparable are ICCs from the fixed design
scenario (assuming two measurements per person and no missing data)
and ICC from a random intercept mixed model (intercept variance /
[total variance or intercept variance + residual unexplained
variance]).

There are two interpretations of the ICC in the case of a mixed
model:
1. Proportion of variance between subjects
2. Correlation between two randomly drawn observations from the
same subject.

Both of these sound acceptable as definitions of measurement
reliability, but I get very different results if I compute ICC from a
simple two-time dataset in fixed vs. random intercept models. Also,
ICC from a mixed model cannot be negative, while ICC from the fixed
model can be negative and I've also seen them greater than and less
than 1.

I don't know much about fixed model ICC as an estimate of
reliability. Any comments? Recommended reading about ICC in the
fixed design world?

Thanks.

Do you have a two-way design (People x Time) with missing data, or
a one-way design with no matching of the time points across people?
Haris
Posted: Thu Feb 01, 2007 12:17 pm
Guest
Thanks for all the replies. I found Shrout & Fleiss (1979)
particularly informative and very helpful. Their ANOVA models are
equivalent to mixed models with different specifications:

(1,1) is equivalent to intercept/total variance ratio in the random
target intercept model
(2,1) is equivalent to intercept / (total variance ratio in the mixed
model with random effects of BOTH Targets and Raters
(3,1) is equivalent to intercept / residual variance ratio in the
mixed model with random effects of BOTH Targets and Raters--variance
due to Raters is factored out.

In my case i have repeated assessments for n Targets. No raters are
ever involved--these are multiple repeated measurements off of ECG.

Follow-up is structured but, in reality, the dates do not strictly
correspond to the set schedule. Therefore, it seems to me that a
simple (1,1) scenario is probably the most appropriate--there really
is no second systematic source of variance like same raters.

On the other hand, systematic increase or decline in the measurements
with time should be factored out. In theory these measurements should
stay stable with age; however, they are supposed to increase with
declining health. If a patient gets sicker, his/her measurements are
not expected to stay stable. In the data, some patients get better
and some get worse. These changes should be factored out since they
represent systematic, not random variability.

I think this should be accomplished with a FIXED effect of TIME
introduced as a slope rather than a categorical covariate
corresponding each individual visit. I tried this in a random
intercept-and-slope model and fixed slope is not significant. There
is substantial variation between random slopes from patient to
patient, though. I am not sure whether it is appropriate to factor
out variance due to random slopes. We have some rough measures of the
overall health of the patients so those may be better covariates than
time. The point is that the quantity under question does not stay
constant across all patients: in some it goes up in some it goes down
and this change may not necessarily be due to unreliability of the
measure.

Any ideas are appreciated.
 
Page 1 of 1       All times are GMT - 5 Hours
The time now is Sat Sep 06, 2008 1:23 pm