Hi,
Suppose I've a regression model as follows:
I'm regressing per capita consumption on level of income, age & family
size in Case 1.
I'm regressing per capita consumption again on income, age & level of
education (say) in Case 2.
Will the beta coefficient of income & age will remain same in both
Case 1 & Case 2? If no, why?
Arijit.

The usual interpretation of the regression coefficients is as the
partial derivative of the dependent variable with respect to the
particular independent variable. This is not affected by collinearity
between the predictors.
-Dick Startz

On Jun 26, 4:09 am, sagar <ariji... at (no spam) gmail.com> wrote:
Hi,
Suppose I've a regression model as follows:
I'm regressing per capita consumption on level of income, age & family
size in Case 1.
I'm regressing per capita consumption again on income, age & level of
education (say) in Case 2.
Will the beta coefficient of income & age will remain same in both
Case 1 & Case 2? If no, why?
Arijit.

They may not stay the same if there is collinearity between your
predictor variables. Check out their covariance matrix to be sure
that's what's happening. If you are using R there is also a tool vif
in the DAAG package that estimates the variance inflation of your
coefficient estimates due to collinearity between the predictors. It
means that the magnitude of your coefficients don't have a ready
physical interpretation due to intertwined effects. You can get
around the variance inflation of coefficient estimates by transforming
onto principal components axes, which should give you better
predictive ability but you still won't have physically interpretable
coefficients.

On Fri, 27 Jun 2008 07:18:26 -0700 (PDT), lw <leviwaldron at (no spam) gmail.com
wrote:

On Jun 26, 4:09 am, sagar <ariji... at (no spam) gmail.com> wrote:
Hi,
Suppose I've a regression model as follows:
I'm regressing per capita consumption on level of income, age & family
size in Case 1.
I'm regressing per capita consumption again on income, age & level of
education (say) in Case 2.
Will the beta coefficient of income & age will remain same in both
Case 1 & Case 2? If no, why?
Arijit.
They may not stay the same if there is collinearity between your
predictor variables. Check out their covariance matrix to be sure
that's what's happening. If you are using R there is also a tool vif
in the DAAG package that estimates the variance inflation of your
coefficient estimates due to collinearity between the predictors. It
means that the magnitude of your coefficients don't have a ready
physical interpretation due to intertwined effects. You can get
around the variance inflation of coefficient estimates by transforming
onto principal components axes, which should give you better
predictive ability but you still won't have physically interpretable
coefficients.

The usual interpretation of the regression coefficients is as the
partial derivative of the dependent variable with respect to the
particular independent variable. This is not affected by collinearity
between the predictors.
-Dick Startz

Richard Startz wrote:
On Fri, 27 Jun 2008 07:18:26 -0700 (PDT), lw <leviwaldron at (no spam) gmail.com
wrote:

On Jun 26, 4:09 am, sagar <ariji... at (no spam) gmail.com> wrote:
Hi,
Suppose I've a regression model as follows:
I'm regressing per capita consumption on level of income, age & family
size in Case 1.
I'm regressing per capita consumption again on income, age & level of
education (say) in Case 2.
Will the beta coefficient of income & age will remain same in both
Case 1 & Case 2? If no, why?
Arijit.
They may not stay the same if there is collinearity between your
predictor variables. Check out their covariance matrix to be sure
that's what's happening. If you are using R there is also a tool vif
in the DAAG package that estimates the variance inflation of your
coefficient estimates due to collinearity between the predictors. It
means that the magnitude of your coefficients don't have a ready
physical interpretation due to intertwined effects. You can get
around the variance inflation of coefficient estimates by transforming
onto principal components axes, which should give you better
predictive ability but you still won't have physically interpretable
coefficients.

The usual interpretation of the regression coefficients is as the
partial derivative of the dependent variable with respect to the
particular independent variable. This is not affected by collinearity
between the predictors.
-Dick Startz

It's the partial derivative of the conditional mean of the d.v. given
the i.v.s. If you change the i.v.s, you're changing which conditional
mean you're talking about.

Suppose I have d.v. Y and possible i.v.s X, Z and W. For simplicity,
we'll assume they all have mean zero and std. dev. 1. Let's also assume
that X and Z are correlated by X and W are uncorrelated. If I look at Y
~ X + Z v. Y ~ X + W, I get equations

Y = b1*X + b2*Z + noise v. Y = c1*X + c2*W + noise

where as usual we'll assume the noise is uncorrelated to everybody in
sight (and the noises in the two equations are different). Let <,
denote the covariance operator; then the first equation gives me

Y,X> = b1 + b2*<Z,X

and the second gives me

Y,X> = c1

exploiting the lack of correlation between X, W and the noises. So b1 =
c1 iff Z and X are uncorrelated; I assumed they were correlated, so I
expect different coefficients for X in the two equations.

/Paul

On Jun 27, 10:47 am, Richard Startz <richardstar... at (no spam) comcast.net
wrote:
The usual interpretation of the regression coefficients is as the
partial derivative of the dependent variable with respect to the
particular independent variable. This is not affected by collinearity
between the predictors.
-Dick Startz

It would be nice to be able to always interpret the regression
coefficients this way, but it only makes sense if the predictors are
actually independent. What does a partial derivative with respect to
one variable mean if other variables necessarily change in correlation
with that variable?

I suppose I should have said multicollinearity, ie where two or more
predictor variables in a multiple regression model are highly
correlated, but it certainly can have a large effect on coefficient
estimates. Maindonald (2003) has a stark example of this in its
chapter on multiple linear regression (6.7.1), or you could also see
the Wikipedia page (http://en.wikipedia.org/wiki/Multicollinearity).
As a simple example, imagine predicting the height of people by using
measurements of the length of each of their feet. You might expect
the coefficient of one foot to simply double if you removed the other
foot from the model, but in reality both estimated coefficients would
be wildly inaccurate due to their high degree of collinearity. If you
removed one foot from the model it may well tell you that someone has
a negative height based on the measurement of one of their feet.

Ref. Maindonald and Braun (2003). Data Analysis and Graphics Using
R, An Example-based Approach. Cambridge University Press, New York NY.

On Jun 27, 10:47 am, Richard Startz <richardstar... at (no spam) comcast.net
wrote:
The usual interpretation of the regression coefficients is as the
partial derivative of the dependent variable with respect to the
particular independent variable. This is not affected by collinearity
between the predictors.
-Dick Startz

It would be nice to be able to always interpret the regression
coefficients this way, but it only makes sense if the predictors are
actually independent. What does a partial derivative with respect to
one variable mean if other variables necessarily change in correlation
with that variable?

I suppose I should have said multicollinearity, ie where two or more
predictor variables in a multiple regression model are highly
correlated, but it certainly can have a large effect on coefficient
estimates. Maindonald (2003) has a stark example of this in its
chapter on multiple linear regression (6.7.1), or you could also see
the Wikipedia page (http://en.wikipedia.org/wiki/Multicollinearity).
As a simple example, imagine predicting the height of people by using
measurements of the length of each of their feet. You might expect
the coefficient of one foot to simply double if you removed the other
foot from the model, but in reality both estimated coefficients would
be wildly inaccurate due to their high degree of collinearity.

If you
removed one foot from the model it may well tell you that someone has
a negative height based on the measurement of one of their feet.

Ref. Maindonald and Braun (2003). Data Analysis and Graphics Using
R, An Example-based Approach. Cambridge University Press, New York NY.

That "but" is too strong.  I *do*  expect the coefficients of each
foot to be the same, and about half the single-foot value,  because
I've seen that sort of thing.  (Well, I'm extrapolating... with
Left-Right feet, I would never start with both, but start with
predictors that used the "average" and the "difference".)

Both estimated coefficients *might*  be wildly inaccurate.
Because the 2-dimensional confidence ellipse is long and narrow,
there is a great range of pairs-of-coefficients which give
very-nearly the same predictions.  

x1 <- 1:10
x2 <- x1+rnorm(10,sd=0.01)
y <- x1+x2+rnorm(10,sd=1)
x1
[1] 1 2 3 4 5 6 7 8 9 10
x2
[1] 0.987971 1.985931 2.995624 3.991472 4.983735 5.987358

cor(x1,x2)
[1] 0.999993
y
[1] 2.873586 4.688333 4.711484 6.487986 10.227832 12.632284

junklm <- lm(y~x1+x2)
summary(junklm)

betterlm <- lm(y~I(x1+x2)+I(x1-x2))
summary(betterlm)

On Jun 27, 4:14 pm, RichUlrich <rich.ulr... at (no spam) comcast.net> wrote:
That "but" is too strong.  I *do*  expect the coefficients of each
foot to be the same, and about half the single-foot value,  because
I've seen that sort of thing.  (Well, I'm extrapolating... with
Left-Right feet, I would never start with both, but start with
predictors that used the "average" and the "difference".)

Both estimated coefficients *might*  be wildly inaccurate.
Because the 2-dimensional confidence ellipse is long and narrow,
there is a great range of pairs-of-coefficients which give
very-nearly the same predictions.  

I don't know, is it too strong? As Dick S. pointed out, the standard
errors of the coefficients will be very large, so if they turned out
to be about the same and about half of the single-foot value, it would
just be lucky. Here's a contrived and rather extreme example I ran in
R. x1 and x2 are nearly the same with a small amount of noise as
difference (Pearson correlation of 0.999993, and y is just the sum of
the two with more noise added. A linear model of the form y ~ x1+x2
tends to result on one large positive coefficient and one large
negative coefficient, adding up to about 2. In this case the
coefficients are 37 and -35, with se of 29.

[snip, detail]

On Sat, 28 Jun 2008 18:46:28 -0700 (PDT), lw <leviwaldron at (no spam) gmail.com
wrote:

On Jun 27, 4:14 pm, RichUlrich <rich.ulr... at (no spam) comcast.net> wrote:
That "but" is too strong.  I *do*  expect the coefficients of each
foot to be the same, and about half the single-foot value,  because
I've seen that sort of thing.  (Well, I'm extrapolating... with
Left-Right feet, I would never start with both, but start with
predictors that used the "average" and the "difference".)

Both estimated coefficients *might*  be wildly inaccurate.
Because the 2-dimensional confidence ellipse is long and narrow,
there is a great range of pairs-of-coefficients which give
very-nearly the same predictions.  

I don't know, is it too strong? As Dick S. pointed out, the standard
errors of the coefficients will be very large, so if they turned out
to be about the same and about half of the single-foot value, it would
just be lucky. Here's a contrived and rather extreme example I ran in
R. x1 and x2 are nearly the same with a small amount of noise as
difference (Pearson correlation of 0.999993, and y is just the sum of
the two with more noise added. A linear model of the form y ~ x1+x2
tends to result on one large positive coefficient and one large
negative coefficient, adding up to about 2. In this case the
coefficients are 37 and -35, with se of 29.

[snip, detail]

Okay, here is a more careful explanation of my point.

Given two measures that are "almost the same", the smart way
to model them is to replace them with their average and difference.
Is the difference important? In your model, it *is*.

If it shows up that way, it is something to take special note of.
You figure out what it means by... taking the difference, and
looking at what that derived score means.

That is not likely to happen in the sort of data that I typically
model, where each variable has its own bit of error, and the
reliabilities combine to make the sum almost ideal. (Data with
small Ns and outliers can behave otherwise.) These are the
data that split the loadings in half, though not very precisely.

On Sun, 29 Jun 2008 20:06:56 -0400, RichUlrich
rich.ulrich at (no spam) comcast.net> wrote:

On Sat, 28 Jun 2008 18:46:28 -0700 (PDT), lw <leviwaldron at (no spam) gmail.com
wrote:

On Jun 27, 4:14 pm, RichUlrich <rich.ulr... at (no spam) comcast.net> wrote:
That "but" is too strong.  I *do*  expect the coefficients of each
foot to be the same, and about half the single-foot value,  because
I've seen that sort of thing.  (Well, I'm extrapolating... with
Left-Right feet, I would never start with both, but start with
predictors that used the "average" and the "difference".)

Both estimated coefficients *might*  be wildly inaccurate.
Because the 2-dimensional confidence ellipse is long and narrow,
there is a great range of pairs-of-coefficients which give
very-nearly the same predictions.  

I don't know, is it too strong? As Dick S. pointed out, the standard
errors of the coefficients will be very large, so if they turned out
to be about the same and about half of the single-foot value, it would
just be lucky. Here's a contrived and rather extreme example I ran in
R. x1 and x2 are nearly the same with a small amount of noise as
difference (Pearson correlation of 0.999993, and y is just the sum of
the two with more noise added. A linear model of the form y ~ x1+x2
tends to result on one large positive coefficient and one large
negative coefficient, adding up to about 2. In this case the
coefficients are 37 and -35, with se of 29.

[snip, detail]

Okay, here is a more careful explanation of my point.

Given two measures that are "almost the same", the smart way
to model them is to replace them with their average and difference.
Is the difference important? In your model, it *is*.

If it shows up that way, it is something to take special note of.
You figure out what it means by... taking the difference, and
looking at what that derived score means.

That is not likely to happen in the sort of data that I typically
model, where each variable has its own bit of error, and the
reliabilities combine to make the sum almost ideal. (Data with
small Ns and outliers can behave otherwise.) These are the
data that split the loadings in half, though not very precisely.

Replacing two variables with their average and difference doesn't
change the regression at all, although it may make for a more
convenient interpretation.

If the estimated coefficients on the original variables (x1 and x2)
are b1 and b2, and the estimated coefficients on (x1-x2) and (x1+x2)/2
are c1 and c2, then

b1 = c1+c2/2 and b2 = -c1+c2/2

The R-squared and residuals will be identical.

-Dick Startz

Hi,
Suppose I've a regression model as follows:
I'm regressing per capita consumption on level of income, age & family
size in Case 1.
I'm regressing per capita consumption again on income, age & level of
education (say) in Case 2.
Will the beta coefficient of income & age will remain same in both
Case 1 & Case 2? If no, why?
Arijit.

Hi,
Suppose I've a regression model as follows:
I'm regressing per capita consumption on level of income, age & family
size in Case 1.
I'm regressing per capita consumption again on income, age & level of
education (say) in Case 2.
Will the beta coefficient of income & age will remain same in both
Case 1 & Case 2? If no, why?
Arijit.