I am running LR. There are high correlations between some of my
continuous predictors. The tests for colllinearity are positive. My
question may be a silly one......what is the difference between
collinearity and confounding. It seems like they would be the same
thing. However, one is encouraged to keep confounders in the model and
encouraged to drop collinear variables. What am I missing. I have
mostly handled my analysis by dropping one where I see evidence.

Mcap

On 11/30/2006 2:59 PM, mcam54@hotmail.com wrote:
I am running LR. There are high correlations between some of my
continuous predictors. The tests for colllinearity are positive. My
question may be a silly one......what is the difference between
collinearity and confounding. It seems like they would be the same
thing. However, one is encouraged to keep confounders in the model and
encouraged to drop collinear variables. What am I missing. I have
mostly handled my analysis by dropping one where I see evidence.

Mcap


Confounding, usually used in reference to a designed experiment, means
that two or more columns of the X matrix are linearly dependent.

"Exact collinearity" means that two variables of the X matrix are
linearly dependent.

"Exact collinearity" means that two (OR MORE) variables of the
X matrix are linearly dependent.

Usually, we talk of high collinearity to mean that
two variables have a high correlation with one another (it is judgment
what "high correlation" means).

With regards to dropping a variable when you see collinearity, how do
you know which one to drop? Suppose the one you choose to drop is the
actual cause of changes in Y? How would you know?

I am not aware of advice that says to keep confounders in the model
(in fact, in a designed experiment, you have to do the opposite -- you
cannot estimate all effects when things are confounded), and I am
opposed to dropping one or more collinear variables simply because it
has high collinearity, without knowing which of the collinear
variables is the cause.

Since in much of my work, we do NOT know a priori causal variables,
collinearity presents a problem, and the solution I most often adopt
is to use Partial Least Squares (PLS) regression instead of Ordinary
Least Squares regression. In PLS, you do not drop regressors in the
presence of collinearity, and according to a paper by Frank and
Friedman in Technometrics (1993), PLS performs better than OLS in the
presence of collinearity. "Performs better" means that PLS has a lower
MSE of predicted values and a lower MSE of regression coefficients
than does OLS (including stepwise methods).

--
Paige Miller
pmiller5@rochester.rr.com

It's nothing until I call it -- Bill Klem, NL Umpire
If you get the choice to sit it out or dance,
I hope you dance -- Lee Ann Womack

Paige Miller wrote:
[snip]

There are MANY threads in sci.stat.math on the keywords
"linear indepdence" and "multicollinearity" that had extended
discussions about what they mean and how they are detected
and remedied.

In particular, one CANNOT draw causal inference from correlations
or regression without well-controlled experiments.

On 2 Dec 2006 15:18:52 -0800, "Reef Fish"
large_nassua_grouper@yahoo.com> wrote:


Paige Miller wrote:
[snip]
RF
There are MANY threads in sci.stat.math on the keywords
"linear indepdence" and "multicollinearity" that had extended
discussions about what they mean and how they are detected
and remedied.

Keep in mind, when you read, that Reef Fish Bob
preserves the strict definition of 'multicollinearity' that is
not of much use any more, and distorts any questions that
entail the more general definition.

[snip, slur]

In particular, one CANNOT draw causal inference from correlations
or regression without well-controlled experiments.

Well, you do need to do multiple studies, and know a lot
about your variables. That's something Bob rejects.

When I cited Fred Mosteller's role in major observational
studies, Reef Fish Bob pretended to concede more, but
he seems to be back to his old research nihilism.

not like that term, but it seems to represent him better than
his own hedging does.

--
Rich Ulrich, wpilib@pitt.edu
http://www.pitt.edu/~wpilib/index.html

I am running LR. There are high correlations between some of my
continuous predictors. The tests for colllinearity are positive. My
question may be a silly one......what is the difference between
collinearity and confounding. It seems like they would be the same
thing. However, one is encouraged to keep confounders in the model and
encouraged to drop collinear variables. What am I missing. I have
mostly handled my analysis by dropping one where I see evidence.

Mcap

Richard Ulrich wrote:
On 2 Dec 2006 15:18:52 -0800, "Reef Fish"
large_nassua_grouper@yahoo.com> wrote:
[snip]

Keep in mind, when you read, that Reef Fish Bob
preserves the strict definition of 'multicollinearity' that is
not of much use any more, and distorts any questions that
entail the more general definition.
RF
Strict definition of multicollinearity? The trouble with Richard
Ulrich
is that he always MISREPRESENT without citing anything I said.

Multicollinearity in statistical usage is an ILL-DEFINED term
meaning "almost linearly dependent". In mathematical that is
NOT a well defined term where as "linear indepdence" or
"linear dependecne" are.

In particular, one CANNOT draw causal inference from correlations
or regression without well-controlled experiments.
RU
Well, you do need to do multiple studies, and know a lot
about your variables. That's something Bob rejects.
RF
At least you QUOTED me, and that is a correct statement
ANY TIME. If did NOT say anything about multiple
experiments -- sometimes one well designed experiment
will do. It also did not say they always succeeed in the
inference -- that's why there is no absolutely certainty in
any result of statistical hypothesis testing, but there are
two types of errors.

When I cited Fred Mosteller's role in major observational
studies, Reef Fish Bob pretended to concede more, but
he seems to be back to his old research nihilism.
RF
That was the experiment you cited many times that I merely
ignore, for the reason cited in the PRECEDING Paragraph.
I wasn't familiar with the experiment so I keep my mouth
SHUT (unlike Ulrich who opens and inserts foot in mouth
whether he knows what he is talking or not).

"So? ALL statisticians make PLENTY of studies on OBSERVATIONAL
data -- the condition is NOT to draw unwarranted conclusions from them."

Suppose I take your word that it was ONE of the "well-
designed experiment that failed" -- first of all I don't have
anything to judge if the failure was due to a fault in the
design or just pure chance (as in Type II error).

He does
not like that term, but it seems to represent him better than
his own hedging does.
RF
That's the typical Richard Ulrich ad hominem phrase without
any statistical substance.

[snip, end]

[cross-posted from sci.stat.edu, since Bob has been slurring
me everywhere lately.]

[snip]
RU
Keep in mind, when you read, that Reef Fish Bob
preserves the strict definition of 'multicollinearity' that is
not of much use any more, and distorts any questions that
entail the more general definition.

RF
Strict definition of multicollinearity? The trouble with Richard
Ulrich
is that he always MISREPRESENT without citing anything I said.

Multicollinearity in statistical usage is an ILL-DEFINED term
meaning "almost linearly dependent". In mathematical that is
NOT a well defined term where as "linear indepdence" or
"linear dependecne" are.

It sounds to me like RF does "preserve the strict definition",
exactly as I said. The part about 'distorting the questions
that arise' is judgmental, but not too harsh, I think.

In statistical usage, Multicollinearity has come to mean that
aspect indicated by (for instance) a high Variance Inflation
Factor (VHF).

Many people have written about that. It is
useful for finding variables that are redundant, and, down the
road, important in working toward equations that are going
to replicate. It is not perfectly "well-defined" but it is usable.

Bob is sort of alone in disparaging those things. I don't quote
him here, but I think I don't misrepresent him. If he *does*
want to endorse those things, I will be surprised, and pleased.

[...]
RF
In particular, one CANNOT draw causal inference from correlations
or regression without well-controlled experiments.
RU
Well, you do need to do multiple studies, and know a lot
about your variables. That's something Bob rejects.
RF
At least you QUOTED me, and that is a correct statement
ANY TIME. If did NOT say anything about multiple
experiments -- sometimes one well designed experiment
will do. It also did not say they always succeeed in the
inference -- that's why there is no absolutely certainty in
any result of statistical hypothesis testing, but there are
two types of errors.

Aha! Here is one source of Bob's errors. He does not say
anything about multiple studies.

Everyone talking about
drawing any sort of firm conclusions will talk about multiple
studies, or, as I do, about knowing a lot about the variables.
At other times, I have mentioned supporting biological
knowledge (studies?) and so on.

How can Bob miss this, through dozens of posts on the subject?
- He does not read well.

The conclusions from one single statistical analysis are limited.
Social science parodies frequently close with, "More study is needed."

But a single, serendipitous, uncontrolled study *can* answer a
single open question, if that is the state of present knowledge.

RU
When I cited Fred Mosteller's role in major observational
studies, Reef Fish Bob pretended to concede more, but
he seems to be back to his old research nihilism.

Multiple studies in Mosteller's career.
September 15, since RF's memory fails him -
=====start quote from post -

Here are some of the contributions of Reef Fish's buddy,
C. Frederick Mosteller, as cited in the NY Times, in its July 27
obituary. He seems to been a scientist making extensive use
of observational data, with several notable citations below.

These studies used multiple variables, and probably OLS regression.

"In the late 1950's Dr. Mosteller assisted in analyzing data from a
large clinical study looking at the anesthetic halothane, which was
suspected of causing fatal liver damage in some patients. The
analysis showed no evidence that halothane was more dangerous than
other forms of anesthesia.

"He worked with Daniel Patrick Moynihan... on studies looking at the
impact of home life on children's performance in school. They
argued that raising families out of poverty would have a greater
educational impact than pouring money directly into schools.
[ ... snip, analysis of prose, Madison writing disputed Federalist
Papers.]

"In the 1970's, Dr. Mosteller worked on studies that questioned
whether the benefits of some surgical procedures were worth their
costs. ...
[snip, more detail.]

=====end quote

In the middle of a long reply to that post, also on Sept. 15,
Reef Fish commented -
http://groups.google.com/group/sci.stat.math/msg/8f91f8a002357c3a
RF
"So? ALL statisticians make PLENTY of studies on OBSERVATIONAL
data -- the condition is NOT to draw unwarranted conclusions from them."

The problem here, and why I suspected that he was hedging, is
that Bob has never accepted anything concrete as an indicator
of "warranted conclusions."

RF
Suppose I take your word that it was ONE of the "well-
designed experiment that failed" -- first of all I don't have
anything to judge if the failure was due to a fault in the
design or just pure chance (as in Type II error).

? I don't know what Bob is talking about. I never mentioned
any well-designed experiment. Another mis-reading by Bob?

RU > > [referring to nihilism]
He does
not like that term, but it seems to represent him better than
his own hedging does.
RF
That's the typical Richard Ulrich ad hominem phrase without
any statistical substance.

[snip, end]


I call that nihilism. Bob may correct my view if I have
misrepresented him

quoting RF
That was the experiment you cited many times that I merely
ignore, for the reason cited in the PRECEDING Paragraph.
I wasn't familiar with the experiment so I keep my mouth
SHUT (unlike Ulrich who opens and inserts foot in mouth
whether he knows what he is talking or not).

Multiple studies in Mosteller's career.
September 15, since RF's memory fails him -
=====start quote from post -

Here are some of the contributions of Reef Fish's buddy,
C. Frederick Mosteller, as cited in the NY Times, in its July 27
obituary. He seems to been a scientist making extensive use
of observational data, with several notable citations below.
These studies used multiple variables, and probably OLS regression.

"In the late 1950's Dr. Mosteller assisted in analyzing data from a
large clinical study looking at the anesthetic halothane, which was
suspected of causing fatal liver damage in some patients. The
analysis showed no evidence that halothane was more dangerous than
other forms of anesthesia.

Richard Ulrich wrote:
[cross-posted from sci.stat.edu, since Bob has been slurring
me everywhere lately.]
RF
Very interesting statement, coming from Richard Ulrich. I can
predict, with 100% certainty that Richard will be repeating whatever
error I had point out which he called it "slur"?

The best thing in this case for Richard Ulrich is that he actually
MADE some statement of statistical content so that I could point
out specifically what his errors ARE. When he makes ad
hominem attacks, either misrepresenting my position or without
quoting me, there is nothing of SPECIFIC statistical substance I
can say in reply other than Richard Ulrich has been a proven
Quack in all regression matters (that's a statement of FACT that
had been widely documented in the archives).

My post appeared on Date: Sun, Dec 3 2006 4:37 pm
more than 6 hours before. You should have replied to
THAT post instead of wasting bandwidth making

That is on the subject of "remote cause" vs "proximity cause"
and the various other meanings of "cause". They are all
about LOGIC, clearly defined and illustrated in Copi's book
on "Introduction to Logic" which many statisticians lack the
knowledge and ALL statistical Quacks in social sciences
lack when they use correlations to make causal inference.

I am SURE I've mentioned Copi's book and the matter of
"cause" to Richard in one of the many corrections of his
errors in using regression to make causal inference.

But a single, serendipitous, uncontrolled study *can* answer a
single open question, if that is the state of present knowledge.
RF
I said that in the post that you question in this one. I explained
it above, repeating what's in the "case study" post, that
multiple studies is NEITHER necessary (meaning a single
study may be sufficient) NOR sufficient (meaning multiple
studies by social scientists with faulty causal methodology
can, and often do, ALL lead to the wrong conclusion).

The data from a well-designed and well-controlled study is
ALSO "observational data".

It is the "observational data" in the ABSENCE of a well designed
study (those in the studies by social scientists on whatever
observation they can get, without a design study) -- THOSE
are the ones that are invalid.

The problem here, and why I suspected that he was hedging, is
that Bob has never accepted anything concrete as an indicator
of "warranted conclusions."
RF
Is it possible that Richard Ulrich never CITED any well designed
study that drew a correct, warranted conclusion?

That was not 'hedging'.

I think I responded sufficiently to most of this 343 line post
in my long reply in the "Case Study" thread. Bob is
unnecessarily verbose.

Richard Ulrich wrote:
[cross-posted from sci.stat.edu, since Bob has been slurring
me everywhere lately.]
RF
Very interesting statement, coming from Richard Ulrich. I can
predict, with 100% certainty that Richard will be repeating whatever
error I had point out which he called it "slur"?

[snip; including 'multicollinearity']

My 'discussions' with Bob on regression covered almost
exactly the same territory as Jerry Dallal's later discussions.
Similar content, differently expressed. In both cases, Bob
lost on all salient points.

Who posted what, when? Who cares?
I've learned that I post better when I take time to think.

[snip, causal diagrams.
[snip]
RF

I am SURE I've mentioned Copi's book and the matter of
"cause" to Richard in one of the many corrections of his
errors in using regression to make causal inference.

Yes, Cobi from 1955 or so.

The data from a well-designed and well-controlled study is
ALSO "observational data".

It is the "observational data" in the ABSENCE of a well designed
study (those in the studies by social scientists on whatever
observation they can get, without a design study) -- THOSE
are the ones that are invalid.

I don't know what Bob means by "a design study."

[snip]
[snip, stuff covered]

--
Rich Ulrich, wpilib@Pitt.edu
http://www.pitt.edu/~wpilib/index.html

Richard Ulrich wrote:
[snip]

My 'discussions' with Bob on regression covered almost
exactly the same territory as Jerry Dallal's later discussions.
Similar content, differently expressed. In both cases, Bob
lost on all salient points.
RF
Quit aligning yourself with Jerry. You FLATTER yourself.
Jerry made ONE error, and that's NOT in regression. It was
his error on his webpage on using the UNPOOLED variance
in Z in testing Ho: p1 = p2.

RF
The data from a well-designed and well-controlled study is
ALSO "observational data".

It is the "observational data" in the ABSENCE of a well designed
study (those in the studies by social scientists on whatever
observation they can get, without a design study) -- THOSE
are the ones that are invalid.
RU
I don't know what Bob means by "a design study."
RF
Of course you don't and are just playing DUMB on my used of
designed studies to establish Causal Inference to require CONTROL
rather than purely observational data without control.

On 5 Dec 2006 04:39:50 -0800, "Reef Fish"
large_nassua_grouper@yahoo.com> wrote:


Richard Ulrich wrote:
[snip]
RU
My 'discussions' with Bob on regression covered almost
exactly the same territory as Jerry Dallal's later discussions.
Similar content, differently expressed. In both cases, Bob
lost on all salient points.
RF
Quit aligning yourself with Jerry. You FLATTER yourself.
Jerry made ONE error, and that's NOT in regression. It was
his error on his webpage on using the UNPOOLED variance
in Z in testing Ho: p1 = p2.

Forgotten already? I had in mind the long discussion
where Jerry always did allow that the Banker, in the example
on his website, *might* have good reason to expect the
direction of a regression coefficient. June, 2005, for part of it --

http://groups.google.com/group/sci.stat.math/msg/58fcfbb9f1788c19

Jerry was always mild, but never gave up that point.
Or others.

I will "align myself with Jerry" because it was several of his
threads with Bob that I used for comparative study. They
demonstrated to me that Bob's failures of reasoning were
frequent and obvious, and were *not* an artifact of my
perceptions of my own threads with Bob. Moreover, his
discussion of regression covered the same ground.

[snip]

I have NOW given my definitive answer to the banker example

RU
I don't know what Bob means by "a design study."
RF
Of course you don't and are just playing DUMB on my used of
designed studies to establish Causal Inference to require CONTROL
rather than purely observational data without control.


No. I am questioning the use of *words* because Bob
sounds so far off base.

Now, he adds 'control' to explain what seemed
confusing in the phrase "design study."

The word 'control' seems to be a problem.

--
Rich Ulrich, wpilib@pitt.edu
http://www.pitt.edu/~wpilib/index.html

Richard Ulrich wrote:
On 5 Dec 2006 04:39:50 -0800, "Reef Fish"
large_nassua_grouper@yahoo.com> wrote:


Richard Ulrich wrote:
[snip]

Forgotten already? I had in mind the long discussion
where Jerry always did allow that the Banker, in the example
on his website, *might* have good reason to expect the
direction of a regression coefficient. June, 2005, for part of it --

http://groups.google.com/group/sci.stat.math/msg/58fcfbb9f1788c19
RF
You forgot that I have photographic memory on the substance of every
STATISTICAL discussion in these groups?

Now, he adds 'control' to explain what seemed
confusing in the phrase "design study."
RF
For drawing CAUSAL inference, "control" is ALWAYS part
of the design. For other designs, it may or may not.

RU
The word 'control' seems to be a problem.
RF
Only for Quacks like Richard Ulrich.

What happened to the kill-file you threatened to put me in
and I URGED you and STRONGLY URGED you to do so?

You're just polluting this group too much with your continued
noise of your ERRORS, now dragging Jerry Dallal down with
you without helping YOUR CASE the slightest.

- I will add little here -

No need.

Ray, you're right. Thanks for the correction.
Jack