Dear list,
is it possible to use PCA on categorical data?
I have a group of 30 continuous and categorical data and would like to
select a subset for modeling a response variable. I read that PCA
would help me doing this data reduction, but all the examples I have
seen involve continuous data.

Thanks for your help,
D.

Dear list,
is it possible to use PCA on categorical data?
I have a group of 30 continuous and categorical data and would like to
select a subset for modeling a response variable. I read that PCA
would help me doing this data reduction, but all the examples I have
seen involve continuous data.

If you have 30 continuous and categorical variables, and you want to
predict Y, I suggest you look into Partial Least Squares (PLS)
regression, which is much more suited to the case where you have many
correlated predictor variables. PLS can be easily modified to handle
categorical variables. It provides a model where all of your 30
predictor variables are used in prediction. You can find variables
with small loadings or small regression coefficients and eliminate
them from your model if you so choose; there is a robust debate among
statisticians and chemometricians whether or not this is a good things
to do.

Hi Paige,

I've seen you recommend PLS on a number of occasions, so I imagine you
might know what's happening. Some years ago, I had used a pre-release
version of Wynne Chin's PLSGraph, but I'm not sure he ever completed the
project. Do you know anything about it? Do you use or recommend any other
software for PLS?

Thanks,

Mike Babyak

Dear list,
is it possible to use PCA on categorical data?
I have a group of 30 continuous and categorical data and would like to
select a subset for modeling a response variable. I read that PCA
would help me doing this data reduction, but all the examples I have
seen involve continuous data.

Thanks for your help,
D.

Dear list,
is it possible to use PCA on categorical data?
I have a group of 30 continuous and categorical data and would like to
select a subset for modeling a response variable. I read that PCA
would help me doing this data reduction, but all the examples I have
seen involve continuous data.

Thanks for your help,
D.

Hi David,

Some suggestions:

1.  If your categorical variables are ordered-categorical, then you
can calcualte:
    a. Pearson correlations between each pair of continuous variables.
    b. Polyserial correlations between continuous and ordered-
categorical variables
    c. Polychoric correlations between ordered-categorical variables

then place these in a single matrix and analyze that matrix by PCA.  A
program like LISREL/Prelis will do all this for you more-or-less
automatically.

2.  Although I agree with what others have posted, personally I prefer
the approach you originally suggested:  to approach data-reduction and
the modeling of your response variable as two separate steps.

3. Since you just want to select a subset of non-redundant variables,
you have other options besides PCA.  For example, you can use
hierarchical cluster analyis on the correlation matrix.  That will
divide your variables into clusters.  Then you can pick 'exemplars'
from each cluster and use those in your data model.  This gives you
more flexibility, because you can use other measures of similarity/
redundancy among your variables besides correlation coefficients.  For
example, if your categorical variables are non-ordered (i.e., purely
nominal variables), you can calculate the canonical correlation
between each pair of them.  Then you can cluster analyze the matrix of
canonical correlation coefficients to divide the variables into
separate groups, and then select exemplars from each group.

Possibly you can include the canonical correlations in the overall
matrix as described in point 1 above -- I'm not sure, becuase they
might tend to run lower overall than Pearson correlations.

Hope this helps.

John Uebersax PhD

On May 8, 6:21 pm, David <david_art... at (no spam) hotmail.com> wrote:



Dear list,
is it possible to use PCA on categorical data?
I have a group of 30 continuous and categorical data and would like to
select a subset for modeling a response variable. I read that PCA
would help me doing this data reduction, but all the examples I have
seen involve continuous data.

Thanks for your help,
D.- Hide quoted text -

- Show quoted text -

- Paige, you suggest PLS, but is PLS not doing effectively what PCA
does or Principal Component Regression? I have just read through it
quickly, and had a look at Faraway`s "Practical Regression and ANOVA
using R" and it says "On the other hand, PLS is virtually useless for
explanation purposes". So how can I trace back my regressors after
doing PLS?

Paige Miller wrote:
When you have 30 input variables, and they are (highly) correlated,
the problem isn't the method -- in this case PLS -- the problem is
that you cannot in any way separate the distinct and independent
effects of each of the individual input variables. Logically, this
cannot be done.

- But there is also a point that with "too many variables" [with regard to
the number of cases], in order to avoid capitalisation on chance (overfit,
lack of generalisability or however-you-call-it) while avoiding learning
stuff too advanced for "only semi-smart people" (regularised discriminant
analysis, shrinkage a la Prof. Harrell etc.), you can get valid results with
simple methods precisely and only by ignoring the outcome when reducing the
dimensionality first (with PCA, clustering [followed by selection of
"representatives" or producing a score on each "varible group"], CATPCA
[after wise discretisation of numeric variables], even FA, or some other
way), doing, of course, everything cum grano salis (i.e., with subject
matter knowledge, "feeling" for data etc.).

Hi David,

Some suggestions:

1. If your categorical variables are ordered-categorical, then you
can calcualte:
a. Pearson correlations between each pair of continuous variables.
b. Polyserial correlations between continuous and ordered-
categorical variables
c. Polychoric correlations between ordered-categorical variables

then place these in a single matrix and analyze that matrix by PCA. A
program like LISREL/Prelis will do all this for you more-or-less
automatically.

2. Although I agree with what others have posted, personally I prefer
the approach you originally suggested: to approach data-reduction and
the modeling of your response variable as two separate steps.

3. Since you just want to select a subset of non-redundant variables,
you have other options besides PCA. For example, you can use
hierarchical cluster analyis on the correlation matrix. That will
divide your variables into clusters. Then you can pick 'exemplars'
from each cluster and use those in your data model. This gives you
more flexibility, because you can use other measures of similarity/
redundancy among your variables besides correlation coefficients. For
example, if your categorical variables are non-ordered (i.e., purely
nominal variables), you can calculate the canonical correlation
between each pair of them. Then you can cluster analyze the matrix of
canonical correlation coefficients to divide the variables into
separate groups, and then select exemplars from each group.

Possibly you can include the canonical correlations in the overall
matrix as described in point 1 above -- I'm not sure, becuase they
might tend to run lower overall than Pearson correlations.

Hope this helps.

John Uebersax PhD

On May 8, 6:21 pm, David <david_art... at (no spam) hotmail.com> wrote:



Dear list,
is it possible to use PCA on categorical data?
I have a group of 30 continuous and categorical data and would like to
select a subset for modeling a response variable. I read that PCA
would help me doing this data reduction, but all the examples I have
seen involve continuous data.

Thanks for your help,
D.- Hide quoted text -

- Show quoted text -

On May 15, 8:59 am, "Gaj Vidmar" <gaj.vid... at (no spam) mf.uni-lj.si> wrote:

- But there is also a point that with "too many variables" [with regard
to
the number of cases], in order to avoid capitalisation on chance
(overfit,
lack of generalisability or however-you-call-it) while avoiding learning
stuff too advanced for "only semi-smart people" (regularised discriminant
analysis, shrinkage a la Prof. Harrell etc.), you can get valid results
with
simple methods precisely and only by ignoring the outcome when reducing
the
dimensionality first (with PCA, clustering [followed by selection of
"representatives" or producing a score on each "varible group"], CATPCA
[after wise discretisation of numeric variables], even FA, or some other
way), doing, of course, everything cum grano salis (i.e., with subject
matter knowledge, "feeling" for data etc.).

This is an extremely long sentence and quite honestly, it is so
convoluted in its structure, that I am afraid I may not understand
what you are saying.

However ... PLS has protection against overfitting. It is called
crossvalidation. Crossvalidation isn't perfect, but it does prevent
certain abuses of the data. The mere fact that PLS may use large
number of variables as X and much fewer observations than variables
does not imply that overfitting has happened. In the original poster's
case, if he had 30 variables, and they are (highly) correlated, there
may not be 30 independent things happening in his X matrix. In fact,
PLS usually chooses a few dimensions to use for prediction, which most
definitely is not overfitting. As I said earlier, it has proven to be
a useful tool in many published examples.

The part of the sentence that reads "you can get valid results with
simple methods precisely and only by ignoring the outcome when
reducing the dimensionality first" disturbs me greatly, if I am
understanding it properly. I will say again, pre-processing your data
without regards to the dependent variable is dangerous and may discard
information that can be used to predict. I strongly urge people to use
pre-processing techniques that do not ignore the dependent variable.

--
Paige Miller
paige\dot\miller \at\ kodak\dot\com

- Art, you suggest some PCA methods, but my initial worry about using
PCA is losing iterpreatability