| |
 |
|
|
Science Forum Index » Statistics - Math Forum » Thinking of covariance matrix as of transformation...
Page 1 of 1
|
| Author |
Message |
| Leonid L... |
 Posted: Sun Jul 13, 2008 11:41 am |
|
|
|
Guest
|
Hi,
I've taken intro to linear algebra and intro to statistics, so I
should have the prerequisites for the question I am about to ask.
I've been told that serious statistics heavily relies on linear
algebra. A covariance matrix should be something that is used
frequently. I think I have a good understanding of what the covariance
matrix is, but I am having a hard time thinking of it as of a
transformation matrix (because any matrix can be though of as some
sort of transformation).
The Wikipedia's article on Covariance Matrix does have this section:
Quote: ==========|==========
As a linear operator
Applied to one vector, the covariance matrix maps a linear combination
(c) of the random variables (X) onto a vector of covariances with
those variables: \mathbf c^\top\Sigma = \operatorname{cov}(\mathbf c^
\top\mathbf X,\mathbf X). Treated as a 2-form, it yields the
covariance between the two linear combinations: \mathbf d^\top\Sigma
\mathbf c=\operatorname{cov}(\mathbf d^\top\mathbf X,\mathbf c^\top
\mathbf X). The variance of a linear combination is then \mathbf c^\top
\Sigma\mathbf c, its covariance with itself.
Quote: ==========|==========
but I find it a little abstract and non-intuitive. What would make it
easier for me is to think of a portfolio of (say) 5 assets, with some
associated weights (say 20% each), and a daily return series for each
one dating back 5 years (should be plenty of data points to compute
variances and covariances accurately, right?).
With this data I can compute covariance between any two assets. Hence,
I can compute the entire covariance matrix. So, I should have a 5x5
matrix that represents some sort of transformation from R^5 to R^5.
What is the meaning of such transformation? What about the (related)
correlation matrix? Is it easier to explain the transformation in
terms of one of the two?
Now, the correlation matrix has some interesting properties:
* It is symmetric. So, it must be similar to some diagonal matrix. Is
it true to say that it also preserves distances (of vectors to which
it is applied)? What about the fact that it has 1s along the
diagonals? What can one say about the properties of the corresponding
transformation?
If you have to use heavy math notation for the answers to these
questions, please also supplement it with word descriptions. I'd like
to be able to understand this intuitively without having to write down
formulas (although formulas are almost surely needed in order to
derive an answer). A few simple examples would also be helpful.
Regards,
- Leonid |
|
|
| Back to top |
|
| Jack Tomsky... |
| Posted: Sun Jul 13, 2008 6:33 pm |
|
|
|
Guest
|
Quote: Hi,
I've taken intro to linear algebra and intro to
statistics, so I
should have the prerequisites for the question I am
about to ask.
I've been told that serious statistics heavily relies
on linear
algebra. A covariance matrix should be something that
is used
frequently. I think I have a good understanding of
what the covariance
matrix is, but I am having a hard time thinking of it
as of a
transformation matrix (because any matrix can be
though of as some
sort of transformation).
The Wikipedia's article on Covariance Matrix does
have this section:
==========|==========
As a linear operator
Applied to one vector, the covariance matrix maps a
linear combination
(c) of the random variables (X) onto a vector of
covariances with
those variables: \mathbf c^\top\Sigma =
\operatorname{cov}(\mathbf c^
\top\mathbf X,\mathbf X). Treated as a 2-form, it
yields the
covariance between the two linear combinations:
\mathbf d^\top\Sigma
\mathbf c=\operatorname{cov}(\mathbf d^\top\mathbf
X,\mathbf c^\top
\mathbf X). The variance of a linear combination is
then \mathbf c^\top
\Sigma\mathbf c, its covariance with itself.
==========|==========
but I find it a little abstract and non-intuitive.
What would make it
easier for me is to think of a portfolio of (say) 5
assets, with some
associated weights (say 20% each), and a daily return
series for each
one dating back 5 years (should be plenty of data
points to compute
variances and covariances accurately, right?).
With this data I can compute covariance between any
two assets. Hence,
I can compute the entire covariance matrix. So, I
should have a 5x5
matrix that represents some sort of transformation
from R^5 to R^5.
What is the meaning of such transformation? What
about the (related)
correlation matrix? Is it easier to explain the
transformation in
terms of one of the two?
Now, the correlation matrix has some interesting
properties:
* It is symmetric. So, it must be similar to some
diagonal matrix. Is
it true to say that it also preserves distances (of
vectors to which
it is applied)? What about the fact that it has 1s
along the
diagonals? What can one say about the properties of
the corresponding
transformation?
If you have to use heavy math notation for the
answers to these
questions, please also supplement it with word
descriptions. I'd like
to be able to understand this intuitively without
having to write down
formulas (although formulas are almost surely needed
in order to
derive an answer). A few simple examples would also
be helpful.
Regards,
- Leonid
I'll start off with the heavy math and then I'll try to interpret it in plain English.
If x is a p by 1 random vector, let the covariance matrix of x be called SIG, which is a p by p symmetric matrix. (Also, SIG is positive semi-definite). In your example, p = 5.
The diagonals of the matrix SIG are the variances of x and the off-diagonals are the covariances of x. So sig_ii is the variance of x_i and sig_ij is the covariance between x_i and x_j.
Now suppose we have a linear combination of the x_i's, Sum(a_i*x_i). You can get its variance from the matrix SIG as a quadratic form. Its variance is
Var(Sum(a_i*x_i)) = Var(a'x) = a'*SIG*a.
Now suppose we have another linear combination of the x_i's, Sum(b_i*x_i). The covariance between these two linear combinations can be obtained from SIG as a bilinear form.
Cov(Sum(a_i*x_i), Sum(b_i*x_i)) = Cov(a'x, b'x) = a'*SIG*b.
If you scale each x_i by dividing it by its standard deviation, the resulting covariance matrix is the correlation matrix of x.
Let y = D^(-1/2)*x, where D is a diagonal matrix whose diagonal elements are the variances of x. Then the correlation matrix of x is the covariance matrix of y or
R = D^(-1/2)*SIG*D^(-1/2).
Going back to your original question, let X be an N by p matrix of data. There are N observations on each of p variables. The p by p sample covariance matrix S can be written in terms of X as
(N-1)S = X'X - X'ee'X/N = X'(I-ee'/N)X.
e is an N by 1 vector consisting of N ones. I is the N by N identity matrix. The X'X part gives you the sums of squares and cross-products, while the X'ee'X/N part subtracts out the means. So what we have is a transformation from X to S.
Let me know if you have further questions.
Jack |
|
|
| Back to top |
|
| |
|
Page 1 of 1
All times are GMT - 5 Hours
The time now is Sat Nov 22, 2008 4:00 pm
|
|