Hi everyone,

I am reading out of a stats book and came across the following
passage, regarding orthogonal projections, that I find confusing:

let x be a vector of p variables. let B be a matrix with orthonormal
columns, of size pxq. Now create a vector of q new variables, call it
y, where y=B'x (thats B transpose times x). The book says that y is
the orthogonal projection of x onto the space spanned by the columns
of B ... and thats what I don't see. Any ideas?

First, I thought projection matrices always had to be square. That
ensures the original vector and its projection live in the same
subspace. Here, I don't know what it means to project a vector from
R^p onto R^q.

Second, by my calculations (x - y) is not orthogonal to y, which is
the definition of y being the orthogonal projection of x, right?

fyi, the book is "principal component analysis" by ian jolliffe,
section 3.2

Thanks in advance,
CP

On Jun 18, 5:25 pm, cprice <cpr... at (no spam) gmail.com> wrote:



Hi everyone,

I am reading out of a stats book and came across the following
passage, regarding orthogonal projections, that I find confusing:

let x be a vector of p variables. let B be a matrix with orthonormal
columns, of size pxq. Now create a vector of q new variables, call it
y, where y=B'x (thats B transpose times x). The book says that y is
the orthogonal projection of x onto the space spanned by the columns
of B ... and thats what I don't see. Any ideas?

First, I thought projection matrices always had to be square. That
ensures the original vector and its projection live in the same
subspace. Here, I don't know what it means to project a vector from
R^p onto R^q.

Second, by my calculations (x - y) is not orthogonal to y, which is
the definition of y being the orthogonal projection of x, right?

fyi, the book is "principal component analysis" by ian jolliffe,
section 3.2

Thanks in advance,
CP

Think about p = 3, q = 2; i.e., dropping a perpendicular from a point
to a plane.

I don't think that concept is possible here. The plane is a 2
dimensional subspace, but is still a part of R^3, it is entirely made
up of vectors from R^3. Absolutely no point on the plane will have a
vector from R^2, so the vector y can not be in the plane. I don't know
what it means to drop a perpendicular from R^3 into R^2.

I am reading out of a stats book and came across the following
passage, regarding orthogonal projections, that I find confusing:

let x be a vector of p variables. let B be a matrix with orthonormal
columns, of size pxq. Now create a vector of q new variables, call it
y, where y=B'x (thats B transpose times x). The book says that y is
the orthogonal projection of x onto the space spanned by the columns
of B ... and thats what I don't see. Any ideas?

First, I thought projection matrices always had to be square. That
ensures the original vector and its projection live in the same
subspace. Here, I don't know what it means to project a vector from
R^p onto R^q.

Second, by my calculations (x - y) is not orthogonal to y, which is
the definition of y being the orthogonal projection of x, right?

I don't think that concept is possible here. The plane is a 2
dimensional subspace, but is still a part of R^3, it is entirely made
up of vectors from R^3. Absolutely no point on the plane will have a
vector from R^2, so the vector y can not be in the plane. I don't know
what it means to drop a perpendicular from R^3 into R^2.

-CP

Hi everyone,

I am reading out of a stats book and came across the following
passage, regarding orthogonal projections, that I find confusing:

let x be a vector of p variables. let B be a matrix with orthonormal
columns, of size pxq.

Now create a vector of q new variables, call it
y, where y=B'x (thats B transpose times x).

First, I thought projection matrices always had to be square. That
ensures the original vector and its projection live in the same
subspace. Here, I don't know what it means to project a vector from
R^p onto R^q.

On Jun 19, 5:13 am, cprice <cpr... at (no spam) gmail.com> wrote:

I don't think that concept is possible here. The plane is a 2
dimensional subspace, but is still a part of R^3, it is entirely made
up of vectors from R^3. Absolutely no point on the plane will have a
vector from R^2, so the vector y can not be in the plane. I don't know
what it means to drop a perpendicular from R^3 into R^2.

-CP

I think the problem may be with two related but different uses of
"projection": you use it to refer to an operator, a matrix, but
Jolliffe uses it to refer to the result of the operation, a vector.

Consider an ordinary room in a building. Choose one of the corners
of the room as the origin, and let the intersections of the walls
and floor define the reference axes: U and V for the two wall-floor
intersections, and W for the wall-wall intersection. Hold a ball
of string in your hand, and let out some string until it touches
the floor. If the coordinates of your hand are x = (u,v,w), then
the coordinates of the point where the string touches the floor
are y = (u,v,0), which is the orthogonal projection of x onto the
U,V plane.

cprice wrote:
Hi everyone,

I am reading out of a stats book and came across the following
passage, regarding orthogonal projections, that I find confusing:

let x be a vector of p variables. let B be a matrix with orthonormal
columns, of size pxq.

Trying to look at things in some coordinate system and formulate
coordinate-free statements at the same time is always confusing.

One can look at it this way (bear with me, it is lengthy but hopefully
will be illuminating):

B is a list of q vectors in p dimensional space V(=R^p) which form an
orthonormal basis b_1, .., b_q of the q dimensional subspace W(=R^q).

If E=<e_1, ..., e_p> is the basis of V(=R^p) then columns of B are
coordinates of vectors b_1, ..., b_q in this basis E.

B also represents a linear transformation from R^q to R^p, it transforms
a vector with R^q coordinates to a vector in R^p coordinates.

Since W is q-dimensional it has (many) coordinate systems. If we want to
pick one it is all natural to choose that one in which basis vectors
b_1,..,b_q become e_1=[1,0,..,0]', ..e_q=[0,..,0,1]'.

B' is a linear transformation from R^p(=V) to R^q(=W). B' applied to the
vectors of the basis B gives e_1,..,e_q in result, which means that it
(matrix, or transformation B') will transform R^p coordinates of any
vector from W to its coordinates within W (without looking out to the
outside p-dimensional world of V).

Now create a vector of q new variables, call it
y, where y=B'x (thats B transpose times x).

OK, y is a q-dimensional vector in W (hence in V, as W is a subspace of
V). The only problem is coordinatization, if you look just at R^p and
R^q you start treating them as separate vector spaces. I suspect that in
this context they are not, they are only separate coordinate systems of
the same thing, the objects described can be naturally embedded (i.e. W
in V).

First, I thought projection matrices always had to be square. That
ensures the original vector and its projection live in the same
subspace. Here, I don't know what it means to project a vector from
R^p onto R^q.

So, all you need to work out now is really what are R^p coordinates of
the vector y. And here matrix B (and illywhacker) come to help.

Hope that helped a little,
JoJo

Thanks for everyone's input.

From illywhacker, I fully agree that BB' does project onto the space
spanned by its orthonormal columns. While this is true, BB' is square,
and performing y=BB'x makes both the input and output vector members
of R^p (just to be clear, thats vectors with p elements), and this
escapes the original problem I was having, where the input vector x
comes from R^p while the supposed projection vector y is in R^q.

I do think JoJo has hit upon the solution, which is that the vector y
may just be the coordinates of that certain vector in R^p, that I
would call the *actual* projection, but since the actual projection
lives in a q dimensional subspace of R^p, its coordinates will belong
to R^q. From Ray Koopmans example, this would be like dropping the
ball onto the point [u,v,0] and calling the projection just [u,v],
whereas I think it would be better to say [u,v] is only the coordinate
vector of the actual projection vector [u,v,0].