The motivations for this post are
(1) The failure of MATLAB functions REGRESS and LSCOV to
yield basic solutions for linear regression when predictors
are linearly dependent.
(2) The lack of documentation on how the reciprocal condition
number "RCOND" is calculated when division by ill-conditioned
predictor matrix factors cause warnings to be written to the
command window.

The post is cross-posted to sci.stat.* because of previous interest
in the topic.

%2345678901234567890123456789012345678901234567890123456789012345
%
% The Effect of Linear Dependence on Linear Regression
%
% X - predictor matrix; size(X) = [ Nobs Nvar ]
% y - response matrix; size(y) = [Nobs 1]
% Xa - ones-augmented predictor matrix
% Xa = [ones(Nobs,1) X]
% b - linear regression coefficient matrix
% Xa*b = y, size(b) = [Nvar1+1 1]
%
% "SVD" - Singular value decomposition
% Xa = U*S*V'; Xa*V = U*S; Xa'*U = V*S
% S - Diagonal Singular Value matrix of Xa
% V - Unitary Right Singular Vector Matrix of Xa
% U - Unitary Left Singular Vector Matrix of Xa
%
% Predictor variables are linearly dependent if, and only
% if, solutions to the null space equation Xa*v = 0 exist.
% The null space is spanned by columns of V corresponding
% to zero singular values. Two forms of SVDs for Xa can
% be obtained from the MATLAB commands [U,S,V] = svd(Xa)
% and [U,S,V] = svd(Xa,0). S can also be obtained from
% S = svd(Xa) (Type "help svd" for details).
%
% Therefore, given y and linearly dependent X, b is not unique
% because arbitrary multiples of null space vectors can be added
% to b without changing y.
%
% In such cases, the MATLAB functions SLASH, REGRESS
% and LSCOV are supposed to yield a basic solution which has
% no more than rank(Xa) nonzero components corresponding
% to the number of linearly independent predictors.
%
% The matrix condition number cond(Xa) is the best machine
% independent quantitative indicator of both the degree of
% linear independence in X and uncertainty in b. Uncertainties
% in b caused by uncertainties in X and y are bounded by
%
% ||db||/||b|| <= cond(Xa)*[ ||dy||/||y|| + ||dX||/||X|| ]
%
% where ||.|| idicates matrix norm.
%
% MATLAB help information (2-norm assumed):
%
% norm(Xa) = max(S)
% cond(Xa) = max(S)/ min(S)
% eps = 2.22044604925031e-016 % (machine epsilon)
% tolXa = max(size(Xa))*norm(Xa)*eps
% = Nobs*max(S)*eps
% rank(Xa) = sum(S > tolXa)
%
% Traditionally, the reciprocal condition number is defined as
% rcondXa = 1/cond(Xa).
%
% When the above definitions are used, the example below
% would yield:
% S = svd(Xa)
% Smax = max(S) % 2.5933
% Smin = min(S) % 1.3963e-016
% normXa = Smax % 2.5933
% tolXa = max(size(Xa))*Smax*eps % 2.8792e-015
% rankXa = sum(S > tolXa ) % 3
% condXa = Smax/Smin % 1.8573e+016
% rcondXa = 1/condXa % 5.3841e-017
%
% However,(because the QR decomposition and matrix
% inversion algorithm of LAPACK is used?), the MATLAB
% functions SLASH, REGRESS and LSCOV use different
% definitions based on the 1-norm of R:
%
% norm(R,1) = max(sum(abs(R)))
% cond(R,1) = norm(R,1) * norm(inv(R),1) % R must be square!
% rcond(R) = 1/cond(R,1)
%
% SLASH: norm and tol
%
% [Q1,R1] = qr(Xa) % (NOT qr(Xa,0)
!)
% norm1R1 = norm(R1,1) % 2.2361
% tol = max(size(R1))*norm1R*eps % 2.4825e-015
%
% REGRESS and LSCOV: rcond (need a square R)
%
% [Q2,R2] = qr(Xa,0) % (NOT qr(Xa) !)
% rcondR2 = rcond(R2) % 2.6377e-017
%
% Notice that rcondR2 is NOT THE SAME as that quoted in the
% warnings of REGRESS and LSCOV caused by the attempted
% matrix division by the ill-conditioned R2. For example:
%
% IL = R2\R2
%
% Warning: Matrix is close to singular or badly scaled.
% Results may be inaccurate. RCOND = 1.055077e-016.
%
% The Example:

clear, close all, clc, format short g

X = [ 1/2 3/8 1/8;
3/8 1/2 1/8;
1/7 3/7 3/7;
1/9 1/3 5/9;
1/8 1/8 3/4];
[Nobs Nvar] = size(X) % [ 5 3]
Xa = [ones(Nobs,1) X];

b0 = [0 1 2 4 ]' % Note: zero intercept
y0 = Xa*b0

disp('')
disp('SLASH SOLUTION')
disp('')

b1 = Xa\y0;

% Warning: Rank deficient, rank = 3 tol = 2.4825e-015.
%
% Checking:
% [Q1,R1] = qr(Xa) %(NOT qr(Xa,0)
!)
% sizeR1 = size(R1) % [ 5 4 ]
% maxsizeR1 = max(sizeR1) % 5
% normR1 = norm(R1,1) % 2.2361
% tol1 = maxsizeR1*normR1*eps % 2.4825e-015
% smin1 = min(svd(R1)) % 1.1796e-016
% rankR1 = rank(R1,tol1) % 3

disp('')
disp('REGRESS SOLUTION')
disp('')

b2 = regress(y0,Xa);
% Warning: Matrix is close to singular or badly scaled.
% Results may be inaccurate. RCOND = 1.055077e-016.
%
% Checking:
% [Q2,R2] = qr(Xa,0) %(NOT qr(Xa) !)
% sizeR2 = size(R2) % [ 4 4 ]
% maxsizeR2 = max(sizeR2) % 4
% normR2 = norm(R2,1) % 2.2361
% tol2 = maxsizeR2*normR2*eps % 1.986e-015
% smin2 = min(svd(R2)) % 1.1796e-016
% rankR2 = rank(R2,tol2) % 3
%
% rcondR2 = rcond(R2) % 2.6377e-017 ???
% condR2 = cond(R2,1) % 3.7912e+016
% rcondR2 = 1/cond(R2,1) % 2.6377e-017 ??

disp('')
disp('LSCOV SOLUTION')
disp('')

b3 = lscov(Xa,y0,eye(Nobs));

% Warning: Matrix is close to singular or badly scaled.
% Results may be inaccurate. RCOND = 1.055077e-016.

B0 = [b1 b2 b3]

% B0 = [ 1 0.70588 0.70588;
% 0 0.29412 0.29412;
% 1 1.2941 1.2941;
% 3 3.2941 3.2941]

Y0 = repmat(y0,1,3);
R0 = Y0 - Xa*B0

% R0 = [ 0 0 0;
% 0 0 0;
% 0 0 0;
% 0 0 0;
% 0 0 0]

%%%%%%%%%%%% COMMENTS %%%%%%%%%%%%

% 1. SLASH yields a basic solution
% 2. REGRESS and LSCOV do not yield basic solutions!
% 3. Cannot verify "RCOND" in REGRESS/LCOV warnings
% 4. v = b0-b1 = [ -1 1 1 1 ] is a null vector
% indicating that x1+x2+x3 = 1 is the underlying
% linear dependence.
% 5. V(:,4) = -0.5*v is the singular vector corresponding
% to min(S) = 1.3963e-016
% 6. Combining b0 and b1 yields the basic solution
% matrix BASIC (to within computer precision)

BASIC = [ b0 b1 2*b1-b0 4*b1-3*b0]

% BASIC=
% [ 0 1 2 4;
% 1 0 -1 -3;
% 2 1 6.6613e-015 -2;
% 4 3 2 1.2434e-014]

Hope this helps.

Greg

Why would you want to build a model (or attempt to build a model) from
data in which the predictors are linearly dependent?
OMU

Greg Heath wrote:
The motivations for this post are
(1) The failure of MATLAB functions REGRESS and LSCOV to
yield basic solutions for linear regression when predictors
are linearly dependent.
(2) The lack of documentation on how the reciprocal condition
number "RCOND" is calculated when division by ill-conditioned
predictor matrix factors cause warnings to be written to the
command window.

The post is cross-posted to sci.stat.* because of previous interest
in the topic.

%2345678901234567890123456789012345678901234567890123456789012345
%
% The Effect of Linear Dependence on Linear Regression
%
% X - predictor matrix; size(X) = [ Nobs Nvar ]
% y - response matrix; size(y) = [Nobs 1]
% Xa - ones-augmented predictor matrix
% Xa = [ones(Nobs,1) X]
% b - linear regression coefficient matrix
% Xa*b = y, size(b) = [Nvar1+1 1]
%
% "SVD" - Singular value decomposition
% Xa = U*S*V'; Xa*V = U*S; Xa'*U = V*S
% S - Diagonal Singular Value matrix of Xa
% V - Unitary Right Singular Vector Matrix of Xa
% U - Unitary Left Singular Vector Matrix of Xa
%
% Predictor variables are linearly dependent if, and only
% if, solutions to the null space equation Xa*v = 0 exist.
% The null space is spanned by columns of V corresponding
% to zero singular values. Two forms of SVDs for Xa can
% be obtained from the MATLAB commands [U,S,V] = svd(Xa)
% and [U,S,V] = svd(Xa,0). S can also be obtained from
% S = svd(Xa) (Type "help svd" for details).
%
% Therefore, given y and linearly dependent X, b is not unique
% because arbitrary multiples of null space vectors can be added
% to b without changing y.
%
% In such cases, the MATLAB functions SLASH, REGRESS
% and LSCOV are supposed to yield a basic solution which has
% no more than rank(Xa) nonzero components corresponding
% to the number of linearly independent predictors.
%
% The matrix condition number cond(Xa) is the best machine
% independent quantitative indicator of both the degree of
% linear independence in X and uncertainty in b. Uncertainties
% in b caused by uncertainties in X and y are bounded by
%
% ||db||/||b|| <= cond(Xa)*[ ||dy||/||y|| + ||dX||/||X|| ]
%
% where ||.|| idicates matrix norm.
%
% MATLAB help information (2-norm assumed):
%
% norm(Xa) = max(S)
% cond(Xa) = max(S)/ min(S)
% eps = 2.22044604925031e-016 % (machine epsilon)
% tolXa = max(size(Xa))*norm(Xa)*eps
% = Nobs*max(S)*eps
% rank(Xa) = sum(S > tolXa)
%
% Traditionally, the reciprocal condition number is defined as
% rcondXa = 1/cond(Xa).
%
% When the above definitions are used, the example below
% would yield:
% S = svd(Xa)
% Smax = max(S) % 2.5933
% Smin = min(S) % 1.3963e-016
% normXa = Smax % 2.5933
% tolXa = max(size(Xa))*Smax*eps % 2.8792e-015
% rankXa = sum(S > tolXa ) % 3
% condXa = Smax/Smin % 1.8573e+016
% rcondXa = 1/condXa % 5.3841e-017
%
% However,(because the QR decomposition and matrix
% inversion algorithm of LAPACK is used?), the MATLAB
% functions SLASH, REGRESS and LSCOV use different
% definitions based on the 1-norm of R:
%
% norm(R,1) = max(sum(abs(R)))
% cond(R,1) = norm(R,1) * norm(inv(R),1) % R must be square!
% rcond(R) = 1/cond(R,1)
%
% SLASH: norm and tol
%
% [Q1,R1] = qr(Xa) % (NOT qr(Xa,0)
!)
% norm1R1 = norm(R1,1) % 2.2361
% tol = max(size(R1))*norm1R*eps % 2.4825e-015
%
% REGRESS and LSCOV: rcond (need a square R)
%
% [Q2,R2] = qr(Xa,0) % (NOT qr(Xa) !)
% rcondR2 = rcond(R2) % 2.6377e-017
%
% Notice that rcondR2 is NOT THE SAME as that quoted in the
% warnings of REGRESS and LSCOV caused by the attempted
% matrix division by the ill-conditioned R2. For example:
%
% IL = R2\R2
%
% Warning: Matrix is close to singular or badly scaled.
% Results may be inaccurate. RCOND = 1.055077e-016.
%
% The Example:

clear, close all, clc, format short g

X = [ 1/2 3/8 1/8;
3/8 1/2 1/8;
1/7 3/7 3/7;
1/9 1/3 5/9;
1/8 1/8 3/4];
[Nobs Nvar] = size(X) % [ 5 3]
Xa = [ones(Nobs,1) X];

b0 = [0 1 2 4 ]' % Note: zero intercept
y0 = Xa*b0

disp('')
disp('SLASH SOLUTION')
disp('')

b1 = Xa\y0;

% Warning: Rank deficient, rank = 3 tol = 2.4825e-015.
%
% Checking:
% [Q1,R1] = qr(Xa) %(NOT qr(Xa,0)
!)
% sizeR1 = size(R1) % [ 5 4 ]
% maxsizeR1 = max(sizeR1) % 5
% normR1 = norm(R1,1) % 2.2361
% tol1 = maxsizeR1*normR1*eps % 2.4825e-015
% smin1 = min(svd(R1)) % 1.1796e-016
% rankR1 = rank(R1,tol1) % 3

disp('')
disp('REGRESS SOLUTION')
disp('')

b2 = regress(y0,Xa);
% Warning: Matrix is close to singular or badly scaled.
% Results may be inaccurate. RCOND = 1.055077e-016.
%
% Checking:
% [Q2,R2] = qr(Xa,0) %(NOT qr(Xa) !)
% sizeR2 = size(R2) % [ 4 4 ]
% maxsizeR2 = max(sizeR2) % 4
% normR2 = norm(R2,1) % 2.2361
% tol2 = maxsizeR2*normR2*eps % 1.986e-015
% smin2 = min(svd(R2)) % 1.1796e-016
% rankR2 = rank(R2,tol2) % 3
%
% rcondR2 = rcond(R2) % 2.6377e-017 ???
% condR2 = cond(R2,1) % 3.7912e+016
% rcondR2 = 1/cond(R2,1) % 2.6377e-017 ??

disp('')
disp('LSCOV SOLUTION')
disp('')

b3 = lscov(Xa,y0,eye(Nobs));

% Warning: Matrix is close to singular or badly scaled.
% Results may be inaccurate. RCOND = 1.055077e-016.

B0 = [b1 b2 b3]

% B0 = [ 1 0.70588 0.70588;
% 0 0.29412 0.29412;
% 1 1.2941 1.2941;
% 3 3.2941 3.2941]

Y0 = repmat(y0,1,3);
R0 = Y0 - Xa*B0

% R0 = [ 0 0 0;
% 0 0 0;
% 0 0 0;
% 0 0 0;
% 0 0 0]

%%%%%%%%%%%% COMMENTS %%%%%%%%%%%%

% 1. SLASH yields a basic solution
% 2. REGRESS and LSCOV do not yield basic solutions!
% 3. Cannot verify "RCOND" in REGRESS/LCOV warnings
% 4. v = b0-b1 = [ -1 1 1 1 ] is a null vector
% indicating that x1+x2+x3 = 1 is the underlying
% linear dependence.
% 5. V(:,4) = -0.5*v is the singular vector corresponding
% to min(S) = 1.3963e-016
% 6. Combining b0 and b1 yields the basic solution
% matrix BASIC (to within computer precision)

BASIC = [ b0 b1 2*b1-b0 4*b1-3*b0]

% BASIC=
% [ 0 1 2 4;
% 1 0 -1 -3;
% 2 1 6.6613e-015 -2;
% 4 3 2 1.2434e-014]

Hope this helps.

Greg

Why would you want to build a model (or attempt to build a model) from
data in which the predictors are linearly dependent?
OMU

Greg Heath wrote:
The motivations for this post are
(1) The failure of MATLAB functions REGRESS and LSCOV to
yield basic solutions for linear regression when predictors
are linearly dependent.
(2) The lack of documentation on how the reciprocal condition
number "RCOND" is calculated when division by ill-conditioned
predictor matrix factors cause warnings to be written to the
command window.

The post is cross-posted to sci.stat.* because of previous interest
in the topic.

%2345678901234567890123456789012345678901234567890123456789012345
%
% The Effect of Linear Dependence on Linear Regression
%
% X - predictor matrix; size(X) = [ Nobs Nvar ]
% y - response matrix; size(y) = [Nobs 1]
% Xa - ones-augmented predictor matrix
% Xa = [ones(Nobs,1) X]
% b - linear regression coefficient matrix
% Xa*b = y, size(b) = [Nvar1+1 1]
%
% "SVD" - Singular value decomposition
% Xa = U*S*V'; Xa*V = U*S; Xa'*U = V*S
% S - Diagonal Singular Value matrix of Xa
% V - Unitary Right Singular Vector Matrix of Xa
% U - Unitary Left Singular Vector Matrix of Xa
%
% Predictor variables are linearly dependent if, and only
% if, solutions to the null space equation Xa*v = 0 exist.
% The null space is spanned by columns of V corresponding
% to zero singular values. Two forms of SVDs for Xa can
% be obtained from the MATLAB commands [U,S,V] = svd(Xa)
% and [U,S,V] = svd(Xa,0). S can also be obtained from
% S = svd(Xa) (Type "help svd" for details).
%
% Therefore, given y and linearly dependent X, b is not unique
% because arbitrary multiples of null space vectors can be added
% to b without changing y.
%
% In such cases, the MATLAB functions SLASH, REGRESS
% and LSCOV are supposed to yield a basic solution which has
% no more than rank(Xa) nonzero components corresponding
% to the number of linearly independent predictors.
%
% The matrix condition number cond(Xa) is the best machine
% independent quantitative indicator of both the degree of
% linear independence in X and uncertainty in b. Uncertainties
% in b caused by uncertainties in X and y are bounded by
%
% ||db||/||b|| <= cond(Xa)*[ ||dy||/||y|| + ||dX||/||X|| ]
%
% where ||.|| idicates matrix norm.
%
% MATLAB help information (2-norm assumed):
%
% norm(Xa) = max(S)
% cond(Xa) = max(S)/ min(S)
% eps = 2.22044604925031e-016 % (machine epsilon)
% tolXa = max(size(Xa))*norm(Xa)*eps
% = Nobs*max(S)*eps
% rank(Xa) = sum(S > tolXa)
%
% Traditionally, the reciprocal condition number is defined as
% rcondXa = 1/cond(Xa).
%
% When the above definitions are used, the example below
% would yield:
% S = svd(Xa)
% Smax = max(S) % 2.5933
% Smin = min(S) % 1.3963e-016
% normXa = Smax % 2.5933
% tolXa = max(size(Xa))*Smax*eps % 2.8792e-015
% rankXa = sum(S > tolXa ) % 3
% condXa = Smax/Smin % 1.8573e+016
% rcondXa = 1/condXa % 5.3841e-017
%
% However,(because the QR decomposition and matrix
% inversion algorithm of LAPACK is used?), the MATLAB
% functions SLASH, REGRESS and LSCOV use different
% definitions based on the 1-norm of R:
%
% norm(R,1) = max(sum(abs(R)))
% cond(R,1) = norm(R,1) * norm(inv(R),1) % R must be square!
% rcond(R) = 1/cond(R,1)
%
% SLASH: norm and tol
%
% [Q1,R1] = qr(Xa) % (NOT qr(Xa,0)
!)
% norm1R1 = norm(R1,1) % 2.2361
% tol = max(size(R1))*norm1R*eps % 2.4825e-015
%
% REGRESS and LSCOV: rcond (need a square R)
%
% [Q2,R2] = qr(Xa,0) % (NOT qr(Xa) !)
% rcondR2 = rcond(R2) % 2.6377e-017
%
% Notice that rcondR2 is NOT THE SAME as that quoted in the
% warnings of REGRESS and LSCOV caused by the attempted
% matrix division by the ill-conditioned R2. For example:
%
% IL = R2\R2
%
% Warning: Matrix is close to singular or badly scaled.
% Results may be inaccurate. RCOND = 1.055077e-016.
%
% The Example:

clear, close all, clc, format short g

X = [ 1/2 3/8 1/8;
3/8 1/2 1/8;
1/7 3/7 3/7;
1/9 1/3 5/9;
1/8 1/8 3/4];
[Nobs Nvar] = size(X) % [ 5 3]
Xa = [ones(Nobs,1) X];

b0 = [0 1 2 4 ]' % Note: zero intercept
y0 = Xa*b0

disp('')
disp('SLASH SOLUTION')
disp('')

b1 = Xa\y0;

% Warning: Rank deficient, rank = 3 tol = 2.4825e-015.
%
% Checking:
% [Q1,R1] = qr(Xa) %(NOT qr(Xa,0)
!)
% sizeR1 = size(R1) % [ 5 4 ]
% maxsizeR1 = max(sizeR1) % 5
% normR1 = norm(R1,1) % 2.2361
% tol1 = maxsizeR1*normR1*eps % 2.4825e-015
% smin1 = min(svd(R1)) % 1.1796e-016
% rankR1 = rank(R1,tol1) % 3

disp('')
disp('REGRESS SOLUTION')
disp('')

b2 = regress(y0,Xa);
% Warning: Matrix is close to singular or badly scaled.
% Results may be inaccurate. RCOND = 1.055077e-016.
%
% Checking:
% [Q2,R2] = qr(Xa,0) %(NOT qr(Xa) !)
% sizeR2 = size(R2) % [ 4 4 ]
% maxsizeR2 = max(sizeR2) % 4
% normR2 = norm(R2,1) % 2.2361
% tol2 = maxsizeR2*normR2*eps % 1.986e-015
% smin2 = min(svd(R2)) % 1.1796e-016
% rankR2 = rank(R2,tol2) % 3
%
% rcondR2 = rcond(R2) % 2.6377e-017 ???
% condR2 = cond(R2,1) % 3.7912e+016
% rcondR2 = 1/cond(R2,1) % 2.6377e-017 ??

disp('')
disp('LSCOV SOLUTION')
disp('')

b3 = lscov(Xa,y0,eye(Nobs));

% Warning: Matrix is close to singular or badly scaled.
% Results may be inaccurate. RCOND = 1.055077e-016.

B0 = [b1 b2 b3]

% B0 = [ 1 0.70588 0.70588;
% 0 0.29412 0.29412;
% 1 1.2941 1.2941;
% 3 3.2941 3.2941]

Y0 = repmat(y0,1,3);
R0 = Y0 - Xa*B0

% R0 = [ 0 0 0;
% 0 0 0;
% 0 0 0;
% 0 0 0;
% 0 0 0]

%%%%%%%%%%%% COMMENTS %%%%%%%%%%%%

% 1. SLASH yields a basic solution
% 2. REGRESS and LSCOV do not yield basic solutions!
% 3. Cannot verify "RCOND" in REGRESS/LCOV warnings
% 4. v = b0-b1 = [ -1 1 1 1 ] is a null vector
% indicating that x1+x2+x3 = 1 is the underlying
% linear dependence.
% 5. V(:,4) = -0.5*v is the singular vector corresponding
% to min(S) = 1.3963e-016
% 6. Combining b0 and b1 yields the basic solution
% matrix BASIC (to within computer precision)

BASIC = [ b0 b1 2*b1-b0 4*b1-3*b0]

% BASIC=
% [ 0 1 2 4;
% 1 0 -1 -3;
% 2 1 6.6613e-015 -2;
% 4 3 2 1.2434e-014]

Hope this helps.

Greg

Why would you want to build a model (or attempt to build a model) from
data in which the predictors are linearly dependent?
OMU

Old Mac User wrote:
Why would you want to build a model (or attempt to build a model) from
data in which the predictors are linearly dependent?
OMU

Very succinct, precise, and to the point!

Greg likens someone who takes a singular matrix to a matrix software
and complains that the software is unable to invert his sigular matrix
and there is not enough documentation to worn that a singular matrix
cannot be inverted.

-- Reef Fish Bob.


Greg Heath wrote:
The motivations for this post are
(1) The failure of MATLAB functions REGRESS and LSCOV to
yield basic solutions for linear regression when predictors
are linearly dependent.
(2) The lack of documentation on how the reciprocal condition
number "RCOND" is calculated when division by ill-conditioned
predictor matrix factors cause warnings to be written to the
command window.

The post is cross-posted to sci.stat.* because of previous interest
in the topic.

%2345678901234567890123456789012345678901234567890123456789012345
%
% The Effect of Linear Dependence on Linear Regression
%
% X - predictor matrix; size(X) = [ Nobs Nvar ]
% y - response matrix; size(y) = [Nobs 1]
% Xa - ones-augmented predictor matrix
% Xa = [ones(Nobs,1) X]
% b - linear regression coefficient matrix
% Xa*b = y, size(b) = [Nvar1+1 1]
%
% "SVD" - Singular value decomposition
% Xa = U*S*V'; Xa*V = U*S; Xa'*U = V*S
% S - Diagonal Singular Value matrix of Xa
% V - Unitary Right Singular Vector Matrix of Xa
% U - Unitary Left Singular Vector Matrix of Xa
%
% Predictor variables are linearly dependent if, and only
% if, solutions to the null space equation Xa*v = 0 exist.
% The null space is spanned by columns of V corresponding
% to zero singular values. Two forms of SVDs for Xa can
% be obtained from the MATLAB commands [U,S,V] = svd(Xa)
% and [U,S,V] = svd(Xa,0). S can also be obtained from
% S = svd(Xa) (Type "help svd" for details).
%
% Therefore, given y and linearly dependent X, b is not unique
% because arbitrary multiples of null space vectors can be added
% to b without changing y.
%
% In such cases, the MATLAB functions SLASH, REGRESS
% and LSCOV are supposed to yield a basic solution which has
% no more than rank(Xa) nonzero components corresponding
% to the number of linearly independent predictors.
%
% The matrix condition number cond(Xa) is the best machine
% independent quantitative indicator of both the degree of
% linear independence in X and uncertainty in b. Uncertainties
% in b caused by uncertainties in X and y are bounded by
%
% ||db||/||b|| <= cond(Xa)*[ ||dy||/||y|| + ||dX||/||X|| ]
%
% where ||.|| idicates matrix norm.
%
% MATLAB help information (2-norm assumed):
%
% norm(Xa) = max(S)
% cond(Xa) = max(S)/ min(S)
% eps = 2.22044604925031e-016 % (machine epsilon)
% tolXa = max(size(Xa))*norm(Xa)*eps
% = Nobs*max(S)*eps
% rank(Xa) = sum(S > tolXa)
%
% Traditionally, the reciprocal condition number is defined as
% rcondXa = 1/cond(Xa).
%
% When the above definitions are used, the example below
% would yield:
% S = svd(Xa)
% Smax = max(S) % 2.5933
% Smin = min(S) % 1.3963e-016
% normXa = Smax % 2.5933
% tolXa = max(size(Xa))*Smax*eps % 2.8792e-015
% rankXa = sum(S > tolXa ) % 3
% condXa = Smax/Smin % 1.8573e+016
% rcondXa = 1/condXa % 5.3841e-017
%
% However,(because the QR decomposition and matrix
% inversion algorithm of LAPACK is used?), the MATLAB
% functions SLASH, REGRESS and LSCOV use different
% definitions based on the 1-norm of R:
%
% norm(R,1) = max(sum(abs(R)))
% cond(R,1) = norm(R,1) * norm(inv(R),1) % R must be square!
% rcond(R) = 1/cond(R,1)
%
% SLASH: norm and tol
%
% [Q1,R1] = qr(Xa) % (NOT qr(Xa,0)
!)
% norm1R1 = norm(R1,1) % 2.2361
% tol = max(size(R1))*norm1R*eps % 2.4825e-015
%
% REGRESS and LSCOV: rcond (need a square R)
%
% [Q2,R2] = qr(Xa,0) % (NOT qr(Xa) !)
% rcondR2 = rcond(R2) % 2.6377e-017
%
% Notice that rcondR2 is NOT THE SAME as that quoted in the
% warnings of REGRESS and LSCOV caused by the attempted
% matrix division by the ill-conditioned R2. For example:
%
% IL = R2\R2
%
% Warning: Matrix is close to singular or badly scaled.
% Results may be inaccurate. RCOND = 1.055077e-016.
%
% The Example:

clear, close all, clc, format short g

X = [ 1/2 3/8 1/8;
3/8 1/2 1/8;
1/7 3/7 3/7;
1/9 1/3 5/9;
1/8 1/8 3/4];
[Nobs Nvar] = size(X) % [ 5 3]
Xa = [ones(Nobs,1) X];

b0 = [0 1 2 4 ]' % Note: zero intercept
y0 = Xa*b0

disp('')
disp('SLASH SOLUTION')
disp('')

b1 = Xa\y0;

% Warning: Rank deficient, rank = 3 tol = 2.4825e-015.
%
% Checking:
% [Q1,R1] = qr(Xa) %(NOT qr(Xa,0)
!)
% sizeR1 = size(R1) % [ 5 4 ]
% maxsizeR1 = max(sizeR1) % 5
% normR1 = norm(R1,1) % 2.2361
% tol1 = maxsizeR1*normR1*eps % 2.4825e-015
% smin1 = min(svd(R1)) % 1.1796e-016
% rankR1 = rank(R1,tol1) % 3

disp('')
disp('REGRESS SOLUTION')
disp('')

b2 = regress(y0,Xa);
% Warning: Matrix is close to singular or badly scaled.
% Results may be inaccurate. RCOND = 1.055077e-016.
%
% Checking:
% [Q2,R2] = qr(Xa,0) %(NOT qr(Xa) !)
% sizeR2 = size(R2) % [ 4 4 ]
% maxsizeR2 = max(sizeR2) % 4
% normR2 = norm(R2,1) % 2.2361
% tol2 = maxsizeR2*normR2*eps % 1.986e-015
% smin2 = min(svd(R2)) % 1.1796e-016
% rankR2 = rank(R2,tol2) % 3
%
% rcondR2 = rcond(R2) % 2.6377e-017 ???
% condR2 = cond(R2,1) % 3.7912e+016
% rcondR2 = 1/cond(R2,1) % 2.6377e-017 ??

disp('')
disp('LSCOV SOLUTION')
disp('')

b3 = lscov(Xa,y0,eye(Nobs));

% Warning: Matrix is close to singular or badly scaled.
% Results may be inaccurate. RCOND = 1.055077e-016.

B0 = [b1 b2 b3]

% B0 = [ 1 0.70588 0.70588;
% 0 0.29412 0.29412;
% 1 1.2941 1.2941;
% 3 3.2941 3.2941]

Y0 = repmat(y0,1,3);
R0 = Y0 - Xa*B0

% R0 = [ 0 0 0;
% 0 0 0;
% 0 0 0;
% 0 0 0;
% 0 0 0]

%%%%%%%%%%%% COMMENTS %%%%%%%%%%%%

% 1. SLASH yields a basic solution
% 2. REGRESS and LSCOV do not yield basic solutions!
% 3. Cannot verify "RCOND" in REGRESS/LCOV warnings
% 4. v = b0-b1 = [ -1 1 1 1 ] is a null vector
% indicating that x1+x2+x3 = 1 is the underlying
% linear dependence.
% 5. V(:,4) = -0.5*v is the singular vector corresponding
% to min(S) = 1.3963e-016
% 6. Combining b0 and b1 yields the basic solution
% matrix BASIC (to within computer precision)

BASIC = [ b0 b1 2*b1-b0 4*b1-3*b0]

% BASIC=
% [ 0 1 2 4;
% 1 0 -1 -3;
% 2 1 6.6613e-015 -2;
% 4 3 2 1.2434e-014]

Hope this helps.

Greg

Old Mac User wrote:
Why would you want to build a model (or attempt to build a model) from
data in which the predictors are linearly dependent?
OMU

1. To test the MATLAB claim that REGRESS and LSCOV will yield
basic solutions.

2. To understand why they do not.

3. To test the ridiculous advice in Wikipedia re detecting and
mitigating multicollinearity

Greg Heath wrote:
Old Mac User wrote:
Why would you want to build a model (or attempt to build a model) from
data in which the predictors are linearly dependent?
OMU

1. To test the MATLAB claim that REGRESS and LSCOV will yield
basic solutions.

Only if the variables are NOT linearly dependent.

2. To understand why they do not.

Because no software package can invert a singular matrix.

3. To test the ridiculous advice in Wikipedia re detecting and
mitigating multicollinearity

I don't know what Wikipedia advive is ridiculous.

Multicollinearity Condition in a regression is one of NEAR linear
dependence but NOT exact linear dependence. Even Wikipedia
acknowledged that while it may have meant exact collinearity at
one time (not in my lifetime in regression analysis),

Wikipedia > to mean a nearly perfect relationship.

A "nearly perfect" linear dependence relationship is FAR form
a linearly dependent set of variable.

In short, it was your misconception about multicollearity condition
to mean LINEAR DEPENDENCE that caused the problem you
observed in Matlab, not something that is MATLAB's fault.

Reef Fish wrote:
Greg Heath wrote:
Old Mac User wrote:
Why would you want to build a model (or attempt to build a model) from
data in which the predictors are linearly dependent?
OMU

1. To test the MATLAB claim that REGRESS and LSCOV will yield
basic solutions.

Only if the variables are NOT linearly dependent.

No, No, No!

Only if the variables ARE linearly dependent!

2. To understand why they do not.

Because no software package can invert a singular matrix.

You don't understand.

1. No matrix inversion is involved.
2. Xa is rectangular; size(Xa) = [Nobs Nvar+1] = [5 4] .
3. rank(Xa) = 3 indicating 1 linear dependence
(x1+x2+x3 = x0 = 1).
4. Xa*b = y has an infinite number of solutions for b
5. Nvar+1-rank(Xa) = 1 indicates that there are Nvar+1
= 4 BASIC solutions which can be obtained by
removing 1 of x0, x1, x2 or x3, respectively.
6. In another thread, a MATLAB employee has replied
that, in the case of linear dependence, REGRESS
and LSCOV will generate one of the 4 basic solutions.
7. My example DID yield a basic solution for the so-called
SLASH solution b = Xa\y.
8. My example DID NOT yield a basic solution from
either REGRESS or LSCOV.
9. From the reply of John D'Errico who DID obtain
a basic solution from REGRESS, it looks like the
basic solution default was not present in my old
version of MATLAB.

3. To test the ridiculous advice in Wikipedia re detecting and
mitigating multicollinearity

I don't know what Wikipedia advive is ridiculous.

"Ridiculous" was probably not the best word choice.
Nonetheless, I'm very surprised that you are satisfied with the
following:

http://en.wikipedia.org/wiki/Multicollinearity

START QUOTE

How to tell if you have multicollinearity:

1) Large changes in the estimated regression coefficients when a
predictor variable is added or deleted

2) Non significant results of simple linear regressions

3) Estimated regression coefficients have an opposite sign from
predicted

4) formal detection-tolerance or the variation inflation factor (VIF)
tolerance = R^2, VIF = 1/tolerance

A tolerance of less than 0.1 indicates a multicollinearity problem.

What to do...

1) The presence of multicollinearity doesn't affect the fitted model
provided that the predictor variables follow the same multicolinearity
pattern as the data on which the regression model is based.
2) A predictor variable may be dropped to lessen multicolinearity. (But
then you don't get any info from the dropped variable)
3) You may be able to add a case to break multicollinearity
4) Estimate the regression coefficients from different data sets

END QUOTE

Multicollinearity Condition in a regression is one of NEAR linear
dependence but NOT exact linear dependence. Even Wikipedia
acknowledged that while it may have meant exact collinearity at
one time (not in my lifetime in regression analysis),

Wikipedia > to mean a nearly perfect relationship.

A "nearly perfect" linear dependence relationship is FAR form
a linearly dependent set of variable.

Especially if you use a measure which is infinite for exact
linear dependence.

In short, it was your misconception about multicollearity condition
to mean LINEAR DEPENDENCE that caused the problem you
observed in Matlab, not something that is MATLAB's fault.

No, I have no misconception about that. The problem is I have an
old version of MATLAB that doesn't have some of the characteristics
referred to in the MATLAB newsgroup.

Greg Heath wrote:
Reef Fish wrote:
Greg Heath wrote:
Old Mac User wrote:
Why would you want to build a model (or attempt to build a model) from
data in which the predictors are linearly dependent?
OMU

1. To test the MATLAB claim that REGRESS and LSCOV will yield
basic solutions.

Only if the variables are NOT linearly dependent.

No, No, No!

Only if the variables ARE linearly dependent!

Really, Greg? MATLAB claimed that REGRESS will yield
basic solutions "if the variables ARE linearly dependent"?

You should learn to THINK for a minute or two before putting
your fingers to work on the keyboard. That has ALWAYS
been your problem.

Blurt nonsense BEFORE you read or understood what was
written.


2. To understand why they do not.

Because no software package can invert a singular matrix.

You don't understand.

1. No matrix inversion is involved.
2. Xa is rectangular; size(Xa) = [Nobs Nvar+1] = [5 4] .
3. rank(Xa) = 3 indicating 1 linear dependence
(x1+x2+x3 = x0 = 1).
4. Xa*b = y has an infinite number of solutions for b
5. Nvar+1-rank(Xa) = 1 indicates that there are Nvar+1
= 4 BASIC solutions which can be obtained by
removing 1 of x0, x1, x2 or x3, respectively.
6. In another thread, a MATLAB employee has replied
that, in the case of linear dependence, REGRESS
and LSCOV will generate one of the 4 basic solutions.
7. My example DID yield a basic solution for the so-called
SLASH solution b = Xa\y.
8. My example DID NOT yield a basic solution from
either REGRESS or LSCOV.
9. From the reply of John D'Errico who DID obtain
a basic solution from REGRESS, it looks like the
basic solution default was not present in my old
version of MATLAB.

I made it simple for everyone to see that a linearly DEPENDENT
set of predictor variables for OLS regression is equivalent to
an attempt to invert a singular matrix of X'X,

Your nine points add to one unsueable gibberish for the fitted
model which can be used to PREDICT.

The only thing you SLASH is your own statistical wrist.


3. To test the ridiculous advice in Wikipedia re detecting and
mitigating multicollinearity

I don't know what Wikipedia advive is ridiculous.

"Ridiculous" was probably not the best word choice.
Nonetheless, I'm very surprised that you are satisfied with the
following:

http://en.wikipedia.org/wiki/Multicollinearity

START QUOTE

How to tell if you have multicollinearity:

1) Large changes in the estimated regression coefficients when a
predictor variable is added or deleted

That is a correct symptom of mullticollnearity.

2) Non significant results of simple linear regressions

This is unrelated to multicollinearity.

3) Estimated regression coefficients have an opposite sign from
predicted

That is poorly stated for the fallacy of "expected signs" in a
multiple regression.

4) formal detection-tolerance or the variation inflation factor (VIF)
tolerance = R^2, VIF = 1/tolerance

That is one commonly used method, far from optimal.

A tolerance of less than 0.1 indicates a multicollinearity problem.

That is the same kind of Rule of Thump as your Rule about
simple correlations of > 0.9 or some arbitrary value.

What to do...

1) The presence of multicollinearity doesn't affect the fitted model
provided that the predictor variables follow the same multicolinearity
pattern as the data on which the regression model is based.
2) A predictor variable may be dropped to lessen multicolinearity. (But
then you don't get any info from the dropped variable)
3) You may be able to add a case to break multicollinearity
4) Estimate the regression coefficients from different data sets

END QUOTE

Nothing obviously wrong about ANY of the items (1) to (4).


Multicollinearity Condition in a regression is one of NEAR linear
dependence but NOT exact linear dependence. Even Wikipedia
acknowledged that while it may have meant exact collinearity at
one time (not in my lifetime in regression analysis),

Wikipedia > to mean a nearly perfect relationship.

A "nearly perfect" linear dependence relationship is FAR form
a linearly dependent set of variable.

Especially if you use a measure which is infinite for exact
linear dependence.

Greg, exact linear dependence is a LINEAR ALGEBRA notion and
definition. It's a very simple definition you should learn. NOTHING
is infinite in finding a linearly dependent set.


In short, it was your misconception about multicollearity condition
to mean LINEAR DEPENDENCE that caused the problem you
observed in Matlab, not something that is MATLAB's fault.

No, I have no misconception about that. The problem is I have an
old version of MATLAB that doesn't have some of the characteristics
referred to in the MATLAB newsgroup.


You dependence on MATLAB to learn your statistics is YOUR problem.

I had already given the reply above before I realize that you had
ALREADY
LOST your FREE Tuition and privilege to get Free lessons from me.

Rather than erasing what I had already wrritten, I'll leave it for the
OTHER readers.

Greg Heath has re-acquaired his Disqualification Status.

The depth of your ignorance about Multiple Regression is so abysmal
that I'll hencewoth invoke the same SHORT answers for all the
Disqualified Discussants

-- Reef Fish Bob.

Greg Heath wrote:
Reef Fish wrote:
Greg Heath wrote:
Old Mac User wrote:
Why would you want to build a model (or attempt to build a model)
from data in which the predictors are linearly dependent?
OMU

1. To test the MATLAB claim that REGRESS and LSCOV will yield
basic solutions.

Only if the variables are NOT linearly dependent.

No, No, No!

Only if the variables ARE linearly dependent!

Really, Greg? MATLAB claimed that REGRESS will yield
basic solutions "if the variables ARE linearly dependent"?

2. To understand why they do not.

Because no software package can invert a singular matrix.

You don't understand.

1. No matrix inversion is involved.
2. Xa is rectangular; size(Xa) = [Nobs Nvar+1] = [5 4] .
3. rank(Xa) = 3 indicating 1 linear dependence
(x1+x2+x3 = x0 = 1).
4. Xa*b = y has an infinite number of solutions for b
5. Nvar+1-rank(Xa) = 1 indicates that there are Nvar+1
= 4 BASIC solutions which can be obtained by
removing 1 of x0, x1, x2 or x3, respectively.
6. In another thread, a MATLAB employee has replied
that, in the case of linear dependence, REGRESS
and LSCOV will generate one of the 4 basic solutions.
7. My example DID yield a basic solution for the so-called
SLASH solution b = Xa\y.
8. My example DID NOT yield a basic solution from
either REGRESS or LSCOV.
9. From the reply of John D'Errico who DID obtain
a basic solution from REGRESS, it looks like the
basic solution default was not present in my old
version of MATLAB.

I made it simple for everyone to see that a linearly DEPENDENT
set of predictor variables for OLS regression is equivalent to
an attempt to invert a singular matrix of X'X,

Your nine points add to one unsueable gibberish for the fitted
model which can be used to PREDICT.

The only thing you SLASH is your own statistical wrist.

3. To test the ridiculous advice in Wikipedia re detecting and
mitigating multicollinearity

I don't know what Wikipedia advive is ridiculous.

"Ridiculous" was probably not the best word choice.
Nonetheless, I'm very surprised that you are satisfied with the
following:

http://en.wikipedia.org/wiki/Multicollinearity

START QUOTE

How to tell if you have multicollinearity:

1) Large changes in the estimated regression coefficients when a
predictor variable is added or deleted

That is a correct symptom of mullticollnearity.

2) Non significant results of simple linear regressions

This is unrelated to multicollinearity.

3) Estimated regression coefficients have an opposite sign from
predicted

That is poorly stated for the fallacy of "expected signs" in a
multiple regression.

4) formal detection-tolerance or the variation inflation factor (VIF)
tolerance = R^2, VIF = 1/tolerance

That is one commonly used method, far from optimal.

A tolerance of less than 0.1 indicates a multicollinearity problem.

That is the same kind of Rule of Thump as your Rule about
simple correlations of > 0.9 or some arbitrary value.

What to do...

1) The presence of multicollinearity doesn't affect the fitted model
provided that the predictor variables follow the same multicolinearity
pattern as the data on which the regression model is based.

2) A predictor variable may be dropped to lessen multicolinearity. (But
then you don't get any info from the dropped variable)

3) You may be able to add a case to break multicollinearity

4) Estimate the regression coefficients from different data sets

END QUOTE

Nothing obviously wrong about ANY of the items (1) to (4).

Multicollinearity Condition in a regression is one of NEAR linear
dependence but NOT exact linear dependence. Even Wikipedia
acknowledged that while it may have meant exact collinearity at
one time (not in my lifetime in regression analysis),

Wikipedia > to mean a nearly perfect relationship.

A "nearly perfect" linear dependence relationship is FAR form
a linearly dependent set of variable.

Especially if you use a measure which is infinite for exact
linear dependence.

Greg, exact linear dependence is a LINEAR ALGEBRA notion and
definition. It's a very simple definition you should learn. NOTHING
is infinite in finding a linearly dependent set.

In short, it was your misconception about multicollearity condition
to mean LINEAR DEPENDENCE that caused the problem you
observed in Matlab, not something that is MATLAB's fault.

No, I have no misconception about that. The problem is I have an
old version of MATLAB that doesn't have some of the characteristics
referred to in the MATLAB newsgroup.

You dependence on MATLAB to learn your statistics is YOUR problem.

I had already given the reply above before I realize that you had
ALREADY LOST your FREE Tuition and privilege to get Free
lessons from me.

Rather than erasing what I had already wrritten, I'll leave it for the
OTHER readers.

Greg Heath has re-acquaired his Disqualification Status.

The depth of your ignorance about Multiple Regression is so abysmal
that I'll hencewoth invoke the same SHORT answers for all the
Disqualified Discussants

On 2006-12-30 18:19:50 -0400, "Reef Fish"
large_nassua_grouper@yahoo.com> said:


Greg Heath wrote:
Reef Fish wrote:
Greg Heath wrote:
Old Mac User wrote:
Why would you want to build a model (or attempt to build a model) from
data in which the predictors are linearly dependent?
OMU

1. To test the MATLAB claim that REGRESS and LSCOV will yield
basic solutions.

Only if the variables are NOT linearly dependent.

No, No, No!

Only if the variables ARE linearly dependent!

Really, Greg? MATLAB claimed that REGRESS will yield
basic solutions "if the variables ARE linearly dependent"?

If the equation system is underdetermined you get involved in
notions from operations research and, in particular, linear
programming.

Basic is straight OR jargon that means that the solution is at
a vertex of some polyhedron. Nonbasic means that the solution is
a (nontrivial) combination of vertices. A common problem in
OR is that several vertices may give the same optimum value.
It is called a degenerate solution. Any combination of the vertices
will yield the same optimum so there is interest in finding
simple representations. Thus basic solutions become important.
Basic and nonbasic are also used to describe variables with the
connection requiring a small amount of discussion of the the
techniques to justify the usage.

It seems strange to see the usage of basic in the context of least
squares but the MatLab folks may have good reasons for it. Most
of their applicatons arise in engineering so it may be the case
that there is a least squares solution which is degenerate in
the OR sense. It would then be reasonable to ask if the MatLab
solutions are both small by being least squares and simple by
being basic in the OR sense. So their basic solution would be
special in a technical jargonistic sense that is not the same
as the common dictionary. A possible application area might
be filter design where least squares is an approximation mode
and the issue of simple solutions might be the number of filter
coefficients.

I have not looked at the MatLab stuff so have no idea what they
are offerring. But it is entirely possible that they are dealing
with a mix of approximation methods that jumbles the jargon of
both least squares and linear programming. I have not looked at
the rest of this thread so can not comment on it. But the notion
that the MatLab pseudoinverse operation has been enhanced to
provide a basic solution when there is a degeneracy, and the
additional computation option has been selected, seems to be
precisely the sort of thing one might expect from them.

Reef Fish wrote:
Greg Heath wrote:
Reef Fish wrote:
Greg Heath wrote:
Old Mac User wrote:
Why would you want to build a model (or attempt to build a model)
from data in which the predictors are linearly dependent?
OMU

1. To test the MATLAB claim that REGRESS and LSCOV will yield
basic solutions.

Only if the variables are NOT linearly dependent.

No, No, No!

Only if the variables ARE linearly dependent!

Really, Greg? MATLAB claimed that REGRESS will yield
basic solutions "if the variables ARE linearly dependent"?

YES!

Apparently you are IGNORANT about the concept of
BASIC solutions.

Have you gone to comp.soft-sys.matlab and read the 9
or more posts that didn't show up in sci.stat.*?

---- SNIP
I made it simple for everyone to see that a linearly DEPENDENT
set of predictor variables for OLS regression is equivalent to
an attempt to invert a singular matrix of X'X,

Although that is a RIDICULOUSLY worded sentence, I understand
what you are trying to say.

Boy, you are DENSE! Maybe you need to brush up on
your linear algebra and numerical analysis.

The only thing you SLASH is your own statistical wrist.

Your attempts at humor are humorous.

3. To test the ridiculous advice in Wikipedia re detecting and
mitigating multicollinearity

I don't know what Wikipedia advive is ridiculous.

"Ridiculous" was probably not the best word choice.
Nonetheless, I'm very surprised that you are satisfied with the
following:

http://en.wikipedia.org/wiki/Multicollinearity

START QUOTE

How to tell if you have multicollinearity:

1) Large changes in the estimated regression coefficients when a
predictor variable is added or deleted

That is a correct symptom of mullticollnearity.

2) Non significant results of simple linear regressions

This is unrelated to multicollinearity.

I see you agree with me.

3) Estimated regression coefficients have an opposite sign from
predicted

That is poorly stated for the fallacy of "expected signs" in a
multiple regression.

I see you agree with me.

4) formal detection-tolerance or the variation inflation factor (VIF)
tolerance = R^2, VIF = 1/tolerance

That is one commonly used method, far from optimal.

I see you agree with me.

HOWEVER, you are not very observant!
Is that the correct definition of tolerance?

A tolerance of less than 0.1 indicates a multicollinearity problem.

That is the same kind of Rule of Thump as your Rule about
simple correlations of > 0.9 or some arbitrary value.

I see you are up to your old trick of fabricating statements
and attributing them to me.

I have no Rule of Thumb about any correlation value (except 1)
that is without the context of the values of Nobs, Nvar and the
other correlations in the correlation matrix.

You must be confusing me with someone else...that
hurts my feelings!

Greg Heath has re-acquaired his Disqualification Status.

Considering the identity of the grantor, I am delighted to accept.

The depth of your ignorance about Multiple Regression is so abysmal
that I'll hencewoth invoke the same SHORT answers for all the
Disqualified Discussants

I'm sure they are all weeping with dismay.

The Opening post of this thread stated

GH> The motivations for this post are
GH> (1) The failure of MATLAB functions REGRESS and LSCOV to
GH> yield basic solutions for linear regression when
predictors
GH> are linearly dependent.

It was stated as a LINEAR REGRESSION problem when the predictors
are linearly dependent.

NOT as a question in the solution of degenerate linear equations.

As such, Greg Heath was completely out of line.

Reef Fish wrote:

The Opening post of this thread stated

GH> The motivations for this post are
GH> (1) The failure of MATLAB functions REGRESS and LSCOV to
GH> yield basic solutions for linear regression when
predictors
GH> are linearly dependent.

It was stated as a LINEAR REGRESSION problem when the predictors
are linearly dependent.

NOT as a question in the solution of degenerate linear equations.

As such, Greg Heath was completely out of line.

That is a completely RIDICULOUS statement since the question arose
from a linear regression problem.