Hi,

I have read the book "Neural Networks for Pattern Recognition" by
Bishop (1995). It explains that a logistic output unit should be used
for a 2-class classification problem and softmax output units should
be used for a multi-class classification problem - each softmax output
corresponds to a class label.

I am just wondering. Is it appropriate to use logistic units for multi-
class classification problems as well as 2-class problems?

Thanks,
David

If the classes are mutually exclusive then you wouldn't be exploiting
the prior knowledge that the outputs ashould sum to one.  In the limit
of an infinite dataset and a sufficiently large network then I expect
there would be no difference, but in the case of a finite dataset this
constraint on the outputs probably wouldn't hold.  One very practical
reason for using the softmax units would be that the network would not
need to devote hidden layer units to try and make the outputs sum
approximately to one, so you may need a larger network for logistic
output units.

In short, I wouldn't go as far to say logistic units would be
inappropriate, but they are certainly sub-optimal (assuming mutually
exclusive classes).

If the classes are mutually exclusive then you wouldn't be exploiting
the prior knowledge that the outputs ashould sum to one.  In the limit
of an infinite dataset and a sufficiently large network then I expect
there would be no difference, but in the case of a finite dataset this
constraint on the outputs probably wouldn't hold.  One very practical
reason for using the softmax units would be that the network would not
need to devote hidden layer units to try and make the outputs sum
approximately to one, so you may need a larger network for logistic
output units.

In short, I wouldn't go as far to say logistic units would be
inappropriate, but they are certainly sub-optimal (assuming mutually
exclusive classes).

By 'mutually exclusive' do you mean that each of the instances belongs
to 1 class only? On page 238 of Bishop's book, it says "If the output
values are to be interpreted as probabilities they must lie in the
range (0,1), and they must sum to unity. This can be achieved by using
a generalization of the logistic sigmoid activation function which
take the form etc.. which is known as th softmax activation function."
It seems that both logistic and softmax activation functions enable
that the sum of the outputs is one.

I have tried using both logistic and softmax outputs for a dataset.
The dataset contains 79 attributes, 5 classes and 208 instances. I
partitioned the dataset into  2/3 and 1/3 ratios. Then, I used 2/3
(139 instances) of the dataset as the training set and the 1/3 (69
instances) as testing dataset. I tried different network structures
for each type of outputs.

1. Logistic Ouputs

 Network Structure               classification accuracy using test
set (%)
 79 x 1 x 5                           82.7
  79 x 5 x 5                          95.7
  79 x 8 x 5                          98.6
  79 x 10 x 5                        97.1
  79 x 11 x 5                        97.1
  79 x 12 x 5                        97.1
  79 x 15 x 5                        97.1
  79 x 42 x 5                        97.1

2. Softmax Outputs

Network Structure              classification accuracy using test test
(%)

79 x 1 x 5                          82.7
79 x 4 x 5                          98.6
79 x 6 x 5                          97.1
79 x 8 x 5                          97.1
79 x 10 x 5                        97.1
79 x 15 x 5                        97.1
79 x 20 x 5                        97.1

3. Single layer networks

 I also tried 2 single layer networks: one network with 5 logistic
outputs, another with 5 softmax outputs.

a single layer network with 5 logistic outputs: 97.1

a single layer network with 5 softmax outputs: 97.1

The confusion matrices for both networks are identical:
                   predicted class
  actual  1
  class   2

Hi,

I have read the book "Neural Networks for Pattern Recognition" by
Bishop (1995). It explains that a logistic output unit should be used
for a 2-class classification problem and softmax output units should
be used for a multi-class classification problem - each softmax output
corresponds to a class label.

I am just wondering. Is it appropriate to use logistic units for multi-
class classification problems as well as 2-class problems?

If the classes are mutually exclusive then you wouldn't be exploiting
the prior knowledge that the outputs ashould sum to one. In the limit
of an infinite dataset and a sufficiently large network then I expect
there would be no difference, but in the case of a finite dataset this
constraint on the outputs probably wouldn't hold. One very practical
reason for using the softmax units would be that the network would not
need to devote hidden layer units to try and make the outputs sum
approximately to one, so you may need a larger network for logistic
output units.

In short, I wouldn't go as far to say logistic units would be
inappropriate, but they are certainly sub-optimal (assuming mutually
exclusive classes).

By 'mutually exclusive' do you mean that each of the instances belongs
to 1 class only?

On page 238 of Bishop's book, it says "If the output
values are to be interpreted as probabilities they must lie in the
range (0,1), and they must sum to unity.

This can be achieved by using
a generalization of the logistic sigmoid activation function which
take the form etc.. which is known as th softmax activation function."
It seems that both logistic and softmax activation functions enable
that the sum of the outputs is one.

I have tried using both logistic and softmax outputs for a dataset.
The dataset contains 79 attributes, 5 classes and 208 instances. I
partitioned the dataset into 2/3 and 1/3 ratios. Then, I used 2/3
(139 instances) of the dataset as the training set and the 1/3 (69
instances) as testing dataset. I tried different network structures
for each type of outputs.

1. Logistic Ouputs

Network Structure classification accuracy using test
set (%)
79 x 1 x 5 82.7
79 x 5 x 5 95.7
79 x 8 x 5 98.6
79 x 10 x 5 97.1
79 x 11 x 5 97.1
79 x 12 x 5 97.1
79 x 15 x 5 97.1
79 x 42 x 5 97.1

2. Softmax Outputs

Network Structure classification accuracy using test test
(%)

79 x 1 x 5 82.7
79 x 4 x 5 98.6
79 x 6 x 5 97.1
79 x 8 x 5 97.1
79 x 10 x 5 97.1
79 x 15 x 5 97.1
79 x 20 x 5 97.1

3. Single layer networks

I also tried 2 single layer networks: one network with 5 logistic
outputs, another with 5 softmax outputs.

a single layer network with 5 logistic outputs: 97.1

a single layer network with 5 softmax outputs: 97.1

A good generalization rule of thumb is

Ratio = Neq / Nw >> 1

A more tolerant ROT is Neq >~ 2*Nw

Thank you very much for your detailed suggestions. These are very
valueable to me.

A good generalization rule of thumb is

Ratio = Neq / Nw >> 1

A more tolerant ROT is Neq >~ 2*Nw

Could you please point me to the references explaining these
performance measures?

I have added a regularization term to the error function for a network
with 5 softmax outputs and the architecture 79 x 10 x 5. Let the error
function be
E + lambda*Omega
where Omega = 0.5*(sum of the squres of the weights) (as defined in
Bishop)
This is achieved by calling mlp(79,10,5,lambda) of Netlab. I used the
Scaled Conjugate Gradiant to train the network. The maximum no. of
iterations of SCG is set to 1000. I used different values for lambda.
For each lambda value, I ran SCG multiple times. All runs give almost
identical results in terms of error per epoch and accuracies.

The results are the following:

Iteration lambda Error per Iteration Accuracy (%)
1000 0.1 2.35 97.1
230 0.01 0.35 97.1
215 0.001 0.047 97.1
1000 0.002 0.086 97.1
350 0.003 0.123 97.1

All the different lambda values give the same accuracy 97.1%. The mlp
with regularization also gives 97.1%. It seems regularization does not
improve the accuracy of mlp for this dataset.

On Oct 4, 4:07 pm, David <dtian.ty... at (no spam) googlemail.com> wrote:
Thank you very much for your detailed suggestions. These are very
valueable to me.

A good generalization rule of thumb is

Ratio = Neq / Nw >> 1

A more tolerant ROT is Neq >~ 2*Nw

Could you please point me to the references explaining these
performance measures?

Any good reference on nonregularized linear regression should
mention practical values of the equations/unknowns ratio for precise
coefficient estimation. I have seen use of 3,5,8,15,20 and 30. If
linear regression errors are Normally distributed, then regression
coefficient estimates will be normally distributed and the standard
deviations of the coefficient error estimates will depend on the
ratio
via a Chi-Squared Distribution.

Also, try a Google searchs using

greg-heath Neq Nw

For most RW problems, distributions are not Gaussian and there is
no universal practical critical value for the ratio. The smallest
value
that yields useful results is problem dependent and the best bet for
confidence is to use trial and error.

The extension of the ROT to nonregularized nonlinear regression
has less theoretical support. Nevertheless it has proved extremely
useful when estimating practical lower bounds by trial and error.

I have added a regularization term to the error function for a network
with 5 softmax outputs and the architecture 79 x 10 x 5. Let the error
function be
E + lambda*Omega
where Omega = 0.5*(sum of the squres of the weights) (as defined in
Bishop)
This is achieved by calling mlp(79,10,5,lambda) of Netlab. I used the
Scaled Conjugate Gradiant to train the network. The maximum no. of
iterations of SCG is set to 1000. I used different values for lambda.
For each lambda value, I ran SCG multiple times. All runs give almost
identical results in terms of error per epoch and accuracies.

The results are the following:

Iteration lambda Error per Iteration Accuracy (%)
1000 0.1 2.35 97.1
230 0.01 0.35 97.1
215 0.001 0.047 97.1
1000 0.002 0.086 97.1
350 0.003 0.123 97.1

All the different lambda values give the same accuracy 97.1%. The mlp
with regularization also gives 97.1%. It seems regularization does not
improve the accuracy of mlp for this dataset.

That is why I said

"Typically, NN comparison studies are more convincing
when the problems are more difficult with larger error
rates"

... say > 5%.

For example, this problem could have been made harder
by adding computer generated noise to the input data and
using the smallest possible values of H.

It is also helpful to

1. Quote confidence levels. For example, assume sum of
regression squared errors are CHISQ distributed and
classification counts are binomially distributed. With Ntst
= 69, one classification error yields 1.45% and with a
similarly sized confidence interval.

2. For small data sets use multiple trials of f-fold cross-validation
to use all of the data (possibly multiple times) for error
estimation to reduce the relative size of confidence interval
to error estimate.

Hope this helps.

Greg

I'd like to know more about what kind of data David is building
models from. Or what kind of models.

Greg's comments are valid and helpful for data which is sampled
in proportion to how it would arise in "the real world".

However if David is building a choice model or something similar
where the data sampling has an underlying experimental design he
needs to be careful.

Classical choice models use MNLs where experimental designs
are based on optimising the estimation of all important parameters.

More recent Bayesian approaches also bias their sampling to
answer questions 'in simulation' with optimal precision, but for
questions/scenarios which are considered worthy of simulating.

These approaches bias the data collection to maximise information
for some kind of pre-supposed model form. MNLs have a bunch
of exp(sumproduct) predictions (normalised) so they yield normalised
(sums to 1.0000) classification probabilities. Bayesian optimality also
presumes an underlying Bayesian network (or at least drops some
conditional dependences).

So... ...if David is modelling this kind of data, a NN model may well
go off the rails. The NN models have more degrees of freedom and
can do a better job for natural sampling, but if the sampling has a
deliberate bias to answer presumed questions, the NNS can go wrong.

On Oct 5, 4:03 am, "Tomasso" <toma... at (no spam) a.a> wrote:
-----SNIP
I'd like to know more about what kind of data David is building
models from. Or what kind of models.

Greg's comments are valid and helpful for data which is sampled
in proportion to how it would arise in "the real world".

However if David is building a choice model or something similar
where the data sampling has an underlying experimental design he
needs to be careful.

Classical choice models use MNLs where experimental designs
are based on optimising the estimation of all important parameters.

Duh. Although it's late morning, I still can't deduce the meaning of
MNL

More recent Bayesian approaches also bias their sampling to
answer questions 'in simulation' with optimal precision, but for
questions/scenarios which are considered worthy of simulating.

These approaches bias the data collection to maximise information
for some kind of pre-supposed model form. MNLs have a bunch
of exp(sumproduct) predictions (normalised) so they yield normalised
(sums to 1.0000) classification probabilities. Bayesian optimality also
presumes an underlying Bayesian network (or at least drops some
conditional dependences).

So... ...if David is modelling this kind of data, a NN model may well
go off the rails. The NN models have more degrees of freedom and
can do a better job for natural sampling, but if the sampling has a
deliberate bias to answer presumed questions, the NNS can go wrong.

I feel comfortable when the following assumptions are satisfied:

1. There is an underlying deterministic nonpathological I/O
relationship.
2. Data are random samples from the I/O probability distributions.
3. The model is sufficiently flexible for approximating the I/O
relationship.
4. The learning algorithm is sufficient for obtaining accurate
estimates
of the model parameters.
5. The amount of data is sufficient for accurate estimation of model
parameters and performance criteria.

Your interesting comments reinforce my concern when assumption 2
is not satisfied.

Hope this helps.

Greg

The training and testing data that I used are presented below.

I feel comfortable when the following assumptions are satisfied:

1. There is an underlying deterministic nonpathological I/O
relationship.
2. Data are random samples from the I/O probability distributions.
3. The model is sufficiently flexible for approximating the I/O
relationship.
4. The learning algorithm is sufficient for obtaining accurate
estimates
of the model parameters.
5. The amount of data is sufficient for accurate estimation of model
parameters and performance criteria.

On 5-Oct-2009, Greg <he... at (no spam) alumni.brown.edu> wrote:

I feel comfortable when the following assumptions are satisfied:

1. There is an underlying deterministic nonpathological I/O
relationship.
2. Data are random samples from the I/O probability distributions.
3. The model is sufficiently flexible for approximating the I/O
relationship.
4. The learning algorithm is sufficient for obtaining accurate
estimates
of the model parameters.
5. The amount of data is sufficient for accurate estimation of model
parameters and performance criteria.

I agree with all of that. But I would add that you need to have a roughly
equal balance of the categories you are trying to predict.

Building a useful model where one category has 100 times as many
training cases as another is very difficult.