Don't put too much faith in the value of the training set error.
See the FAQ and archives re validation sets and overfitting
(overtraining).

Don't put too much faith in the value of the training set error.
See the FAQ and archives re validation sets and overfitting
(overtraining).

First, apologies for my long-winded reply.

On Sep 11, 11:07 am, Greg <he... at (no spam) alumni.brown.edu> wrote:
Don't put too much faith in the value of the training set error.
See the FAQ and archives re validation sets and overfitting
(overtraining).

I wouldn't put too much faith in the value of training set
error if my inequality is true:

(variance of predicted error) > (variance of training error)

Regarding training set error in general:

What I believe I have discovered is that this error is a function of
the output magnitude.

For example, if you train until you reach an error of .01,

there is no
way to know what the error associated with a particular output
magnitude interval is. In other words, this error is a kind of
aggregate error. It gives you no indication as to how much better
high magnitude outputs are fit and no indication of how poorly low
output magnitudes are fit.

About the only thing you can prove is that there exists a "middle"
interval which has an error of .01

This is fairly useless and even more useless when combined with my
inequality hypothesis.
________________________________

But here is the interesting part. Can it be that training error is so
useless because it is a non-uniform property and does not represent a
great majority of cases?

Keep in mind the contention that Memorization results when you "learn
the noise" of the input cases.

If that is true, then "learning the noise" would mean that the input
cases would have to be nailed with little error by the network. (A)

And the inverse, NOT "learning the noise" would mean that the
inputcases would NOT have to be nailed with little error by the
network. (B)

My inequality holds for (A) and I believe it should hold for (B).

NOTE: When simplistic plots of generalization & training error -v-
training are rendered, the training error curve is usually always
below the generalized error curve. This is exactly the inequality I
believe is true.

But So What?

I believe it means the following:

a) if you can't begin to fit the training data, you have no hope
whatsoever or any meaningful generalization.

b) if you training algorithm favors output magnitude intervals, it is
difficult to say anything about the results of your training.

c) further, a training method which would treat output magnitudes
without bias, should produce better generalization since you don't
overtrain big magnitudes and undertrain small magnitudes.

Concluding:

I think something needs to be "adjusted", "fudged", or weighted so
that uniform training occurs.

I also believe that this phenomenon is currently dealt with "down
wind" where weight values are adjusted by things like regularization
etc.

Perhaps a little effort "up wind" might be worth investing.

What is the transfer function on your output layer? Linear, logistic, exponential, etc?

What kind of function are you trying to predict (in terms of the above).

Are there sampling biases across the range of output values? Eg, are large values also
rare?

Your "inequality" isn't necessarily true.

Tom (the other one).

"TomH488" <tom... at (no spam) gmail.com> wrote in message

news:0f66a3e4-07d5-4f58-81a5-69a8bc919961 at (no spam) s39g2000yqj.googlegroups.com...
First, apologies for my long-winded reply.

On Sep 11, 11:07 am, Greg <he... at (no spam) alumni.brown.edu> wrote:



Don't put too much faith in the value of the training set error.
See the FAQ and archives re validation sets and overfitting
(overtraining).

I wouldn't put too much faith in the value of training set error if my
inequality is true:

(variance of predicted error) > (variance of training error)

Regarding training set error in general:

What I believe I have discovered is that this error is a function of
the output magnitude.

For example, if you train until you reach an error of .01, there is no
way to know what the error associated with a particular output
magnitude interval is.  In other words, this error is a kind of
aggregate error.  It gives you no indication as to how much better
high magnitude outputs are fit and no indication of how poorly low
output magnitudes are fit.

About the only thing you can prove is that there exists a "middle"
interval which has an error of .01

This is fairly useless and even more useless when combined with my
inequality hypothesis.
________________________________

But here is the interesting part.  Can it be that training error is so
useless because it is a non-uniform property and does not represent a
great majority of cases?

Keep in mind the contention that Memorization results when you "learn
the noise" of the input cases.

If that is true, then "learning the noise" would mean that the input
cases would have to be nailed with little error by the network.  (A)

And the inverse, NOT "learning the noise" would mean that the
inputcases would NOT have to be nailed with little error by the
network.  (B)

My inequality holds for (A) and I believe it should hold for (B).

NOTE:  When simplistic plots of generalization & training error -v-
training are rendered, the training error curve is usually always
below the generalized error curve.  This is exactly the inequality I
believe is true.
_______________________________

But So What?

I believe it means the following:

a) if you can't begin to fit the training data, you have no hope
whatsoever or any meaningful generalization.

b) if you training algorithm favors output magnitude intervals, it is
difficult to say anything about the results of your training.

c) further, a training method which would treat output magnitudes
without bias, should produce better generalization since you don't
overtrain big magnitudes and undertrain small magnitudes.

Concluding:

I think something needs to be "adjusted", "fudged", or weighted so
that uniform training occurs.

I also believe that this phenomenon is currently dealt with "down
wind" where weight values are adjusted by things like regularization
etc.

Perhaps a little effort "up wind" might be worth investing.

I'll attempt to answer "The Other Tom's" questions first (!)  (Greg,
you will be next)

On Sep 11, 11:21 pm, "Tomasso" <toma... at (no spam) a.a> wrote:

What is the transfer function on your output layer? Linear, logistic, exponential, etc?

Single output with linear activation or transfer function.



What kind of function are you trying to predict (in terms of the above)..

The output is "5 day change in price of the Russell 2000"

The input are "images" of stock charts.  These "images" are large
samplings of the stock price time history.  Many sampling models have
been tried.  Currently, a very simple, "last 25 days" is being used to
as part of an ensemble model that includes other types of inputs/
models.



Are there sampling biases across the range of output values? Eg, are large values also
rare?

The distribution of outputs is not terribly unlike a Normal
distribution centered at Zero.

Usually, for even output magnitude intervals, the variance of each
interval, plotted on semi-log appears as a straight line.

The larger the output, the greater the importance as far as investing
goes.
In fact, a traditional MSE is not a good metric since getting the
"correct sign" is the most important - I also plan to investigate
models where the output will be +1 or -1.  This may be another member
of the ensemble.





Your "inequality" isn't necessarily true.

Tom (the other one).

"TomH488" <tom... at (no spam) gmail.com> wrote in message

news:0f66a3e4-07d5-4f58-81a5-69a8bc919961 at (no spam) s39g2000yqj.googlegroups.com....
First, apologies for my long-winded reply.

On Sep 11, 11:07 am, Greg <he... at (no spam) alumni.brown.edu> wrote:

Don't put too much faith in the value of the training set error.
See the FAQ and archives re validation sets and overfitting
(overtraining).

I wouldn't put too much faith in the value of training set error if my
inequality is true:

(variance of predicted error) > (variance of training error)

Regarding training set error in general:

What I believe I have discovered is that this error is a function of
the output magnitude.

For example, if you train until you reach an error of .01, there is no
way to know what the error associated with a particular output
magnitude interval is.  In other words, this error is a kind of
aggregate error.  It gives you no indication as to how much better
high magnitude outputs are fit and no indication of how poorly low
output magnitudes are fit.

About the only thing you can prove is that there exists a "middle"
interval which has an error of .01

This is fairly useless and even more useless when combined with my
inequality hypothesis.
________________________________

But here is the interesting part.  Can it be that training error is so
useless because it is a non-uniform property and does not represent a
great majority of cases?

Keep in mind the contention that Memorization results when you "learn
the noise" of the input cases.

If that is true, then "learning the noise" would mean that the input
cases would have to be nailed with little error by the network.  (A)

And the inverse, NOT "learning the noise" would mean that the
inputcases would NOT have to be nailed with little error by the
network.  (B)

My inequality holds for (A) and I believe it should hold for (B).

NOTE:  When simplistic plots of generalization & training error -v-
training are rendered, the training error curve is usually always
below the generalized error curve.  This is exactly the inequality I
believe is true.
_______________________________

But So What?

I believe it means the following:

a) if you can't begin to fit the training data, you have no hope
whatsoever or any meaningful generalization.

b) if you training algorithm favors output magnitude intervals, it is
difficult to say anything about the results of your training.

c) further, a training method which would treat output magnitudes
without bias, should produce better generalization since you don't
overtrain big magnitudes and undertrain small magnitudes.

Concluding:

I think something needs to be "adjusted", "fudged", or weighted so
that uniform training occurs.

I also believe that this phenomenon is currently dealt with "down
wind" where weight values are adjusted by things like regularization
etc.

Perhaps a little effort "up wind" might be worth investing

On Sep 13, 4:11 pm, TomH488 <tom... at (no spam) gmail.com> wrote:
...
In the few stock market prediction problems that I have tried, the
inportant outputs were either change in price or per cent change
in price. If more than one price was being forecasted, the data
was always normalized so that neither value nor range of values
of one output had a deleterious effect on the prediction of other
outputs.

Greg

On Sep 13, 4:11 pm, TomH488 <tom... at (no spam) gmail.com> wrote:



I'll attempt to answer "The Other Tom's" questions first (!)  (Greg,
you will be next)

On Sep 11, 11:21 pm, "Tomasso" <toma... at (no spam) a.a> wrote:

What is the transfer function on your output layer? Linear, logistic, exponential, etc?

Single output with linear activation or transfer function.

What kind of function are you trying to predict (in terms of the above).

The output is "5 day change in price of the Russell 2000"

The input are "images" of stock charts.  These "images" are large
samplings of the stock price time history.  Many sampling models have
been tried.  Currently, a very simple, "last 25 days" is being used to
as part of an ensemble model that includes other types of inputs/
models.

Are there sampling biases across the range of output values? Eg, are large values also
rare?

The distribution of outputs is not terribly unlike a Normal
distribution centered at Zero.

Usually, for even output magnitude intervals, the variance of each
interval, plotted on semi-log appears as a straight line.

The larger the output, the greater the importance as far as investing
goes.
In fact, a traditional MSE is not a good metric since getting the
"correct sign" is the most important - I also plan to investigate
models where the output will be +1 or -1.  This may be another member
of the ensemble.

Your "inequality" isn't necessarily true.

Tom (the other one).

"TomH488" <tom... at (no spam) gmail.com> wrote in message

news:0f66a3e4-07d5-4f58-81a5-69a8bc919961 at (no spam) s39g2000yqj.googlegroups.com....
First, apologies for my long-winded reply.

On Sep 11, 11:07 am, Greg <he... at (no spam) alumni.brown.edu> wrote:

Don't put too much faith in the value of the training set error.
See the FAQ and archives re validation sets and overfitting
(overtraining).

I wouldn't put too much faith in the value of training set error if my
inequality is true:

(variance of predicted error) > (variance of training error)

Regarding training set error in general:

What I believe I have discovered is that this error is a function of
the output magnitude.

For example, if you train until you reach an error of .01, there is no
way to know what the error associated with a particular output
magnitude interval is.  In other words, this error is a kind of
aggregate error.  It gives you no indication as to how much better
high magnitude outputs are fit and no indication of how poorly low
output magnitudes are fit.

About the only thing you can prove is that there exists a "middle"
interval which has an error of .01

This is fairly useless and even more useless when combined with my
inequality hypothesis.
________________________________

But here is the interesting part.  Can it be that training error is so
useless because it is a non-uniform property and does not represent a
great majority of cases?

Keep in mind the contention that Memorization results when you "learn
the noise" of the input cases.

If that is true, then "learning the noise" would mean that the input
cases would have to be nailed with little error by the network.  (A)

And the inverse, NOT "learning the noise" would mean that the
inputcases would NOT have to be nailed with little error by the
network.  (B)

My inequality holds for (A) and I believe it should hold for (B).

NOTE:  When simplistic plots of generalization & training error -v-
training are rendered, the training error curve is usually always
below the generalized error curve.  This is exactly the inequality I
believe is true.
_______________________________

But So What?

I believe it means the following:

a) if you can't begin to fit the training data, you have no hope
whatsoever or any meaningful generalization.

b) if you training algorithm favors output magnitude intervals, it is
difficult to say anything about the results of your training.

c) further, a training method which would treat output magnitudes
without bias, should produce better generalization since you don't
overtrain big magnitudes and undertrain small magnitudes.

Concluding:

I think something needs to be "adjusted", "fudged", or weighted so
that uniform training occurs.

I also believe that this phenomenon is currently dealt with "down
wind" where weight values are adjusted by things like regularization
etc.

Perhaps a little effort "up wind" might be worth investing

In the few stock market prediction problems that I have tried, the
inportant outputs were either change in price or per cent change
in price. If more than one price was being forecasted, the data
was always normalized so that neither value nor range of values
of one output had a deleterious effect on the prediction of other
outputs.

Hope this helps.

Greg

Greg wrote:
On Sep 13, 4:11 pm, TomH488 <tom... at (no spam) gmail.com> wrote:
...
In the few stock market prediction problems that I have tried, the
inportant outputs were either change in price or per cent change
in price. If more than one price was being forecasted, the data
was always normalized so that neither value nor range of values
of one output had a deleterious effect on the prediction of other
outputs.

Greg

Market time series usually contain a mixture of noise. There is gaussian
noise, price movements that happen extraneously (eg, dividend payments),
and impulsive noise (spikes) which is rarer, but still happens a lot.

NNs handle gaussian noise well. Extraneous movements should be adjusted
for. BUT spikes cause learning problems for NNs. They shock the convergence
process, so learning does not get very far, and sometimes you merely learn
to fit the spikes. They are like very nasty outliers. And that might explain
TomH's Q.

I am aware of trading systems which filter the spikes out and learn from
the rest of the data. Then they use the spikes for triggering trades. This
can work for tick data and 15 minute data.

Tomasso.

On Sep 14, 5:08 pm, Greg <he... at (no spam) alumni.brown.edu> wrote:





On Sep 13, 4:11 pm, TomH488 <tom... at (no spam) gmail.com> wrote:

I'll attempt to answer "The Other Tom's" questions first (!)  (Greg,
you will be next)

On Sep 11, 11:21 pm, "Tomasso" <toma... at (no spam) a.a> wrote:

What is the transfer function on your output layer? Linear, logistic, exponential, etc?

Single output with linear activation or transfer function.

What kind of function are you trying to predict (in terms of the above).

The output is "5 day change in price of the Russell 2000"

The input are "images" of stock charts.  These "images" are large
samplings of the stock price time history.  Many sampling models have
been tried.  Currently, a very simple, "last 25 days" is being used to
as part of an ensemble model that includes other types of inputs/
models.

Are there sampling biases across the range of output values? Eg, are large values also
rare?

The distribution of outputs is not terribly unlike a Normal
distribution centered at Zero.

Usually, for even output magnitude intervals, the variance of each
interval, plotted on semi-log appears as a straight line.

The larger the output, the greater the importance as far as investing
goes.
In fact, a traditional MSE is not a good metric since getting the
"correct sign" is the most important - I also plan to investigate
models where the output will be +1 or -1.  This may be another member
of the ensemble.

Your "inequality" isn't necessarily true.

Tom (the other one).

"TomH488" <tom... at (no spam) gmail.com> wrote in message

news:0f66a3e4-07d5-4f58-81a5-69a8bc919961 at (no spam) s39g2000yqj.googlegroups.com...
First, apologies for my long-winded reply.

On Sep 11, 11:07 am, Greg <he... at (no spam) alumni.brown.edu> wrote:

Don't put too much faith in the value of the training set error.
See the FAQ and archives re validation sets and overfitting
(overtraining).

I wouldn't put too much faith in the value of training set error if my
inequality is true:

(variance of predicted error) > (variance of training error)

Regarding training set error in general:

What I believe I have discovered is that this error is a function of
the output magnitude.

For example, if you train until you reach an error of .01, there is no
way to know what the error associated with a particular output
magnitude interval is.  In other words, this error is a kind of
aggregate error.  It gives you no indication as to how much better
high magnitude outputs are fit and no indication of how poorly low
output magnitudes are fit.

About the only thing you can prove is that there exists a "middle"
interval which has an error of .01

This is fairly useless and even more useless when combined with my
inequality hypothesis.
________________________________

But here is the interesting part.  Can it be that training error is so
useless because it is a non-uniform property and does not represent a
great majority of cases?

Keep in mind the contention that Memorization results when you "learn
the noise" of the input cases.

If that is true, then "learning the noise" would mean that the input
cases would have to be nailed with little error by the network.  (A)

And the inverse, NOT "learning the noise" would mean that the
inputcases would NOT have to be nailed with little error by the
network.  (B)

My inequality holds for (A) and I believe it should hold for (B).

NOTE:  When simplistic plots of generalization & training error -v-
training are rendered, the training error curve is usually always
below the generalized error curve.  This is exactly the inequality I
believe is true.
_______________________________

But So What?

I believe it means the following:

a) if you can't begin to fit the training data, you have no hope
whatsoever or any meaningful generalization.

b) if you training algorithm favors output magnitude intervals, it is
difficult to say anything about the results of your training.

c) further, a training method which would treat output magnitudes
without bias, should produce better generalization since you don't
overtrain big magnitudes and undertrain small magnitudes.

Concluding:

I think something needs to be "adjusted", "fudged", or weighted so
that uniform training occurs.

I also believe that this phenomenon is currently dealt with "down
wind" where weight values are adjusted by things like regularization
etc.

Perhaps a little effort "up wind" might be worth investing

In the few stock market prediction problems that I have tried, the
inportant outputs were either change in price or per cent change
in price. If more than one price was being forecasted, the data
was always normalized so that neither value nor range of values
of one output had a deleterious effect on the prediction of other
outputs.

Hope this helps.

Greg

I'm pretty much in agreement with you except that we decided to
abandon multiple outputs.

In the very beginning we predicted a +1 and +2 days into future price
but then decided to build and train, 2 separate nets for that desired
result.  The intent was to not have one output error interact with the
other output error.  Whether that was a wise decision is unknown at
this time since that interaction may be more of a smoothing effect
which would be welcome.

On Sep 16, 6:21 pm, "Tomasso" <toma... at (no spam) a.a> wrote:
Greg wrote:
On Sep 13, 4:11 pm, TomH488 <tom... at (no spam) gmail.com> wrote:
...
In the few stock market prediction problems that I have tried, the
inportant outputs were either change in price or per cent change
in price. If more than one price was being forecasted, the data
was always normalized so that neither value nor range of values
of one output had a deleterious effect on the prediction of other
outputs.

Greg

Market time series usually contain a mixture of noise. There is gaussian
noise, price movements that happen extraneously (eg, dividend payments),
and impulsive noise (spikes) which is rarer, but still happens a lot.

When dealing with dP (change in price), a gap turns into a spike.
Usually gaps occur when there is news, both related and unrelated to
the stock.
So those points are impossible to predict or learn from - unless maybe
you
have some kind of news input.


NNs handle gaussian noise well. Extraneous movements should be adjusted
for. BUT spikes cause learning problems for NNs. They shock the convergence
process, so learning does not get very far, and sometimes you merely learn
to fit the spikes. They are like very nasty outliers. And that might explain
TomH's Q.

It sure does to me. I found that the net was fitting the large delta
outputs which are
a small fraction of the input. While I wouldn't call them spikes, I'd
call them concentrated
regions of extreme price movement.

If I removed the upper 50% of the output magnitude, I think I would
only remove about
10% of the data.

I did make a run where I took my lowest interval of price delta,
duplicated it 3 times and added
it to the input. The convergence was more difficult but the
forecasting on some preliminary data
was dramatically different. I will be taking a broad look at its
performance to see if any improved
generalization occurred.



I am aware of trading systems which filter the spikes out and learn from
the rest of the data. Then they use the spikes for triggering trades. This
can work for tick data and 15 minute data.

That sounds interesting, learning the rest of the data is like
characterizing the
state of the stock such as overbought or oversold, and then using the
spike
as an external force acting on our "spring-mass stock model." For
example,
if the stock is on the verge of breaking down and we get a spike
(news) that
reinforces this direction, it is probably a good time to short.

I did make a run where I took my lowest interval of price delta,
duplicated it 3 times and added
it to the input.  The convergence was more difficult but the
forecasting on some preliminary data
was dramatically different.  I will be taking a broad look at its
performance to see if any improved
generalization occurred.

Market time series usually contain a mixture of noise. There is gaussian
noise, price movements that happen extraneously (eg, dividend payments),
and impulsive noise (spikes) which is rarer, but still happens a lot.

When dealing with dP (change in price), a gap turns into a spike.
Usually gaps occur when there is news, both related and unrelated to
the stock.
So those points are impossible to predict or learn from - unless maybe
you
have some kind of news input.

NNs handle gaussian noise well. Extraneous movements should be adjusted
for. BUT spikes cause learning problems for NNs. They shock the convergence
process, so learning does not get very far, and sometimes you merely learn
to fit the spikes. They are like very nasty outliers. And that might explain
TomH's Q.

It sure does to me.  I found that the net was fitting the large delta
outputs which are
a small fraction of the input.  While I wouldn't call them spikes, I'd
call them concentrated
regions of extreme price movement.

In the literature it's called Impulsive noise. I happens in signal processing
where events like short circuits and lightning stikes add large noise, usually
unrelated to the channel and the channel's Gaussian noise.

Other things you should experiment with:

1. Whether you suppress the spikes or mask them without losing them
completely. There are several kinds of filter that can mask the spikes,
the simplest being a median filter...

2. Whether you suppress them for the output of the NN data, or both
the output and input. The spikes in output screw up the error feedback
and hence convergence. The spikes in input may have a predictive role
for price time series.

Since my input is a sequence of 25 daily closing prices, output spikes

If I removed the upper 50% of the output magnitude, I think I would
only remove about
10% of the data.

This is confirmed to be true.

I did make a run where I took my lowest interval of price delta,
duplicated it 3 times and added
it to the input.  The convergence was more difficult but the
forecasting on some preliminary data
was dramatically different.  I will be taking a broad look at its
performance to see if any improved
generalization occurred.

I am aware of trading systems which filter the spikes out and learn from
the rest of the data. Then they use the spikes for triggering trades. This
can work for tick data and 15 minute data.

That sounds interesting, learning the rest of the data is like
characterizing the
state of the stock such as overbought or oversold, and then using the
spike
as an external force acting on our "spring-mass stock model."   For
example,
if the stock is on the verge of breaking down and we get a spike
(news) that
reinforces this direction, it is probably a good time to short.

What ratio of signal to noise are you getting in your models?

Tomasso.

Tomasso.