 |
|
| Computers Forum Index » Computer Artificial Intelligence - Neural Nets » Early stopping in cross Validation... |
|
Page 1 of 1 |
|
| Author |
Message |
| Tony... |
 Posted: Thu Jun 04, 2009 3:48 am |
|
|
|
Guest
|
Hello all,
I want to use early stopping in my 10-fold cross validation neural net
training.
I'm using newff along with train functions to create and train the NN.
As far as I know, train function divides the dataset into tr/val/tst
sets automatically. how can I set "train" function to use my
designated folds (8-1-1) for tr/val/tst?
I have used net.divideParam.trainRatio=0.8889;
net.divideParam.valRatio=0.1111;
and net.divideParam.testRatio=0; meaning 1/9(folds) =0.11 % of
data are used for validation and 1-0.11=0.89% are used for training.
Is there a better way to do that?
here is some parts of my code:
indices=crossvalind('kfold',Tar,10);
for i=1:10
test=(indices==i);trains= ~test;
net=newff(P1(:,trains),Tar(:,trains),4);
net=init(net);
net.divideParam.trainRatio=0.8889;
net.divideParam.valRatio=0.1111;
net.divideParam.testRatio=0;
[net,tr]=train(net,P1(:,trains),Tar(:,trains));
out = round(sim(net,P(:,test)));
%%% evaluate performance......
end
I appreciate your help and comments...
-Tony |
|
|
| Back to top |
|
|
|
| Greg Heath... |
| Posted: Fri Jun 05, 2009 3:32 pm |
|
|
|
Guest
|
On Jun 3, 11:48 pm, Tony <tanaby.zibamanzar.mof... at (no spam) gmail.com> wrote:
Quote: Hello all,
I want to use early stopping in my 10-fold cross validation neural net
training.
I'm using newff along with train functions to create and train the NN.
As far as I know, train function divides the dataset into tr/val/tst
sets automatically.
It depends on what is specified in the DDF input of NEWFF.
Quote: how can I set "train" function to use my
designated folds (8-1-1) for tr/val/tst?
What version do you have? The commands for the 2009a version
of TRAIN is
[net, tr, Y, E, Pf, Af] = train( net, P, T, Pi, Ai )
which, for some god-forsaken reason, contains no reference to data
division.
On the other hand, the command for the 2009a version of TRAINLM is
[net,TR] = trainlm(net,TR,trainV,valV,testV)
So I guess MATLAB thinks that the following makes sense:
net = newff( P, T, [ S1 S2...S(N-1) ], { TF1 TF2...TFN }, ...
BTF, BLF, PF, IPF, OPF, DDF)
1. In NEWFF override the default for DDF to get the 8/1/1 split but
use
the default for BTF to use TRAINLM.
2. Then TRAIN will call the default TRAINLM which will obtain the
divide
information from the override specification of DDF in NEWFF.
YIKES!
Quote: I have used net.divideParam.trainRatio=0.8889;
net.divideParam.valRatio=0.1111;
and net.divideParam.testRatio=0; meaning 1/9(folds) =0.11 % of
data are used for validation and 1-0.11=0.89% are used for training.
Is there a better way to do that?
What about
net.divideParam.trainRatio = 0.8;
net.divideParam.valRatio = 0.1;
net.divideParam.testRatio = 0.1;
Quote: here is some parts of my code:
indices=crossvalind('kfold',Tar,10);
for i=1:10
test=(indices==i);trains= ~test;
BZZT. TRAINS is a MATLAB function. Use another name.
What happened to val?
Why not something like
tst = (indices==i);
val = (indices== mod(i+1,10));
trn = ~[tst,val];
Quote: net=newff(P1(:,trains),Tar(:,trains),4);
net=init(net);
Init not needed. Initialization is automatic.
Quote: net.divideParam.trainRatio=0.8889;
net.divideParam.valRatio=0.1111;
net.divideParam.testRatio=0;
Why not override DDF in the call of NEWFF?
Quote: [net,tr]=train(net,P1(:,trains),Tar(:,trains));
out = round(sim(net,P(:,test)));
%%% evaluate performance......
end
I appreciate your help and comments...
The new code is so convoluted, it is impossible to understand how
to do this without trial and error. If I had to use the 2009a
version
I would try to use TRAINLM directly instead of TRAIN.
Hope this helps.
Greg |
|
|
| Back to top |
|
|
|
|
|
All times are GMT
The time now is Sun Mar 21, 2010 6:28 am
|
|