| |
 |
|
|
Science Forum Index » Environment Forum » Audition the auditors #3
Page 1 of 1
|
| Author |
Message |
| Josh Halpern |
 Posted: Mon Dec 15, 2003 11:53 pm |
|
|
|
Guest
|
Since David Ball has taken care of #2 let us move onto #3 Abbreviations
as in #1
***************************************************
3. Where are the calculations of principal components for series
in the range #73-92 that would show that these have been collated into
the correct year? Do you have any working papers that show these, and if
so, would you make them FTP or otherwise publicly available?
********************************************
We will see that the same problem exists in 72 so let's start there.
72-80 are the principal components from Stahle's tree ring data for
the US southwest and mexico. The data are found in
.../pub/MBH98/TREE/STAHLE/SWM. There are four subdirectories called
/BACKTO_1700, /BACKTO_1600, /BACKTO_1500, /BACKTO_1450, and
/BACKTO_1400. Each of these subdirectories contains a complete set the
principal components, eigenvalues and eofs.
Now, to follow the argument you have to download pcproxy from C2300,
turn it into a spreadsheet, and insert a column for years starting with
1400 for the first year. You should also label the columns serially so
you can find column 72, etc.
Go to your spreadsheet row 1 (yr=1400) column 72 (pc1 for SWM). Note
that for columns 72-80, ONLY column 72 has numerical entries. If you
page down, you will find that the first entry in column 73 is at year
1500. This means, as we shall discover, that for the periods 1400-1450,
and 1450-1500 only the first principal component describing /STAHLE/SWM
was deemed significant. Others can comment on whether that is
reasonable or not.
Open ../BACKTO_1400/pc01.out. If you compare this with pcproxy
column 72, you will see that the numbers agree (I only spot checked
this) down to 1449. Now open /BACKTO_1450/pc01.out and compare with
the spreadsheet for 1450-1499. You will see that the spreadsheet
numbers are offset one year from the output file numbers, ie the value
for 1499 in the output file is the value for 1498 in the spreadsheet.
Moving on to /BACKTO_1500/pc01.out we find that the output file values
for 1500-1599 correspond to the pcproxy values for column 72 for
1499-1598, ie, the one year offset has been maintained. Now open
/BACKTO_1500/pc02. Note that the values correspond to the values in
column 73 with the offset.
This takes us to 1600. OK, same drill, go to the line corresponding to
1600 on the spreadsheet. There are now four principal components listed
open /BACKTO_1600/ pc01.out, pc02.out, pc03.out and pc04.out. compare
them respectively with columns 72, 73, 74 and 75. The match, with the
single year offset we have seen before. (I spot checked this but you
guys ain't paying me by the hour)
On to 1700. Same drill. We now have all nine pcs listed, we go to
/BACKTO_1600/ pc01.out...pc09.out. We compare with columns 72...80 and
we find the match (after taking the offset into account) from 1700 to 1979.
Where the 0.02303 in 1980 comes from, who knows.
Now, permit me a speculation. I believe that pcproxy was prepared from
a linux (or unix) script. The errors are the kind of thing that could
easily sneak in by mis-setting a loop, especially in the context of my
advisor gave me this pain in the ass thing to do. The "backto1820.txt
file would have been used to name the files to be included, and the
whole thing looks like a pointer or counter error of one. It would have
been much more difficult to make these mistakes if you were using a
spreadsheet or a word processor application.
Let me repeat the question
3. Where are the calculations of principal components for series
in the range #73-92 that would show that these have been collated into
the correct year?
********************
Answer ../pub/MBH98/TREE/STAHLE/SWM/BACKTO_XXXX
********************
Do you have any working papers that show these,
**********************
See above
**********************
and if so, would you make them FTP or otherwise publicly available?
***********************
same (given Paul Farrar's recollection these were most probably
available before pcproxy.txt was prepared and sent)
***********************
The other columns refer to the Vaganov Soviet tree ring records (81-83)
and the ITRDB north american tree ring records (84-92). Follow the same
algorithm. I have spot checked here also and it appears the same same.
The outcome of this, is between pcproxy and the pcout files on MBH98
everyone should now have a good picture of which principal component
series and which data series were used in each section of the
reconstruction.
josh halpern |
|
|
| Back to top |
|
| David Ball |
| Posted: Tue Dec 16, 2003 8:49 am |
|
|
|
Guest
|
On Tue, 16 Dec 2003 04:53:29 GMT, Josh Halpern
<j.halpern@incoming.verizon.net> wrote:
Quote: Since David Ball has taken care of #2 let us move onto #3 Abbreviations
as in #1
***************************************************
3. Where are the calculations of principal components for series
in the range #73-92 that would show that these have been collated into
the correct year? Do you have any working papers that show these, and if
so, would you make them FTP or otherwise publicly available?
********************************************
We will see that the same problem exists in 72 so let's start there.
72-80 are the principal components from Stahle's tree ring data for
the US southwest and mexico. The data are found in
../pub/MBH98/TREE/STAHLE/SWM. There are four subdirectories called
/BACKTO_1700, /BACKTO_1600, /BACKTO_1500, /BACKTO_1450, and
/BACKTO_1400. Each of these subdirectories contains a complete set the
principal components, eigenvalues and eofs.
Now, to follow the argument you have to download pcproxy from C2300,
turn it into a spreadsheet, and insert a column for years starting with
1400 for the first year. You should also label the columns serially so
you can find column 72, etc.
Go to your spreadsheet row 1 (yr=1400) column 72 (pc1 for SWM). Note
that for columns 72-80, ONLY column 72 has numerical entries. If you
page down, you will find that the first entry in column 73 is at year
1500. This means, as we shall discover, that for the periods 1400-1450,
and 1450-1500 only the first principal component describing /STAHLE/SWM
was deemed significant. Others can comment on whether that is
reasonable or not.
I think you're on the right track, Josh. My number crunching
from this period retains only a single PC, although the numbers don't
match with Mann's exactly. I think that stems from minor differences
in the way the input data are processed. I'm working from memory here,
but I think only 6 or 7 of the input series go back this far.
Quote:
Open ../BACKTO_1400/pc01.out. If you compare this with pcproxy
column 72, you will see that the numbers agree (I only spot checked
this) down to 1449. Now open /BACKTO_1450/pc01.out and compare with
the spreadsheet for 1450-1499. You will see that the spreadsheet
numbers are offset one year from the output file numbers, ie the value
for 1499 in the output file is the value for 1498 in the spreadsheet.
Moving on to /BACKTO_1500/pc01.out we find that the output file values
for 1500-1599 correspond to the pcproxy values for column 72 for
1499-1598, ie, the one year offset has been maintained. Now open
/BACKTO_1500/pc02. Note that the values correspond to the values in
column 73 with the offset.
This takes us to 1600. OK, same drill, go to the line corresponding to
1600 on the spreadsheet. There are now four principal components listed
open /BACKTO_1600/ pc01.out, pc02.out, pc03.out and pc04.out. compare
them respectively with columns 72, 73, 74 and 75. The match, with the
single year offset we have seen before. (I spot checked this but you
guys ain't paying me by the hour)
On to 1700. Same drill. We now have all nine pcs listed, we go to
/BACKTO_1600/ pc01.out...pc09.out. We compare with columns 72...80 and
we find the match (after taking the offset into account) from 1700 to 1979.
Where the 0.02303 in 1980 comes from, who knows.
Which is pretty much what I found in answering M&M's question
2.
Quote:
Now, permit me a speculation. I believe that pcproxy was prepared from
a linux (or unix) script. The errors are the kind of thing that could
easily sneak in by mis-setting a loop, especially in the context of my
advisor gave me this pain in the ass thing to do. The "backto1820.txt
file would have been used to name the files to be included, and the
whole thing looks like a pointer or counter error of one. It would have
been much more difficult to make these mistakes if you were using a
spreadsheet or a word processor application.
Interesting. It doesn't answer where the odd 1980 but it would
deal with the increment problem. A similar problem would account for
the problems in the other series M&M inquire about in question 2.
Quote:
Let me repeat the question
3. Where are the calculations of principal components for series
in the range #73-92 that would show that these have been collated into
the correct year?
********************
Answer ../pub/MBH98/TREE/STAHLE/SWM/BACKTO_XXXX
********************
Do you have any working papers that show these,
**********************
See above
**********************
and if so, would you make them FTP or otherwise publicly available?
***********************
same (given Paul Farrar's recollection these were most probably
available before pcproxy.txt was prepared and sent)
***********************
The other columns refer to the Vaganov Soviet tree ring records (81-83)
and the ITRDB north american tree ring records (84-92). Follow the same
algorithm. I have spot checked here also and it appears the same same.
The outcome of this, is between pcproxy and the pcout files on MBH98
everyone should now have a good picture of which principal component
series and which data series were used in each section of the
reconstruction.
In addition, the software used to produce the PC's is also
available on the ftp site, so the author's could have used it to redo
the calculations. As I said in a previous note, the fortran code for
processing the SWM and Vaganov series is one and the same program, but
the software used to process the NOAM data from the ITRDB is
different. That could be due to differences in the format of the input
data, but I will check to make sure that the same analysis is done. |
|
|
| Back to top |
|
| Nigel Persaud |
| Posted: Sat Dec 20, 2003 6:26 pm |
|
|
|
Guest
|
Josh,
if you go to http://www.uoguelph.ca/~rmckitri/research/MM-nov12-part1.pdf
and look at page 10, you see that M&M carried out a virtually
identical analysis of the relationship of pcproxy.txt to the series in
the BACKTO directories and actually a little more.
So one can see the relationship between pcproxy.txt to the BACKTO
files. We know that the collation in pcproxy.txt and pcproxy.mat was
wrong. Mann has stated that MBH98 did not collate incorrectly. I think
that this is possible, but the only collation provided to data is
incorrect. Mann has not shown either a correct collation or a copy of
the computer program which uses the raw files correctly.
Mann has included computer programs for PCA calculations at his FTP
site - why wouldn't he simply post up evidence that he did it
correctly in MBH98. It would take about one second to do.
Nigel
Josh Halpern <j.halpern@incoming.verizon.net> wrote in message news:<ddwDb.2680$G9.533@nwrddc01.gnilink.net>...
Quote: Since David Ball has taken care of #2 let us move onto #3 Abbreviations
as in #1
***************************************************
3. Where are the calculations of principal components for series
in the range #73-92 that would show that these have been collated into
the correct year? Do you have any working papers that show these, and if
so, would you make them FTP or otherwise publicly available?
********************************************
We will see that the same problem exists in 72 so let's start there.
72-80 are the principal components from Stahle's tree ring data for
the US southwest and mexico. The data are found in
../pub/MBH98/TREE/STAHLE/SWM. There are four subdirectories called
/BACKTO_1700, /BACKTO_1600, /BACKTO_1500, /BACKTO_1450, and
/BACKTO_1400. Each of these subdirectories contains a complete set the
principal components, eigenvalues and eofs.
Now, to follow the argument you have to download pcproxy from C2300,
turn it into a spreadsheet, and insert a column for years starting with
1400 for the first year. You should also label the columns serially so
you can find column 72, etc.
Go to your spreadsheet row 1 (yr=1400) column 72 (pc1 for SWM). Note
that for columns 72-80, ONLY column 72 has numerical entries. If you
page down, you will find that the first entry in column 73 is at year
1500. This means, as we shall discover, that for the periods 1400-1450,
and 1450-1500 only the first principal component describing /STAHLE/SWM
was deemed significant. Others can comment on whether that is
reasonable or not.
Open ../BACKTO_1400/pc01.out. If you compare this with pcproxy
column 72, you will see that the numbers agree (I only spot checked
this) down to 1449. Now open /BACKTO_1450/pc01.out and compare with
the spreadsheet for 1450-1499. You will see that the spreadsheet
numbers are offset one year from the output file numbers, ie the value
for 1499 in the output file is the value for 1498 in the spreadsheet.
Moving on to /BACKTO_1500/pc01.out we find that the output file values
for 1500-1599 correspond to the pcproxy values for column 72 for
1499-1598, ie, the one year offset has been maintained. Now open
/BACKTO_1500/pc02. Note that the values correspond to the values in
column 73 with the offset.
This takes us to 1600. OK, same drill, go to the line corresponding to
1600 on the spreadsheet. There are now four principal components listed
open /BACKTO_1600/ pc01.out, pc02.out, pc03.out and pc04.out. compare
them respectively with columns 72, 73, 74 and 75. The match, with the
single year offset we have seen before. (I spot checked this but you
guys ain't paying me by the hour)
On to 1700. Same drill. We now have all nine pcs listed, we go to
/BACKTO_1600/ pc01.out...pc09.out. We compare with columns 72...80 and
we find the match (after taking the offset into account) from 1700 to 1979.
Where the 0.02303 in 1980 comes from, who knows.
Now, permit me a speculation. I believe that pcproxy was prepared from
a linux (or unix) script. The errors are the kind of thing that could
easily sneak in by mis-setting a loop, especially in the context of my
advisor gave me this pain in the ass thing to do. The "backto1820.txt
file would have been used to name the files to be included, and the
whole thing looks like a pointer or counter error of one. It would have
been much more difficult to make these mistakes if you were using a
spreadsheet or a word processor application.
Let me repeat the question
3. Where are the calculations of principal components for series
in the range #73-92 that would show that these have been collated into
the correct year?
********************
Answer ../pub/MBH98/TREE/STAHLE/SWM/BACKTO_XXXX
********************
Do you have any working papers that show these,
**********************
See above
**********************
and if so, would you make them FTP or otherwise publicly available?
***********************
same (given Paul Farrar's recollection these were most probably
available before pcproxy.txt was prepared and sent)
***********************
The other columns refer to the Vaganov Soviet tree ring records (81-83)
and the ITRDB north american tree ring records (84-92). Follow the same
algorithm. I have spot checked here also and it appears the same same.
The outcome of this, is between pcproxy and the pcout files on MBH98
everyone should now have a good picture of which principal component
series and which data series were used in each section of the
reconstruction.
josh halpern |
|
|
| Back to top |
|
| David Ball |
| Posted: Sat Dec 20, 2003 10:06 pm |
|
|
|
Guest
|
On 20 Dec 2003 15:26:44 -0800, persaud23@yahoo.com (Nigel Persaud)
wrote:
Quote: Josh,
if you go to http://www.uoguelph.ca/~rmckitri/research/MM-nov12-part1.pdf
and look at page 10, you see that M&M carried out a virtually
identical analysis of the relationship of pcproxy.txt to the series in
the BACKTO directories and actually a little more.
So one can see the relationship between pcproxy.txt to the BACKTO
files. We know that the collation in pcproxy.txt and pcproxy.mat was
wrong. Mann has stated that MBH98 did not collate incorrectly. I think
that this is possible, but the only collation provided to data is
incorrect. Mann has not shown either a correct collation or a copy of
the computer program which uses the raw files correctly.
Mann has included computer programs for PCA calculations at his FTP
site - why wouldn't he simply post up evidence that he did it
correctly in MBH98. It would take about one second to do.
I'm not entirely sure I understand what it is you want Mann to
do. The data are available as are the programs to analyze it. |
|
|
| Back to top |
|
| Josh Halpern |
| Posted: Sat Dec 20, 2003 11:26 pm |
|
|
|
Guest
|
David Ball wrote:
Quote: On 20 Dec 2003 15:26:44 -0800, persaud23@yahoo.com (Nigel Persaud)
wrote:
Josh,
if you go to http://www.uoguelph.ca/~rmckitri/research/MM-nov12-part1.pdf
and look at page 10, you see that M&M carried out a virtually
identical analysis of the relationship of pcproxy.txt to the series in
the BACKTO directories and actually a little more.
So one can see the relationship between pcproxy.txt to the BACKTO
files. We know that the collation in pcproxy.txt and pcproxy.mat
since pcproxy.mat is M&Ms file, I would not refer to it. This has
already confused the issue in Quark Soup with claims that MBH98 was done
using matlab. Probably other places.
Quote: was wrong. Mann has stated that MBH98 did not collate incorrectly. I think
that this is possible, but the only collation provided to data is
incorrect. Mann has not shown either a correct collation or a copy of
the computer program which uses the raw files correctly.
Mann has included computer programs for PCA calculations at his FTP
site - why wouldn't he simply post up evidence that he did it
correctly in MBH98. It would take about one second to do.
I'm not entirely sure I understand what it is you want Mann to
do. The data are available as are the programs to analyze it.
Mostly I agree. Since everything is post hoc, I don't see how posting
anything else would change anything. There was a long discussion here
about how ../pub/MBH98 was or was not created yesterday, today or
tomorrow. I think David's approach is about the only one that is useful.
josh halpern |
|
|
| Back to top |
|
| Steve Schulin |
| Posted: Sun Dec 21, 2003 12:30 am |
|
|
|
Guest
|
In article <yh9Fb.19864$jG4.11034@nwrddc02.gnilink.net>,
Josh Halpern <j.halpern@incoming.verizon.net> wrote:
Quote: David Ball wrote:
On 20 Dec 2003 15:26:44 -0800, persaud23@yahoo.com (Nigel Persaud)
wrote:
Josh,
if you go to http://www.uoguelph.ca/~rmckitri/research/MM-nov12-part1.pdf
and look at page 10, you see that M&M carried out a virtually
identical analysis of the relationship of pcproxy.txt to the series in
the BACKTO directories and actually a little more.
So one can see the relationship between pcproxy.txt to the BACKTO
files. We know that the collation in pcproxy.txt and pcproxy.mat
since pcproxy.mat is M&Ms file, ...
You are mistaken about this. pcproxy.mat was obtained from the newly
disclosed Mann et al ftp directory -- within a few hours of its public
disclosure on Oct 29, 2003. It (and pcproxy.txt too) was deleted from
that directory sometime before Nov 8.
Quote: ... I would not refer to it. This has
already confused the issue in Quark Soup with claims that MBH98 was done
using matlab. Probably other places.
was wrong. Mann has stated that MBH98 did not collate incorrectly. I think
that this is possible, but the only collation provided to data is
incorrect. Mann has not shown either a correct collation or a copy of
the computer program which uses the raw files correctly.
Mann has included computer programs for PCA calculations at his FTP
site - why wouldn't he simply post up evidence that he did it
correctly in MBH98. It would take about one second to do.
I'm not entirely sure I understand what it is you want Mann to
do. The data are available as are the programs to analyze it.
Mostly I agree. Since everything is post hoc, I don't see how posting
anything else would change anything. There was a long discussion here
about how ../pub/MBH98 was or was not created yesterday, today or
tomorrow. I think David's approach is about the only one that is useful.
josh halpern
|
|
|
| Back to top |
|
| Josh Halpern |
| Posted: Sun Dec 21, 2003 12:50 am |
|
|
|
Guest
|
Josh Halpern wrote:
Quote: David Ball wrote:
On 20 Dec 2003 15:26:44 -0800, persaud23@yahoo.com (Nigel Persaud)
wrote:
Josh,
if you go to
http://www.uoguelph.ca/~rmckitri/research/MM-nov12-part1.pdf
and look at page 10, you see that M&M carried out a virtually
identical analysis of the relationship of pcproxy.txt to the series in
the BACKTO directories and actually a little more.
So one can see the relationship between pcproxy.txt to the BACKTO
files. We know that the collation in pcproxy.txt and pcproxy.mat
since pcproxy.mat is M&Ms file, I would not refer to it. This has
already confused the issue in Quark Soup with claims that MBH98 was
done using matlab. Probably other places.
Ok according to M&M it was, but we also have the FORTRAN programs.
Strange. Could have been done two ways I suppose. However, if you poke
around in ftp://eclogite.geo.umass.edu/pub/mann/, or the holocene
computer, what you find is in FORTRAN which leads me to believe that
this whole mathlab thing is a red herring, maybe something Mann was
poking around with
The main point of this is that if you count PC files at the break points
described by MBH98 (1400, 1450, 1600, 1700 and 1760 you get 76 pc files
and with the 81 data files you get 157 total files.
Another piece of information which no one seems to have noticed is
ftp://eclogite.geo.umass.edu/pub/mann/ONLINE-PREPRINTS/MultiProxy/stats-supp.html
which has a bunch of statistics on various trial runs.
josh halpern |
|
|
| Back to top |
|
| Nigel Persaud |
| Posted: Sun Dec 21, 2003 2:00 am |
|
|
|
Guest
|
David Ball <wraith7@mb.sympatico.ca> wrote in message news:<ig3auvg9o370bs4pfv0pd3jdvb09cldepc@4ax.com>...
Quote: On 20 Dec 2003 15:26:44 -0800, persaud23@yahoo.com (Nigel Persaud)
wrote:
Josh,
if you go to http://www.uoguelph.ca/~rmckitri/research/MM-nov12-part1.pdf
and look at page 10, you see that M&M carried out a virtually
identical analysis of the relationship of pcproxy.txt to the series in
the BACKTO directories and actually a little more.
So one can see the relationship between pcproxy.txt to the BACKTO
files. We know that the collation in pcproxy.txt and pcproxy.mat was
wrong. Mann has stated that MBH98 did not collate incorrectly. I think
that this is possible, but the only collation provided to data is
incorrect. Mann has not shown either a correct collation or a copy of
the computer program which uses the raw files correctly.
Mann has included computer programs for PCA calculations at his FTP
site - why wouldn't he simply post up evidence that he did it
correctly in MBH98. It would take about one second to do.
I'm not entirely sure I understand what it is you want Mann to
do. The data are available as are the programs to analyze it.
The programs to do PCA on tree rings are there, but no programs to
carry out temperature reconstruction (presumably reading in the tree
ring PCA correctly). I would assume that he did this in Fortran and
used .inf files to pick up series from the BACKTO directories.
There's nothing like that. |
|
|
| Back to top |
|
| Nigel Persaud |
| Posted: Sun Dec 21, 2003 9:08 am |
|
|
|
Guest
|
Quote: The main point of this is that if you count PC files at the break points
described by MBH98 (1400, 1450, 1600, 1700 and 1760 you get 76 pc files
and with the 81 data files you get 157 total files.
This still won't balance with the required number of proxies at each
of the periods quoted in the text of Nature or in the stat-supp info
below (which is also at Nature).
|
|
|
| Back to top |
|
| David Ball |
| Posted: Sun Dec 21, 2003 10:00 am |
|
|
|
Guest
|
On 20 Dec 2003 23:00:42 -0800, persaud23@yahoo.com (Nigel Persaud)
wrote:
Quote: David Ball <wraith7@mb.sympatico.ca> wrote in message news:<ig3auvg9o370bs4pfv0pd3jdvb09cldepc@4ax.com>...
On 20 Dec 2003 15:26:44 -0800, persaud23@yahoo.com (Nigel Persaud)
wrote:
Josh,
if you go to http://www.uoguelph.ca/~rmckitri/research/MM-nov12-part1.pdf
and look at page 10, you see that M&M carried out a virtually
identical analysis of the relationship of pcproxy.txt to the series in
the BACKTO directories and actually a little more.
So one can see the relationship between pcproxy.txt to the BACKTO
files. We know that the collation in pcproxy.txt and pcproxy.mat was
wrong. Mann has stated that MBH98 did not collate incorrectly. I think
that this is possible, but the only collation provided to data is
incorrect. Mann has not shown either a correct collation or a copy of
the computer program which uses the raw files correctly.
Mann has included computer programs for PCA calculations at his FTP
site - why wouldn't he simply post up evidence that he did it
correctly in MBH98. It would take about one second to do.
I'm not entirely sure I understand what it is you want Mann to
do. The data are available as are the programs to analyze it.
The programs to do PCA on tree rings are there, but no programs to
carry out temperature reconstruction (presumably reading in the tree
ring PCA correctly). I would assume that he did this in Fortran and
used .inf files to pick up series from the BACKTO directories.
There's nothing like that.
Now I follow you. I'm not sure why. It's going to make my
recontructing the temperature series a little harder, but I don't
think it will be too hard. That will also make things truly
independent. |
|
|
| Back to top |
|
| |
|
Page 1 of 1
All times are GMT - 5 Hours
The time now is Fri Jan 09, 2009 7:42 am
|
|