Dear all,

let there be conducted some experiment with a lot of independent
variables and therefore a lot of simple association tests
(assume n=550,000) of any of these factors with only one
dependent variable be carried out (case control study
in epidemiology with disease status as outcome);

let 0.1% (n_sig1=550) of them get significant at the
0.001 level (p<0.001), according to null hypothesis.

Now these significantly tested factors are carried on
forward to a 2nd stage of the study in an independent
sample. These factors are finally tested at the 5%-level.

Now I'd like to calculate the expected number of significant
tests (n_sig2) under null) and a critical value (upper 95%
confidence bound) in order to get some kind of "global p-value"
(Before availability of final test results of stage 2.
At this time we have only results of stage 1.)

The question now: How can this be done?
Should I:
a) just use a binomial test like:
'binom.test(n_sig2,n_sig1,0.05,"greater")' in R
disregarding the total number of tests conducted
in the 1st stage, as we are still assuming the null?

b) do a kind of 2-dimensional Chi² (or Fisher's test)
using a table like the following:

| significant | not significant
| at stage one | at stage one
----------------+--------------+-------------------
significant | a | b
at stage two | (=n_sig2) |
----------------+--------------+-------------------
not significant| c | d
at stage two | |
----------------+--------------+-------------------
total | (=n_sig1) |

#again R-code with some sample:
#p1<0.001; p2<0.05
a<-37
b<-27500-a
c<-550-a
d<-550000-a-b-c
t<-matrix(c(a,b,c,d),2,byrow=TRUE)
t
fisher.test(t,alternative="greater")

problem here: b and d are actually 0 as they are
not done in 2nd stage. One could only use expected
values from 1st and 2nd stage; but that's exactly
what I am to do: compare some hypothetical observed
counts against expected values under null.
And it takes into account the total number of
tests from the 1st stage, in contrast to the
upper one that neglects that these factors have
been tested with a significant p-value twice
(in *two* *independent* samples).
So I would prefer that.

c) go to something completely different?
what then exactly.

And what would I do finally (when having all test results)
to correct the p-values for multiple testing
over these _two stages_?

Maybe someone could even point me to some
reference in literature?

Thanks a lot
best wishes
Josef Frank