| |
 |
|
|
Science Forum Index » Statistics - Education Forum » Experimental Research In Education: The Most Exciting Talk
Page 1 of 1
|
| Author |
Message |
| Donald Macnaughton |
 Posted: Sun Feb 11, 2007 10:26 pm |
|
|
|
Guest
|
Some Basic Ideas of Science
This post replies to comments by Mike Palij. But first, to sup-
port the discussion, it is helpful to propose definitions of some
basic ideas of science, as follows:
EMPIRICAL RESEARCH is any activity in which measurements
(observations) are gathered from some area of experience
and then reasonable conclusions are drawn from the meas-
urements.
Measurements in empirical research are usefully viewed as
reflecting the values of VARIABLES, which reflect the
measured values of properties of entities. (Entities may
be people, other living things, physical objects, or any
other type of object or thing.)
An EMPIRICAL RESEARCH PROJECT (or logical sub-unit of an
empirical research project) is usually usefully viewed as
studying the relationship between a single response vari-
able and one or more predictor variables in the entities
in the population of interest. This study is performed
by analyzing the measured values of the variables ob-
tained from a sample of entities from the population.
An EXPERIMENT is an empirical research project in which
at least one of the predictor variables is "manipulated"
(i.e., caused to take certain values in the entities in
the sample) by the researcher.
An OBSERVATIONAL RESEARCH PROJECT is an empirical re-
search project in which none of the predictor variables
is manipulated by the researcher, and the values of the
predictor variables are simply observed.
I discuss the ideas in more detail in a paper (2002).
The description of an empirical research project is qualified
with the word "usually". This implies that some empirical re-
search projects (or sub-units) don't satisfy the description --
that is, they can't be reasonably viewed as studying the rela-
tionship between a single response variable and one or more pre-
dictor variables. My experience suggests that research projects
that can't be reasonably viewed as satisfying the description ap-
pear in less than three percent of published empirical research
reports (in science, technology, and business). I discuss some
of these research projects in the paper (2002, Appendix I.2).
For economy of words, this small group of research projects is
mostly ignored in the discussion below.
Novelty of Experiments in Education
Quoting my August 25, 2005 post, Mike Palij wrote (on August 26
in the EdStat e-mail list)
< snip >
Quote: Experimental studies in education are not a new idea.
I agree. I suspect that the idea of performing proper experi-
ments in education can be traced back to the 1920's or 1930's
when Fisher, Neyman, Pearson, and other statisticians first ex-
plained and debated the idea of a proper experiment. However,
proper experiments in education haven't often been performed.
What is a "proper" experiment? Here is a reasonable definition:
An experiment (or randomized trial) is a PROPER EXPERI-
MENT if it has been performed according to widely ac-
cepted principles of scientific practice, experimental
design, and data analysis, as described by Bailar and
Mosteller (1992), Box, Hunter, and Hunter (2005), Fleiss
(1986), Kirk (1995), Winer, Brown, and Michels (1991),
and many others.
I suspect that proper experiments haven't often been performed in
education for two reasons: (a) proper experiments in education
are difficult, and (b) perhaps simply due to tradition, research-
ers in education have lacked a strong connection to the ideas of
experimental research.
Problematic Nature of Experiments in Education
< snip >
Quote: There are a number of reasons why experiments are problematic
in educational settings
I fully agree. Some key problems in performing a good education
experiment to compare teaching approaches are
1. What is the RESPONSE VARIABLE (or variables) that we will use
to compare the teaching approaches? Will it be marks, or at-
titudes, or aptitudes, or some other measure of the students
(or the classes of students) under study?
2. What are the PREDICTOR VARIABLES that we will measure in the
research? One predictor variable will reflect the different
teaching approaches under study. What other predictor vari-
ables (e.g., students' age or gender) should we measure to as-
sist our understanding? Is it reasonable to use students'
race as a predictor variable?
3. How can we design the experiment in a way that will ELIMINATE
teacher effects and other REASONABLE ALTERNATIVE EXPLANATIONS
of any significant differences we find in the values of the
response variable between the teaching approaches?
4. How can we design the experiment so that it is as LIKELY as
possible that we will FIND STRONG EVIDENCE of the relationship
we are looking for between the response variable and the pre-
dictor variable(s) (assuming that the relationship actually
exists)?
5. How can we RECRUIT TEACHERS to participate in the experiment?
6. How can we OBTAIN FUNDING to pay for the experiment?
7. After we have performed the experiment, how can we ANALYZE the
RESULTS and DRAW SCIENTIFICALLY VALID CONCLUSIONS?
8. While properly addressing the preceding seven problems, how
can we MINIMIZE the COSTS of the experiment?
Appendix A expands the eight problems and discusses some solu-
tions.
Necessity of Experiments in Education
< snip >
Quote: As for the necessity of experimental research to provide the
basis for either (a) science or (b) valid conclusions, the as-
tronomers say "Hi! We've been engaging in science for hundreds
of years without having performed a single experiment!".
Mike suggests that if astronomers don't perform experiments, then
education researchers also don't need to perform experiments. I
think that this is a thought-provoking point, but an invalid ar-
gument. The argument is invalid because astronomers CAN'T per-
form experiments because they can't manipulate distant astronomi-
cal events. If they could manipulate these events (at a reason-
able cost), it seems doubtless that they would. That is, they
would perform proper experiments just like scientists in disci-
plines that regularly perform experiments, such as in most
branches of physics, chemistry, engineering, medicine, biology,
and psychology.
The fact that astronomers can't perform astronomical experiments
makes astronomy an "observational" discipline. Other observa-
tional disciplines that generally can't perform experiments (due
to the remoteness or untouchability of the phenomena they study)
include anthropology, archaeology, economics, epidemiology, geol-
ogy, paleontology, and some areas of sociology. Such observa-
tional disciplines base their inferences on careful observational
empirical research, often studying relationships between vari-
ables. (In historical disciplines, observational research is
sometimes [due to the paucity of data] reduced to careful consid-
eration of physical or anecdotal information about entities,
properties of entities, or variables, without focusing on the
concept of 'relationship between variables'.)
Proper observational research often enables us to reliably
PREDICT the values of the response variable (in new situations),
which is an important benefit. However, in education research we
would generally like to learn how to reliably CONTROL the values
of the response variable (in new situations). That is, we would
like to learn how to structure an education program so that it
provides students with the BEST education. Unfortunately, obser-
vational research is almost always equivocal about control --
subject to multiple competing reasonable explanations.
For example, suppose we are presented with the results of an ob-
servational research project in education that suggests that a
certain teaching approach A is better (in the sense of exhibiting
significantly better average values in students of the chosen re-
sponse variable) than another teaching approach B. In this case
(due to the nature of observational research) it is almost always
possible to find a reasonable alternative explanation of the re-
search finding, and this explanation implies that approach A may
NOT be better than approach B. But if we find such an explana-
tion, this implies that the research is equivocal. This means
that the research is of substantially less value because it can't
reliably help us to decide which teaching approach is better.
(A reasonable alternative explanation in observational education
research is often in terms of "confounding" of the teaching ap-
proaches under study in the research with some other aspect
[i.e., variable] of the research situation. Then it is generally
possible that this other variable can fully account for the dif-
ference between the average values of the response variable under
the different teaching approaches. For example, an observational
research project might confound two teaching approaches with two
different schools -- school 1 uses teaching approach A, and
school 2 uses teaching approach B. In this case if we find a
significant difference in the average values of the response
variable between the two approaches, it is possible that the
teaching approaches have no differential effect on the values of
the response variable. That is, [unless the School variable is
appropriately (and expensively) taken account of in the design of
the research] it is possible that a certain difference between
the SCHOOLS caused the observed significant differences in the
average values of the response variable [in students or classes]
between the teaching approaches.)
In contrast, suppose that a proper EXPERIMENT provides good evi-
dence that teaching approach A is better than teaching approach
B. In this case the finding is unequivocal. This is because
proper experiments are explicitly designed to eliminate confound-
ing and other reasonable alternative explanations. Thus in this
case we can safely (tentatively) conclude that approach A will be
better than approach B in new situations (if the relevant condi-
tions are sufficiently similar to those of the experiment). Thus
proper experiments are preferred to observational research pro-
jects in education research.
(The equivocation in observational research relates to drawing
conclusions about causation. That is, we are interested in
whether teaching approach A CAUSES students to do better than
teaching approach B. Evidence about relationships between vari-
ables obtained in observational research is generally equivocal
about causation. In contrast, evidence about relationships ob-
tained in proper experimental research is unequivocal about cau-
sation.)
Changing Attitudes Toward Experiments in Education
Mike noted that members of the respected American Educational Re-
search Association (AERA) have carefully considered the issue of
experimental research in education. Mike's point is directly re-
flected in the opening sentence of the official description of
the theme of the 2006 AERA annual meeting:
Current social and political pressures on education re-
search suggest that research must meet the demands of
evidence-based and scientifically based inquiry (Ladson-
Billings and Tate 2006).
The idea of "current" pressures reflects the fact that the pres-
sures on education research are new, having arrived over the last
decade or so. The sentence implies that education researchers
are moving toward "evidence-based and scientifically based" re-
search. This suggests that education researchers should gener-
ally perform proper experiments (because observational research
results are generally equivocal).
(Having acknowledged the importance of proper research, the de-
scription of the theme of the 2006 AERA meeting turns to the
theme itself, which pertains to education research in the public
interest, education research that will "increase the common-
wealth". The discussion of the theme is available at
http://www.aera.net/annualmeeting/?id=694 )
Opportunities for Experiments in Education
The preceding discussion suggests that (a) the area of experimen-
tal studies in education is only now beginning to open up and (b)
this area will become the mainstream of education research as
granting agencies and journal editors reinforce the point that
proper experiments are preferred to observational research. Be-
cause the area is opening up, it has many opportunities for
thoughtful researchers.
To perform a proper education experiment a researcher must be fa-
miliar with the principles of experimental design, power analy-
sis, and (often) repeated measurements analysis of variance.
Some education researchers are less familiar with these topics.
They may find it helpful to follow the path of many medical re-
searchers who collaborate with a statistician with experience in
the topics. To ensure that the research design is efficient, I
recommend that this collaboration begin early in the design phase
of the research.
Opportunities exist for statisticians to present courses to edu-
cation researchers about the statistical and scientific aspects
of education research. I propose topics for such a course in ap-
pendix B.
I believe that the movement toward experimentally based education
research will yield a body of reliable research results that will
substantially improve education.
Don Macnaughton
Donald B. Macnaughton
donmac@matstat.com
Appendices and References
The appendices and references for this post are available at
http://www.matstat.com/teach/p0048.htm#apps
Appendix A: Eight Problems in Experiments in Education
http://www.matstat.com/teach/p0048.htm#AppA
Appendix B: Courses About Experiments For Education Researchers
http://www.matstat.com/teach/p0048.htm#AppB
Appendix C: Can Human Performance or Behavior Be Predicted from
a Person’s Race?
http://www.matstat.com/teach/p0048.htm#AppC
Appendix D: Specifying a Repeated Measurements Analysis of
Variance
http://www.matstat.com/teach/p0048.htm#AppD |
|
|
| Back to top |
|
| |
|
Page 1 of 1
All times are GMT - 5 Hours
The time now is Fri Aug 29, 2008 10:37 pm
|
|