Tag Archives: economics

When It Makes Sense to Experiment on Students — or “The Zone of RCTs”

The Nobel Prize was recently awarded to economists Abhijit Banerjee and Esther Duflo of MIT and Michael Kremer of Harvard. NPR described the researchers’ work as “applying the scientific method to an enterprise that, until recently, was largely based on gut instincts.”

An enterprise based on gut instincts? That sounds like education! The Institute of Education Sciences (IES), the arm of the U.S. Department of Education charged with providing reliable information about the effectiveness of education programs, is not even old enough to vote. It was not until 2015 that evidence was given much attention in federal education law, and state plans submitted under the Every Student Succeeds Act “mostly ignored research on what works,” according to Pemberton Research founder Mark Dynarski.

But attention to experimental evidence has been growing. The Nobel Prize press release specifically mentioned the use of Randomized Control Trial (RCT) experiments to inform social policy intended to alleviate poverty. RCTs are studies that randomly assign individuals to an intervention group or to a control group in order to measure the effects of the intervention (for a visual, see here). RCTs are considered the strongest form of evidence by IES and under the Every Student Succeeds Act. Examples of RCTs in the education field include some of the Nobel prize winners’ work in India, and IES’s What Works Clearinghouse, which catalogues the evidence base for education interventions. 

Questions on causality — that is, when we want to know if some policy or program causes some outcome — are best served by experiments, at least for narrowly defined research questions. Yet the idea of experimenting on students (especially low-income or low-achieving students) can make people feel queasy, and so it is worth asking in what circumstances it makes sense to conduct an experimental study.

An experiment might make sense if we believe a policy or practice has some positive impact on people, but we’re not sure about the size of the impact. Researchers should not experiment if they have reason to believe a policy or practice to be harmful, because the students in the “treatment” group would be harmed. Nor should researchers experiment if they are fairly confident that it is beneficial; in that case, students who were assigned to the “control” group would be harmed by being deprived of the treatment. See my simple graphic explanation below:

Inspired by a tweet from Vinay Prasad, Associate Professor of Medicine at Oregon Health and Science University

Continue reading