Author Archives: Cara Jackson

When It Makes Sense to Experiment on Students — or “The Zone of RCTs”

The Nobel Prize was recently awarded to economists Abhijit Banerjee and Esther Duflo of MIT and Michael Kremer of Harvard. NPR described the researchers’ work as “applying the scientific method to an enterprise that, until recently, was largely based on gut instincts.”

An enterprise based on gut instincts? That sounds like education! The Institute of Education Sciences (IES), the arm of the U.S. Department of Education charged with providing reliable information about the effectiveness of education programs, is not even old enough to vote. It was not until 2015 that evidence was given much attention in federal education law, and state plans submitted under the Every Student Succeeds Act “mostly ignored research on what works,” according to Pemberton Research founder Mark Dynarski.

But attention to experimental evidence has been growing. The Nobel Prize press release specifically mentioned the use of Randomized Control Trial (RCT) experiments to inform social policy intended to alleviate poverty. RCTs are studies that randomly assign individuals to an intervention group or to a control group in order to measure the effects of the intervention (for a visual, see here). RCTs are considered the strongest form of evidence by IES and under the Every Student Succeeds Act. Examples of RCTs in the education field include some of the Nobel prize winners’ work in India, and IES’s What Works Clearinghouse, which catalogues the evidence base for education interventions. 

Questions on causality — that is, when we want to know if some policy or program causes some outcome — are best served by experiments, at least for narrowly defined research questions. Yet the idea of experimenting on students (especially low-income or low-achieving students) can make people feel queasy, and so it is worth asking in what circumstances it makes sense to conduct an experimental study.

An experiment might make sense if we believe a policy or practice has some positive impact on people, but we’re not sure about the size of the impact. Researchers should not experiment if they have reason to believe a policy or practice to be harmful, because the students in the “treatment” group would be harmed. Nor should researchers experiment if they are fairly confident that it is beneficial; in that case, students who were assigned to the “control” group would be harmed by being deprived of the treatment. See my simple graphic explanation below:

Inspired by a tweet from Vinay Prasad, Associate Professor of Medicine at Oregon Health and Science University

Continue reading

Teacher Residencies Can Translate Into a More Diverse Workforce, But Who Will Bear the Expenses?

A growing number of studies have documented the benefits of teacher diversity, as my colleagues have previously discussed (see here and here). And research drawing on data from the Measures of Effective Teaching project found that all students preferred teachers of color. Yet despite the value of diversifying the teaching workforce, Black teachers remain underrepresented. They made up around 7 to 8 percent of all teachers between school years 1999-2000 and 2015-2016, while Black students accounted for between 16 to 17 percent of all students in the same time period. The proportion of Hispanic public elementary and secondary school teachers appeared to be increasing slightly, but not nearly as fast as the proportion of Hispanic students, as seen in the figure below.

Proportion of Hispanic students and teachers over time

As my colleague Katrina has noted, a number of barriers to diversifying the teacher workforce exist, including the considerable cost to become a teacher. Traditional preparation programs have high out-of-pocket and opportunity costs (i.e., limited income while enrolled in a program). As Ashley LiBetti and Justin Trinidad describe in their recent report, these costs limit the pool of teacher candidates:  

In the traditional model, candidates must invest more than $24,000 and 1,500 hours to become a teacher…This upfront financial and opportunity cost limits the pool of candidates to those who can afford the risk, effectively cutting out nontraditional candidates, low- and lower-middle-income candidates, and career-changers.

Given well-documented racial disparities in wealth, the high cost of becoming a teacher is likely to have a disproportionate impact on the career decisions of people of color. Alternate routes to teaching may be an attractive option for prospective teachers who are sensitive to costs. As seen below, teachers from alternate routes tend to be more racially diverse than teachers from traditional teacher preparation programs. 

Racial/ethnic diversity for tradition route and alternate route teachers

Yet alternate routes have proven controversial, even when evidence suggests that alternatively certified teachers are equally or more effective at increasing student achievement on standardized tests, relative to their counterparts. In Houston, for example, school district trustees recently voted to end the district’s contract with Teach For America, citing concerns about teacher turnover. 

Teacher residency programs have emerged as perhaps a more politically viable alternative certification route than “fast-track” programs, by emphasizing on-the-job training prior to becoming a teacher of record. Because residents are typically paid a stipend during their apprenticeship period, entering teaching through a residency tends to cost less than entering through a traditional route. However, residencies typically pay a stipend that is less than what a teacher of record would earn. As a result, the cost of entering the profession through a residency program is higher than the cost of entering through a fast-track alternative certification program.  Continue reading

Correlation is Not Causation and Other Boring but Important Cautions for Interpreting Education Research

Journalists, as a general rule, use accessible language. Researchers, as a general rule, do not. So journalists who write about academic research and scholarship, like the reporters at Chalkbeat who cover school spending studies, can help disseminate research to education leaders since they write more plainly.

But the danger is that it’s easy for research to get lost in translation. Researchers may use language that appears to imply some practice or policy causes an outcome. Journalists can be misled when terms like “effect size” are used to describe the strength of the association even though they are not always causal effects.

To help journalists make sense of research findings, the Education Writers Association recently put together several excellent resources for journalists exploring education research, including 12 questions to ask about studies. For journalists (as well as practitioners) reading studies that imply that some program or policy causes the outcomes described, I would add one important consideration (a variation on question 3 from this post): if a study compares two groups, how were people assigned to the groups? This question gets at the heart of what makes it possible to say whether a program or policy caused the outcomes examined, as opposed to simply being correlated with those outcomes.

Randomly assigning people creates a strong research design for examining whether a policy or program causes certain outcomes. Random assignment minimizes pre-existing differences among the groups, so that differences in the outcomes can be attributed to the treatment (program or policy) instead of different characteristics of the people in the groups. In the image below, random assignment results in having similar-looking treatment and control groups. Continue reading

Why Some Educators Are Skeptical of Engaging in Rigorous Research — And What Can Be Done About It

In my previous post, I talked about the importance of rigorous research and the need for researchers to engage directly with education stakeholders. Yet some educators remain skeptical about the value of partnering with researchers, even if the research is relevant and rigorous. Why might education agencies fail to see the value of conducting rigorous research in their own settings?

For one thing, letting a researcher into the nitty gritty of your outcomes or practices might reveal that something isn’t working. And since it’s rare that educators/practitioners and researchers are even in the same room, education agency staff may be concerned about how findings will be framed once publicized. If they don’t even know one another, how can we expect researchers and educators to overcome their lack of trust and work together effectively?

Furthermore, engaging with researchers takes time and a shift in focus for staff in educational agencies, who are often stretched to capacity with compliance and accountability work. Additionally, education stakeholders may have strong preferences for certain programs or policies, and thus fail to see the importance of assessing whether these are truly yielding measurable improvements in outcomes. Finally, staff at educational agencies may need to devote time to help researchers translate findings, since researchers are not accustomed to creating summaries of research that are accessible to a broad audience.

Given all this, why am I still optimistic about connecting research, practice, and policy? Continue reading

Why Is There a Disconnect Between Research and Practice and What Can Be Done About It?

What characteristics of teacher candidates predict whether they’ll do well in the classroom? Do elementary school students benefit from accelerated math coursework? What does educational research tell us about the effects of homework?

three interconnected cogs, one says policy, one says practice, one says research

These are questions that I’ve heard over the past few years from educators who are interested in using research to inform practice, such as the attendees of researchED conferences. These questions suggest a demand for evidence-based policies and practice among educators. And yet, while the past twenty years have witnessed an explosion in federally funded education research and research products, data indicate that many educators are not aware of federal research resources intended to support evidence use in education, such as the Regional Education Laboratories or What Works Clearinghouse.

Despite a considerable federal investment in both education research and structures to support educators’ use of evidence, educators may be unaware of evidence that could be used to improve policy and practice. What might be behind this disconnect, and what can be done about it? While the recently released Institute of Education Sciences (IES) priorities focus on increasing research dissemination and use, their focus is mainly on producing and disseminating: the supply side of research. Continue reading