Category Archives: Evaluation

Bellwether Recognized For Core Value of Synergy as Key Partner in $20 Million Grant

Collaborating across teams sounds simple, but it’s not easy, and it has even been described as “dangerous.” 

When organizations and individuals join forces with those outside their immediate circles, they do so to reach common goals, recognizing that their combined efforts, if synergistic, can create greater impact than their separate ones. But because human beings, and the groups with which we affiliate, bring our cultures, jargon, preferred solutions, and power dynamics to collaboration, synergy is not inevitable — or easy. It requires deliberate effort, a willingness to work outside of one’s typical patterns, and a desire to meld together the best of various disciplines to provide holistic solutions for complex problems. 

Synergy matters because education is complex: its problems are not one-dimensional, so its solutions can’t be either. 

Entering our tenth year as a nonprofit, Bellwether has long harnessed our organizational superpower of synergy — and in fact it’s one of our core values. We intentionally bring together people with different specialties, work experiences, identities, political affiliations, and approaches to advance a shared mission of dramatically changing education and life outcomes for underserved students. Some of my colleagues are former teachers who think through the lens of facilitation and skill-building when working with adults. Some, like myself, come from the background of evaluation, and reference the scientific method when we approach problems. Others have a business or consulting background and think about making organizations more effective and joyful places to work. Still others within Bellwether think about the policy contexts in which these organizations operate. 

This approach is getting some recognition. In late September, the U.S. Department of Education awarded a five-year grant worth $20 million to a collaborative partnership involving Bellwether Education Partners, Westat (the lead grantee), RMC Research, and Academic Development Institute to support a National Comprehensive Center (“National Center”) aimed at improving educational outcomes for all students, closing achievement gaps, and enhancing the quality of instruction. Through this work, we’ll get to take synergy to the next level by leveraging skill sets within our team and across these partnering organizations. 

Some context: Since 2002, regionally based comprehensive centers (RCCs) have been quietly providing capacity-building services to education agencies across all 50 states and territories. Funded by the Department of Education, RCCs are intentionally focused on the specific needs of the state and local education agencies they serve, which can vary based on local priorities, policies, and student populations. 

Until now, the benefits and lessons from this targeted support have remained largely within the states and regions being served. The Comprehensive Centers have done important work, but they have lacked the synergy superpower. Continue reading

Warren Wants to Slash the Program I Evaluate — But I’ve Seen It Work

Last week, Sen. Elizabeth Warren released her education plan, where she proposed slashing federal funding for the expansion of public charter schools by ending the Charter School Program (CSP). Warren claims that the CSP is an “abject failure,” citing a report by an anti-charter organization that the federal government has wasted up to $1 billion on charter schools. (The report’s lack of substance and evidence has already been raised by others.)

U.S. Senator and presidential candidate Elizabeth Warren visiting Roosevelt High School in Des Moines, Iowa. She spoke to a group of about 400 students to outline her K-12 education plan, answer questions, and pose for selfies.

Photo of Elizabeth Warren via Flickr user tabor-roeder

I know Warren is wrong firsthand. As part of my work at Bellwether Education Partners, I serve as an external evaluator for three CSP grantees (as my colleague Cara mentioned yesterday). Our unbiased, rigorous, and data-driven evaluations indicate that when implemented well, high-quality charters use their CSP funds to improve their model and successfully serve more students. These charter schools are the reason their students experience gains in achievement beyond their traditional public school peers. 

Let me step back. The CSP began in 1994 as an amendment to the federal Elementary and Secondary Education Act. It has three goals:

  1. To create promising new public charter schools,
  2. To replicate high-quality public charter schools, and
  3. To disseminate information about effective practices within charter schools.

In 15 years, CSP has awarded close to $4 billion to charters serving disproportionately more low-income and diverse students than traditional public schools. The average award is about $500K. In exchange for CSP replication funds, the schools expand to serve more students. CSP replication grantees are required to evaluate their progress, and it is strongly recommended that they use an external evaluator to do so. The evaluation must demonstrate that the grantee is doing what it said it would do and compare the achievement of its students to students attending nearby traditional public schools for evidence of impact. This is how the CSP holds grantees accountable. 

My colleagues and I use advanced statistical modeling techniques to compare charter-attending students to similar students attending nearby traditional public schools on a 1:1 basis. In the model, when all things are considered, the difference in achievement found can be attributed to the school. In fact, an emerging trend in our current data analysis suggests that the longer a student attends a well-implemented, high-quality charter school, the larger the gains in achievement over their traditional public school student peer become. Continue reading

Media: “Culture-based education — a path to healing for Native youth?” in The Hechinger Report

Today, I have an op-ed at The Hechinger Report about the benefits of culture-based education, for Native youth and all students. The piece was inspired by work our evaluation team did with The National Indian Education Association at Riverside Indian School, the nation’s oldest federally operated American Indian Boarding School.

An excerpt from my op-ed:

Culture-based education provides a path to healing and responsible citizenship for all of us. It helps students become aware of and comfortable with other belief and value systems. It furthers the goals of democracy and leads students of all ethnicities and races to think more deeply about their own cultural identities while also broadening their understanding of the experiences and perspectives of others.

Finally, the fruits of culture-based education can help us understand this country’s moral debts and how to pay them. Native Americans have for too long lived in a country controlled by men who, for nearly 300 years, have consistently “elevated armed robbery to a governing principle.” Through forced removal, boarding schools and relocation, our government stole and erased Native Americans’ languages and cultural knowledge. An investment in recovering, restoring and revitalizing lost and stolen indigenous cultural knowledge could guide us in understanding this country’s bloody history and place us on a path toward reconciliation and equity.

Read the rest of my piece at The Hechinger Report, and read more of my writing about Native education here.

Correlation is Not Causation and Other Boring but Important Cautions for Interpreting Education Research

Journalists, as a general rule, use accessible language. Researchers, as a general rule, do not. So journalists who write about academic research and scholarship, like the reporters at Chalkbeat who cover school spending studies, can help disseminate research to education leaders since they write more plainly.

But the danger is that it’s easy for research to get lost in translation. Researchers may use language that appears to imply some practice or policy causes an outcome. Journalists can be misled when terms like “effect size” are used to describe the strength of the association even though they are not always causal effects.

To help journalists make sense of research findings, the Education Writers Association recently put together several excellent resources for journalists exploring education research, including 12 questions to ask about studies. For journalists (as well as practitioners) reading studies that imply that some program or policy causes the outcomes described, I would add one important consideration (a variation on question 3 from this post): if a study compares two groups, how were people assigned to the groups? This question gets at the heart of what makes it possible to say whether a program or policy caused the outcomes examined, as opposed to simply being correlated with those outcomes.

Randomly assigning people creates a strong research design for examining whether a policy or program causes certain outcomes. Random assignment minimizes pre-existing differences among the groups, so that differences in the outcomes can be attributed to the treatment (program or policy) instead of different characteristics of the people in the groups. In the image below, random assignment results in having similar-looking treatment and control groups. Continue reading

What This Washington Post Opinion Piece Got Wrong on Charter Schools

Over the weekend, the Washington Post Outlook section ran a frustrating cover story on charter schools that offered a narrow and biased picture of the charter sector and perpetuated a number of misconceptions.

Jack Schneider’s “School’s out: Charters were supposed to save public education. Why are Americans turning against them?” argues that the charter sector as a whole isn’t living up to its promises, leading public support for the schools to shrink. Schneider is correct that the charter school hasn’t lived up to all of its most enthusiastic boosters’ promises, but his piece flatly misrepresents data about charter quality. For example, Schneider writes that “average charter performance is roughly equivalent to that of traditional public schools.” This is simply inaccurate, as my colleagues indicated in a recent analysis of charter data and research (slide 37 here). The full body of currently available, high-quality research finds that charters outperform traditional public schools on average, with especially positive effects for historically underserved student groups (a recent Post editorial acknowledged this as well).

slide from Bellwether's "State of the Charter Sector" resource, summarizing research on charter sector performance

To be clear, research also shows that charter performance varies widely across schools, cities, and states — and too many schools are low-performing. Yet Schneider cherry picks examples that illustrate low points in the sector. He cites Ohio, whose performance struggles — and the poorly designed policies that led to them — Bellwether has previously written about. He also (inexplicably, given where his piece ran) overlooks Washington, D.C., where charters not only significantly outperform comparable district-run schools, but have also helped spur improvement systemwide. Over the past decade, public schools in D.C. (including both charters and DC Public Schools, DCPS) have improved twice as fast as those in any other state in the country, as measured by the National Assessment of Educational Progress (NAEP). DCPS was the nation’s fastest growing district in 4th grade math and among the fastest in 4th grade reading and 8th grade math. These gains can be partially attributed to the city’s changing demographics, but are also the result of reforms within DCPS — which the growth of charters created the political will to implement. Over the past decade, Washington, DC has also increased the number of high-performing charter schools while systematically slashing the number of students in the lowest-performing charter schools. When I served on the District of Columbia Public Charter School Board from 2009-2017, I had the chance to observe these exciting changes firsthand, so it was particularly disappointing to see a major feature in our city’s paper overlook them.

It’s frustrating that this biased and narrow picture drew prime real estate in one of the nation’s leading papers, because the charter sector does have real weaknesses and areas for improvement that would benefit from thoughtful dialogue. For example, as Schneider notes, transportation issues and lack of good information can prevent many families from accessing high-quality schools. In cities with high concentrations of charters, such as Washington, D.C. and New Orleans, there is a real need to better support parents in navigating what can feel like a very fragmented system. And despite progress in closing down low-performing charter schools, too many remain in operation. Schneider could have referenced the real work charter leaders are undertaking to address these lingering challenges (more on this in slide 112 of our deck).

Schneider is correct that public support for charters has waned in recent years, due in part to some of the challenges he references, but also because of orchestrated political opposition from established interests threatened by charter school growth. Given the increasingly polarized political environment around charter schools, the need for nuanced, balanced, and data-informed analysis and dialogue about them is greater than ever. Bellwether’s recent report on the state of the charter sector, and our past work on charter schools more broadly, seeks to provide that kind of analysis. Unfortunately, Schneider’s piece falls short on that score.