Category Archives: Evaluation

How Can Educators Evaluate Virtual Student Engagement During the COVID-19 Pandemic?

Before the COVID-19 pandemic, educators commonly assessed student engagement using student self-report measures, teacher reports, and observational measures. These measures work for in-person learning environments where student participation and connectivity can be easily visible.

Now most teaching and learning occurs in virtual spaces, where instruction is delivered through learning management systems (e.g., Google Classroom) either synchronously or asynchronously. Teachers and school leaders have to engage students — and evaluate their engagement — in a remote environment with little formal or comprehensive training and often a limited ability to even see their students

There has been a great deal of writing recently about improving student engagement, but few resources provide guidance around measuring student engagement during remote learning. 

Based on a close read of the existing research and resources, and our own in-house expertise, Bellwether’s evaluation team is currently helping three clients to design and implement tools to evaluate virtual student engagement. While the measures themselves have not changed much since the pandemic, the uses and evaluation of these measures must adapt to our current realities, like many other approaches in education, to better serve and engage students and families. 

Here are two key considerations for educators and school leaders to keep in mind when evaluating virtual student engagement:

There is no one-size-fits-all model for measurement  Continue reading

FAQs for Future Applicants to the Federal Charter School Program Grant

As applicants anxiously await the results of the FY2020 Charter School Program (CSP) State Entities grant competition, we want to offer some tips for prospective future applicants. As my Bellwether colleagues recently wrote, the CSP is a discretionary grant that provides federal resources to create, replicate, and support high-quality public charter schools. Developing a strong CSP application takes significant time and forethought. Although future funding of the CSP hangs in the balance, charter networks thinking about applying should plan far in advance to develop a strong application. 

Bellwether has partnered with a number of charter management organizations to develop winning federal education grant proposals, including CSP Replication and Expansion grants. The Frequently Asked Questions below explain what differentiates a successful application and provide advice on developing a winning proposal. 

Logistics of applying 

When should I start thinking about applying for a CSP grant? 

Six-to-eight-week turnarounds are fairly common: in 2019, the notice inviting applications appeared on November 26, 2019 and the deadline for transmittal of applications was January 10, 2020. Because the turnaround is pretty quick, occurs at a time of year when many staff may be planning time off, and the applications themselves are often over sixty pages long, preparing in advance is very helpful. 

As you think about applying, consider your network’s readiness to grow and increase impact. Indicators of readiness to grow can cross multiple dimensions, such as quality of programming, strength of student outcomes, clarity of instructional and cultural visions, student and staff retention and satisfaction, and financial health and sustainability. Bellwether offers a “Readiness to Grow” diagnostic tool that can help organizations assess their strengths and areas for focus before or during a growth process (see case study that used this tool here).  Continue reading

Bellwether Recognized For Core Value of Synergy as Key Partner in $20 Million Grant

Collaborating across teams sounds simple, but it’s not easy, and it has even been described as “dangerous.” 

When organizations and individuals join forces with those outside their immediate circles, they do so to reach common goals, recognizing that their combined efforts, if synergistic, can create greater impact than their separate ones. But because human beings, and the groups with which we affiliate, bring our cultures, jargon, preferred solutions, and power dynamics to collaboration, synergy is not inevitable — or easy. It requires deliberate effort, a willingness to work outside of one’s typical patterns, and a desire to meld together the best of various disciplines to provide holistic solutions for complex problems. 

Synergy matters because education is complex: its problems are not one-dimensional, so its solutions can’t be either. 

Entering our tenth year as a nonprofit, Bellwether has long harnessed our organizational superpower of synergy — and in fact it’s one of our core values. We intentionally bring together people with different specialties, work experiences, identities, political affiliations, and approaches to advance a shared mission of dramatically changing education and life outcomes for underserved students. Some of my colleagues are former teachers who think through the lens of facilitation and skill-building when working with adults. Some, like myself, come from the background of evaluation, and reference the scientific method when we approach problems. Others have a business or consulting background and think about making organizations more effective and joyful places to work. Still others within Bellwether think about the policy contexts in which these organizations operate. 

This approach is getting some recognition. In late September, the U.S. Department of Education awarded a five-year grant worth $20 million to a collaborative partnership involving Bellwether Education Partners, Westat (the lead grantee), RMC Research, and Academic Development Institute to support a National Comprehensive Center (“National Center”) aimed at improving educational outcomes for all students, closing achievement gaps, and enhancing the quality of instruction. Through this work, we’ll get to take synergy to the next level by leveraging skill sets within our team and across these partnering organizations. 

Some context: Since 2002, regionally based comprehensive centers (RCCs) have been quietly providing capacity-building services to education agencies across all 50 states and territories. Funded by the Department of Education, RCCs are intentionally focused on the specific needs of the state and local education agencies they serve, which can vary based on local priorities, policies, and student populations. 

Until now, the benefits and lessons from this targeted support have remained largely within the states and regions being served. The Comprehensive Centers have done important work, but they have lacked the synergy superpower. Continue reading

Warren Wants to Slash the Program I Evaluate — But I’ve Seen It Work

Last week, Sen. Elizabeth Warren released her education plan, where she proposed slashing federal funding for the expansion of public charter schools by ending the Charter School Program (CSP). Warren claims that the CSP is an “abject failure,” citing a report by an anti-charter organization that the federal government has wasted up to $1 billion on charter schools. (The report’s lack of substance and evidence has already been raised by others.)

U.S. Senator and presidential candidate Elizabeth Warren visiting Roosevelt High School in Des Moines, Iowa. She spoke to a group of about 400 students to outline her K-12 education plan, answer questions, and pose for selfies.

Photo of Elizabeth Warren via Flickr user tabor-roeder

I know Warren is wrong firsthand. As part of my work at Bellwether Education Partners, I serve as an external evaluator for three CSP grantees (as my colleague Cara mentioned yesterday). Our unbiased, rigorous, and data-driven evaluations indicate that when implemented well, high-quality charters use their CSP funds to improve their model and successfully serve more students. These charter schools are the reason their students experience gains in achievement beyond their traditional public school peers. 

Let me step back. The CSP began in 1994 as an amendment to the federal Elementary and Secondary Education Act. It has three goals:

  1. To create promising new public charter schools,
  2. To replicate high-quality public charter schools, and
  3. To disseminate information about effective practices within charter schools.

In 25 years, CSP has awarded close to $4 billion to charters serving disproportionately more low-income and diverse students than traditional public schools. The average award is about $500K. In exchange for CSP replication funds, the schools expand to serve more students. CSP replication grantees are required to evaluate their progress, and it is strongly recommended that they use an external evaluator to do so. The evaluation must demonstrate that the grantee is doing what it said it would do and compare the achievement of its students to students attending nearby traditional public schools for evidence of impact. This is how the CSP holds grantees accountable. 

My colleagues and I use advanced statistical modeling techniques to compare charter-attending students to similar students attending nearby traditional public schools on a 1:1 basis. In the model, when all things are considered, the difference in achievement found can be attributed to the school. In fact, an emerging trend in our current data analysis suggests that the longer a student attends a well-implemented, high-quality charter school, the larger the gains in achievement over their traditional public school student peer become. Continue reading

Media: “Culture-based education — a path to healing for Native youth?” in The Hechinger Report

Today, I have an op-ed at The Hechinger Report about the benefits of culture-based education, for Native youth and all students. The piece was inspired by work our evaluation team did with The National Indian Education Association at Riverside Indian School, the nation’s oldest federally operated American Indian Boarding School.

An excerpt from my op-ed:

Culture-based education provides a path to healing and responsible citizenship for all of us. It helps students become aware of and comfortable with other belief and value systems. It furthers the goals of democracy and leads students of all ethnicities and races to think more deeply about their own cultural identities while also broadening their understanding of the experiences and perspectives of others.

Finally, the fruits of culture-based education can help us understand this country’s moral debts and how to pay them. Native Americans have for too long lived in a country controlled by men who, for nearly 300 years, have consistently “elevated armed robbery to a governing principle.” Through forced removal, boarding schools and relocation, our government stole and erased Native Americans’ languages and cultural knowledge. An investment in recovering, restoring and revitalizing lost and stolen indigenous cultural knowledge could guide us in understanding this country’s bloody history and place us on a path toward reconciliation and equity.

Read the rest of my piece at The Hechinger Report, and read more of my writing about Native education here.