Tag Archives: USEII

Reactions to the U.S. Education Innovation Index

One of the main goals of creating and publishing the U.S. Education Innovation Index Prototype and Report was to stimulate evidence-based conversations about innovation in the education sector and push the field to consider more sophisticated tools, methods, and practices. Since its release three weeks ago at the Digital Promise Innovation Clusters convening in Providence, the index has been met with an overwhelmingly positive reception.

I’m grateful for the many fruitful one-on-one conversations that have pushed my thinking, raised interesting questions, and provoked new ideas.

Here are a few takeaways on the report itself:

People love radar charts. And I’m one of those people. In the case of the innovation index, radar charts were a logical choice for visualizing nine dimensions and a total score. Here they are again in all their glory.

City Comparisons

Readers weren’t always clear on the intended audience or purpose. This concern came up often and hit close to home as someone who strives to produce work that is trusted, relevant, timely, and useful. One of the benefits of the prototype is that we can test the tool’s utility before expanding the scope of the project to more cities or an even more complicated theoretical framework. So far the primary audience for the index funders, policy makers, superintendents, education leaders, and city leaders have demonstrated interest in learning more about the thinking behind the index and how it can be applied to their work. Ultimately I hope it will influence high-stakes funding, policy, and strategic decisions.

The multidimensionality of innovation challenges assumptions. When I explain that we measured the innovativeness of education sectors in four cities New Orleans, San Francisco, Indianapolis, and Kansas City, MO inevitably, the next question I get is “how do they rank?” Instead of answering, I ask my conversant for his/her rankings. I’ve had this exchange dozens of times, and in almost every case, New Orleans topped the list because of the unique charter school environment. When I then explain that the index was sector agnostic it doesn’t give preference to charter, district, or private schools people immediately reconsider and put San Francisco in the number one slot. What this tells me is that many people associate innovation with one approach rather than treating it as the multidimensional concept that it is. This misperception has real policy and practice implications, and I hope the index provides nuance to the thinking of decision makers.

Dynamism” and “district deviance” are intriguing but need more research. Two of the measures that I’m most excited about are also ones that have invited scrutiny and criticism: dynamism and district deviance. Dynamism is the entry and exit of schools, nonprofits, and businesses from a city’s education landscape. Too much dynamism can destabilize communities and economies. Too little can keep underperforming organizations operating at the expense of new and potentially better ones. In the private sector, healthy turnover rates are between five and 20 percent, depending on the industry. We don’t know what that number is for education yet, but it’s likely on the low end of the range. More research is needed. Our district deviance measure assumes that districts that spend their money differently compared to their peers and are trying new things, which is good. It’s a novel approach, but its accuracy is vulnerable if the assumptions don’t pan out. Again more research is needed.

Measure more cities! Everyone wants to see more cities measured with the index for one of two reasons. The first is that they want to know how their city is doing on our nine dimensions. The second is that they want to compare cities to each other. Both make my heart sing. Knowing how a specific city measures up is the first step to improving it. Knowing how it compares to others is the first step to facilitate knowledge transfer and innovation diffusion.
Continue reading

How the Innovation Paradox Rocked My World

Innovation-Student Achievement Relationship Plot

Credit: Jason Weeby

When my colleagues Kelly Robson and George Mu and I began a project to measure innovation in a city’s education sector, we knew it was going to be challenging. Innovation isn’t a single thing; you can’t just go out and count innovation. Instead, it’s a combination of many factors, some of which matter more than others.

So we created a composite indicator, a macroeconomic tool that is formed when individual indicators are compiled into a single index, based on an underlying model of the multi-dimensional concept that is being measured. Composite indicators are commonly referred to as indices. They measure concepts like competitiveness, sustainability, and opportunity. We call ours the U.S. Education Innovation Index.

Some of the categories that we included are novel like District Deviation and Dynamism (topics for other posts). Others are more predictable, like innovation-friendly policies and the level of funding available for innovation-specific activities.

However, one category continues to disorient me. It isn’t unusual. I read about it every day. It’s something one would fully expect to see in a measure of a city’s education sector, yet it requires an explanation to anyone who has interrogated our methodology: Student Achievement.

How we ultimately decided to measure student achievement was the product of hours of discussion and analysis. At first, we considered it an output of innovation. “Makes sense,” I thought. If you turn up the dial on innovation activities, new solutions emerge, and then student achievement goes up.

I was content with our tidy framework of inputs and outputs until Julia Freeland Fisher, director of education at the Clayton Christensen Institute for Disruptive Innovation, and Matt Candler, founder and CEO of 4.0 Schools, introduced me to the innovation paradox and blew up my well-laid plans.

What I was missing was the relationship between success and the motivation to innovate.

Innovation, especially disruptive innovation, is more likely to happen in cities where student achievement is low. In theory, poor or declining student achievement is likely to embolden entrepreneurs and catalyze innovation. In cities where student achievement is perennially low, policymakers and education officials may feel pressure to try new tactics or adopt new policies or methodologies, and thus embrace innovative ideas.

I sketched out what the relationship might look like on a graph above. It’s not as cool looking as Candler’s sketches, but shows the time lag between the implementation of innovation activities and how they should increase student achievement.

The paradox emerges in stage two where innovation activities decline as student achievement remains high. In cities where schools are consistently performing at a high level or are improving steadily, officials may be hesitant or reluctant to change anything out of fear of reversing a positive academic trajectory. Continue reading