Charting A Course: Evaluation Design Of The National School District And Network Grants Program

Introduction

Over the past decade, a number of reform efforts have aimed at reducing the size of the learning communities in the nation’s schools. Most recently, the Bill & Melinda Gates Foundation has committed more than $375 million to improving America’s high schools through its National School District and Network Grants Program. Likely to be one of “the most publicly scrutinized educational initiatives” in recent history, the foundation’s program is providing a catalyst to educators and policy makers to radically reshape secondary education in the United States. The foundation is working with school districts and network organizations to create education systems that work for all students, including those students who have been characterized in the past as most underserved by the nation’s education system. At the secondary level, the foundation is supporting small schools as a corrective to the large, impersonal “shopping mall” high schools that dominate today. To this end, the foundation supports: 1) districts and network organizations in converting large high schools into smaller learning communities, and 2) network organizations’ efforts to replicate successful model schools or to implement small, effective schools from the ground up.

Through a competitive award process, the foundation selected our organizations—the American Institutes for Research (AIR) and SRI International—to evaluate its initiative. We are currently beginning our third year of data collection. The purpose of this paper is to describe the framework underlying our evaluation design of a large-scale, multifaceted school reform initiative and to identify some of the methodological challenges inherent in work of this kind. Throughout this paper, we use travel as a metaphor for our evaluation design process. Specifically, we begin by describing the earliest stages of our journey—our starting point. Here we outline the foundation’s theory of change—their goals and vision for the initiative. We then turn to a discussion of defining the destination—the research questions we seek to answer—and finding the coordinates—the conceptual framework that will guide our evaluation. Next, we explore the terrain, providing an overview of the grantees and schools in the population, and determine our route (e.g., the design basics of our evaluation). We conclude with a discussion section that we have entitled, Reading the Map While Driving. Here we discuss a few of the challenges of our evaluation and our need for reflection and refinement as the initiative grows and changes and as we learn through our research activities.