U.S. News and World Report recently announced that in addition to its annual rankings of high schools, colleges and graduate schools, the company was releasing a list ranking the top elementary and middle schools in the country. Immediately, teachers, school officials, scholars and commentators on both the left and the right overwhelmingly panned the announcement.

Although U.S. News has long been criticized for distorting perceptions of schools and the choices of school leaders by ranking schools based on a handful of “performance” variables, Americans have always been concerned with the quality of their schools, a concern that has frequently led them to quantify schooling. In fact, the impulse driving the U.S. News rankings, which so many found disturbing, also drives much of contemporary education policy.

Evaluation and comparison have always been a way to assuage anxieties and uncertainties about educating the young. The very first standardized tests in American public schools were given in Boston in 1845. The brainchild of Horace Mann and his colleague, Samuel Gridley Howe, the tests were intended to demonstrate the need for serious reforms in public schools.

When test score information wasn’t available, school officials sought other information as proxies for school quality. For instance, the percentage of enrolled students attending school on a daily basis and the number of “over aged” students in a grade were understood as indicators of school efficiency and objective points of comparison across school systems.

Throughout the first decades of the 20th century, school statistics and ranked lists became common features of annual reports and newspaper coverage as the public sought to understand how well their schools were doing.

But even as these statistics circulated, experts recognized their considerable limitations. Education was an inherently local affair, so information was valuable only in the context of local decision-making. Without standard curriculums, textbooks, funding formulas, teacher licensure or graduation requirements across states or even districts, how useful could statistical comparisons really be?

Indeed, in 1959, after conducting a nationwide study of high schools, Harvard University president James Conant concluded that it was “impossible” to discriminate between them. There were “too many high schools of too many different types” to allow for generalizations — one could “make valid judgments about American secondary education, but only school by school.”

Conant’s warning went unheeded. Throughout the 1950s and ’60s policymakers and analysts became increasingly convinced that schools, like businesses, were just “systems” that brought various inputs together to produce desired outputs. In this view, the system could be optimized simply by measuring, monitoring and adjusting the inputs. Such a view was deeply appealing to federal policymakers who, in the midst of the Cold War, had become interested in maximizing the development of American brainpower.

The problem: It was deeply out of step with the reality of American schooling and required standardized information that simply didn’t exist.

In 1958, researchers with the New York Quality Measurement Project, one of the first federally-funded projects to scrutinize the relationships between inputs and outputs, found that even in a state with a relatively centralized school system like New York there was too much variation in district record keeping to collect usable information. The result was an effort that would be repeated regularly over the next half century: In the absence of a standardized system of schools, researchers produced standardized data about schools — about the quality of their institutional resources, the character of their communities, the performance of their students — to stand in its place and enable statistical analysis.

The stylized statistical portraits of the U.S. school system created by these data sets provided descriptive insight but imposed an artificial order on an inevitably messy reality. For instance, in the absence of national curriculum, the National Assessment of Educational Progress (NAEP), which was first given in 1969, promised a nationalized picture of student achievement. But the administered test didn’t resemble the curriculum students were exposed to. How could it in a country with no national standards and no national curriculum? As a result, the information produced could not be used to improve school quality. One researcher remarked that even if uniform data about the nation’s schools could be produced, little would be gained through its analysis because “Chicago and San Francisco differ on so many dimensions that it is not an interpretable comparison.”

Nevertheless, the possibility of collecting and mining mountains of newly available data proved extremely appealing to a new generation of policymakers and analysts trained in quantitative analysis as well as to reformers coming to terms with the dimming prospects for radical systemic change. In the absence of big budgets or social movements for change, policymakers narrowed their focus to variables — class size, algebra for all, teacher credentials — that required only organizational change and, therefore, were available irrespective of local context or the prevailing politics.

In 1972, Harvard sociology professor Christopher Jencks’ book “Inequality” sought to draw on the newly available school performance data to argue that school systems were fundamentally incapable of addressing social inequality.

Unsatisfied with this “new quantification,” a group of Black scholars including Ronald Edmonds, Andrew Billingsley and James Comer pointed out the local realities that these generalized statistical accounts ignored, but that profoundly shaped racial inequality in schools. “We hasten to point out,” they wrote, “that public schools are not now, nor have they ever been committed to the radical notion that they are obliged to teach certain minimum school skills … to all pupils.” To announce a statistical relationship between schools and inequality without consideration of past and present inequities was, in their view, to absolve society of its obligation to provide quality education to all children. These scholars worried that the statistics would be used to short-circuit the political push for equality.

Those concerns proved well founded.

Indeed, the foundational premise of No Child Left Behind in 2001 and its successor, the Every Student Succeeds Act, was that schools can achieve equality in student test scores, irrespective of history or place. Attempts to excuse or explain variation by pointing to historical injustice or contemporary inequality was taken as a sign of what President George W. Bush described in 2000 as the “soft bigotry of low expectations.”

Today, the production of quantitative data about schools — the very same data used to compute the U.S. News rankings — has become the backbone of U.S. school reform. Whereas a century ago these data served as a basis for political debates about schools, their production is now often seen as an end in itself – not to facilitate public debate but to enable private decision-making, often through parent choice.

But numbers narrow attention and shift blame: The data displayed in ranking tables imply that if only the leaders of a particular school would offer more AP classes, improve student-teacher ratios or raise test scores, then theirs could be among the “best” schools. This simplicity is appealing: It implies a clear silver bullet for school improvement. It also paints a picture of schooling that is divorced from reality. Regardless of the picture presented in the ranking tables, the performance of schools cannot be understood separated from place, politics and history.

Our willingness to accept comparisons at face value, without interrogating the historical and contemporary processes that produced them, has left us chasing statistical trends instead of taking on the political challenges necessary to improve our schools.