let's take a deep dive into that Times article on school choice

The other day the New York Times published a big investigative piece into the school choice movement here in New York, by Elizabeth A. Harris and Ford Fessenden. Typical of its venue, it's well reported and a serious attempt to grapple with complex issues. But it fails in ways that are very common to our educational reporting and which stem in part from the Official Dogma. Let's dive in.

Student Outcomes ≠ Teacher Quality

First, a biggie: the assumption that you can simply compare relative outcomes of different groups of students and learn something meaningful about teacher or school quality, using graduation rates and similar raw measures to reflect on school quality.

This is galling in part because the Times just made the same mistake in a piece on the Upshot, where they misused research on performance relative to grade level. That's not me saying it was misused, that's the researchers themselves saying it was misused. As I hope is clear to anyone who reads this blog, simply comparing different outcomes is an invalid way to measure schools and teachers because students come in with large, preexisting differences in prerequisite ability. The sources of that inequality are multiple and complex, but they are real and inarguable. Even if you have a higher opinion than I do of various value added models and other efforts to adjust for ability effects, you should know that you have to at least do something to account for these differences. Here's the caveat we get:

Graduation rates are not a perfect proxy for education quality. In many schools, students arrive far behind, and it is a major effort to help them graduate on time. Elsewhere, ninth graders show up on Day 1 doing work at grade level or above, so the steps required to get them diplomas are less onerous. And it is difficult to say how much of a school’s success is because of what happens within its walls — the curriculum, the teachers, the leadership — and how much is because of advantages children bring from home. But graduation remains a meaningful measure of a school, and of the opportunities it provides.

Graduation remains a meaningful measure of social outcomes - but not a measure of school quality.

Selection Bias at Its Most Stark

Here's the kind of paragraph that lays out the basic dynamics so starkly it amazes me that the writers don't really catch on.

The citywide graduation rate for all kinds of high schools is 72.6 percent, according to the Education Department. But that average masks sharp variations between schools based on their admissions methods. When Measure of America analyzed the rate for each method, it found that selectivity and graduation rates declined essentially in lock step, and that as graduation rates fell, the students were more likely to be poor and black or Hispanic.

Yes, indeed - selectivity and graduation rates decline in step. The screening mechanism is the single most powerful determiner of their own metric of school quality! The black and Hispanic students live and learn in conditions that are, on average, profoundly different than those of their white peers, in ways that are apparent to anyone who knows the distribution of advantage in this country. That the reporters don't make the obvious conclusion here would be remarkable if not for the context of education journalism in general. In that context, it's sadly common.

Everyone Cannot Be Above Average

Again, I don't mean to be endlessly previewing a post, but I keep meaning to lay out this basic point about education policy: we think what we care about is absolute educational gains (are students learning?) but what we really care about is relative educational gains (are students who typically perform poorly relative to peers making gains?). If the question was simply whether students are learning as they age, the news would be very good. Average people keep getting smarter all the time, across every identifiable demographic grouping. Meanwhile, the biggest impediment to closing the racial achievement gap is the fact that white and Asian kids keep learning too; black students, as a class, learn as much as white peers during the school year. They simply start off behind, even prior to formal schooling. So "are students learning?" isn't really the question. Are students learning "enough" in an absolute sense? It's not a question that we can answer coherently because no one knows what enough might be.

What people really care about is relative performance, but they don't know it, and lack a vocabulary to really understand what they're asking for. So consider this passage.

Kristen Lewis, one of the directors of Measure of America, said the data revealed, in essence, two separate public school systems operating in the city. There are some great options for the families best equipped to navigate the application process. But there are not enough good choices for everyone, so every year thousands of children, including some very good students, end up in mediocre high schools, or worse.

“The average kid has to be able to get a good education, because most people are average,” Ms. Lewis said. “It’s great that the highfliers are succeeding, and they deserve the chance to succeed. But so do the average kids.”

Now: I absolutely agree that what matters is the median student, not the ones at the top. Ed talk tends to be obsessed with the kid who pulls herself up out of poverty, goes to Stanford, and founds a startup. But those cases are necessarily few and far between. (When people talk about creating equality in education, what kind of equality do they mean? Actual, summative equality in terms of quality of life? Or an equal chance to leave the rest behind? I don't think the people who talk like that have the slightest idea.) So I want to affirm that focus on Johnny Average and Jane Average.

But: what then can a "good education" mean, if it's combined with the idea that most people are average? What can succeeding mean if we concede that most people are average? What people typically mean by a good education is strong relative performance on academic benchmarks. But the average student can only ever reach an average position on those benchmarks, in the basic, Lake Wobegon sense. If a student climbs above others, those others necessarily move down the rankings. There's only one valedictorian, only 10% of a graduating class in the top 10%, only so many spots at Harvard. Indeed, the value of those relative rankings derives directly from the limited number of slots within them. And it's truly a zero sum game. One kid climbing up to the top 10 of her class means another kid loses his spot. That's inevitable; it's a basic fact of life. If you're her, or her parents, then fine - you got yours. But from the point of view of the system, which is the point of view of policy, swapping one kid out and another one in is only a victory if you favor the interests of the kid who's made it to the top. This is an persistent source of enormous confusion in our conversations about education.

You could say that what we want is to squash the variance in outcomes - to narrow up the bell curve on test scores and GPA and the like so that all students are more tightly spread near the average. That would necessarily reduce the number of high outliers too, which seems contrary to the whole American ethos of excellence, but as a commie I'm cool with it. But is that what we really want? And do we have any evidence that, at scale, we can narrow the spread in that way? I'm skeptical.

Rather, the implicit endorsement here and elsewhere seems to be of academic mobility, not of academic equality. Just as people implicitly treat the economic system as more just if any individual can rise out of poverty to get a good job, people think that the system is more just if a poor black kid can rise out of a Bronx public school and go to Yale. While I would love it if more poor black kids could rise out of Bronx public schools to Yale, those students would simply be leaving most other kids behind. Why should we value the interests of the risers over the mass of students? Again, we just said that the average student is what matters, and the average student will always be average. And as with students, so with schools. If you close the poorest performing schools, you simply move the next set of schools down the rankings of performance. There's always going to be a distribution of outcomes in any ordinal metric; that's a basic feature of a world of variability.

The fact of the matter is, mobility is necessarily antagonistic to equality. Every student who moves up pushes another one down. These values are in direct tension, and yet no one seems to pause for a moment and really critically evaluate what we're asking for. If your interest is in promoting equality then you should agitate against mobility, as true mobility will result only in more outliers - both above and below the mean. If instead you are concerned with simply providing a better quality of life for the most possible people, then you should focus on redistributive economic systems that ameliorate the effects of poverty and create a downward pressure on the wealth of those at the very top. Then you can allow education to go back to being education, rather than seeing it constantly as an instrument of economic manipulation - a role for which it has proven totally unsuited.

What We Want From Education is Completely Undertheorized

I don't want to be harsh to the reporters here. Their concern for these kids is genuine and commendable. But I am struck again and again in the piece by the sense that they don't know, really, what we want from schools. The reporters want, in a vague way, for all students to be able to attend a "good school." And so do I, of course. But they don't define what a good school is in a way that makes sense, and ultimately this contributes to an environment where schools and teachers are blamed for conditions they can't possibly control. Consider this passage.

Sean P. Corcoran, an associate professor of economics and education policy at New York University, has researched the choice process and how students match. He said that the best option is for students to reach for the best possible school for which they are qualified, and indeed, most students get one of their top choices. But in many cases, students reach either too low or too high.

Again: there are limited seats at every school. And if you could somehow dramatically upscale the carrying capacity of individual schools, there's no reason to believe that they would maintain the relative advantage Dr. Corcoran is talking about here. Indeed, the selection bias that haunts the entire article suggests that expanding access to disadvantaged students will result in poorer outcomes and thus lower perceived school quality. Moreover, what about the students who are not qualified for a "good" school? What happens to them? Here's a sentence from later in the article:

These improvements make real differences in the lives of students, but they leave plenty of room for children to fall through the cracks.

This should go without saying: if we're looking for the students who are most qualified, we are ensuring that some students fall through the cracks. To elevate one student up an ordered ranking is to deepen the fall of the student left behind. If instead we insist that all students are equally qualified, we are necessarily erasing the notion of a good student entirely. These contradictions are basic and unavoidable, and yet they simply are not discussed in prominent educational writing and reporting.

All of this stems from good intentions and a commitment to social justice. But what's utterly clear is that neither of the reporters who put together this article, nor many of the stakeholders, really understands their own desires. This is broadly true of education policy in general. It's not just that we can't get what we want. It's that nobody really knows what they're trying to accomplish. Our educational system cannot simultaneously be a tool for creating equality and an instrument for rewarding excellence. Nor has education ever been demonstrated to end poverty or reduce inequality. And until we have some tough conversations about what schools can and can't accomplish, all of our promises are going to be broken.