No, That's Not What "the Research" Says About Exam Schools

pretty much the opposite, in fact

The New Yorker has just run a piece by Jessica Winter about New York City’s endless controversies over its gifted & talented programs generally and specifically its famed exam schools, special high schools that require students to do well on a standardized test and boast many famed alumni, schools like Stuyvesant High School and Bronx High School of Science. As is always true of American education reporting, the piece is really about meritocracy, opportunity, and racial inequality.

Because New York is New York, the roiling debates about these programs and schools are often used as a microcosm of the broader American educational landscape. The enduring controversy is that these programs and these schools demonstrate the same racial inequities that are present in schooling writ large - Asian students are significantly overrepresented relative to their portion of the population, Hispanic students are somewhat underrepresented, and Black students are significantly underrepresented. Racial inequality in public schooling, as I’ve said many times, has been the obsessive focus of the policy apparatus for at least 45 years. In general, achievement gap fixation has crowded out many more fundamental questions - for example, whether to spend on gifted & talented programs or exam schools. As that NYT piece points out, just paying for the exam that sorts students into specialized high schools will cost NYC $17 million.

I’m not very moved by the whole gifted & talented debate. On the one hand, progressive criticisms of the existence of such programs do reflect the broader Official Dogma and all of its problems, that is to say, a rigidly environmentalist view of academic potential that suggests that our overarching problem is an inexplicable societal decision to only give the best educations to the most privileged, and that if we got rid of the whole concept of being gifted or talented and insisted on pure educational egalitarianism, there would be no academic hierarchy and no academic inequality. This is, to put it mildly, not compatible with reality. On the other hand, I’m not super invested in this topic because there’s really no evidence that G&T programs actually improve outcomes (Winter conspicuously doesn’t provide any) and the kids who would be in such programs are going to be fine because… they’re gifted and talented. Possessing the pre-entry ability necessary to get into these programs is vastly more valuable than attending them, and an immense amount of data supports that fact.

Despite my general ambivalence towards the larger issue here, I feel compelled to throw the flag on this piece. I’m afraid that Winter pulls something that is not at all surprising in the context of liberal attitudes and liberal publications, but which is nonetheless misleading and in need of response. She writes

It’s easy to caricature some G. & T.-curious parents as grasping, status-obsessed, or slightly deluded about their child’s special brand of specialness. But research shows that the kinds of kids who might just miss a shot at Hunter or Anderson—not necessarily geniuses or savants, just very bright, driven, academically oriented kids—are likely to become inattentive, frustrated, or disruptive in a gen-ed classroom, with possible long-term effects on their academic performance and social-emotional development.

Yeah, this is just wrong.

There’s no link and no references to what “research” refers to, so it’s hard to assess what Winter is talking about. But in the context of exam schools specifically, which is what the piece largely concerns, the evidence is very strong: once you correct for pre-entry ability, such schools do not offer any quantitative advantage over regular high schools from the same cities or districts. There are a variety of ways to assess these things and a variety of outcome variables, but the research points in only one direction. A lot of this research uses a regression discontinuity design¹, taking advantage of the fact that exam school cut scores provide a clean break between those who get in and those who don’t, even though the test results are themselves continuous. (Sometimes you hear these referred to as “last in, last out” models.) The logic is intuitive: students who score similarly on entry exams but fall on either sides of the cut line can provide pretty good evidence as to whether there’s a big effect from attending the exam school itself. You can compare those who just meet the cut score and those who just miss that score, statistically correct for differences in their scores, and then check academic and life outcomes, such as high school graduation rates, GPAs, SAT scores, college completion rates, post-collegiate income, etc.

The most obvious citation here is Atila Abdulkadiroğlu, Joshua Angrist, and Parag Pathak's NBER paper “The Elite illusion: Achievement Effects at Boston and New York Exam Schools.” They find that, looking at the famed selective high schools in both cities, “there is little effect of an exam school education on achievement even for the highest-ability marginal applicants and for applicants to the right of admissions cutoffs… the intense competition for exam school seats does not appear to be justified by improved learning for a broad set of students.” A follow up from the same researchers focused on students further from the cut point, which means there’s more statistical adjustment and more noise. Still, the result was consistent: there’s no quantitative reason to believe that the students who go to exam schools are seeing any systematic advantage for doing so. Earlier NBER research on New York exam schools found similarly, with the abstract summarizing results in saying “attending an exam school has little impact on Scholastic Aptitude Test scores, college enrollment, or college graduation -- casting doubt on their ultimate long term impact.” You can find similar research about selective high schools from other cities and other countries. For example, here’s research from China which finds that “elite exam high schools, which are the most selective, have no effects on student test scores.” There’s much more research to find and very little of it cuts in the other direction.

I do not and have never claimed expertise in this domain, not any that would be recognized by the august New Yorker, anyway. But I am intimately familiar with the exam school literature, in large part because the controversy about them - and the lack of any quantitative justification for that controversy - so perfectly speaks to our broader education debates. And I can tell you that there’s no responsible summary of “research” that suggests that attending these schools matter for either high school performance, college attendance, or later economic outcomes. This makes perfect sense if you assume, as is sensible, that educational outcomes are dominantly the product of student-side factors rather than school side factors, as Douglas Detterman explained a decade ago. Winter talks about the emotional, affective problems faced by overqualified students stuck in slower-moving classrooms and asserts future negative academic consequences. But there simply is no responsible evidence that suggesting that this dynamic exists. Instead, what we see is that students gravitate to a particular level of performance consistent with their individual talent, situation, and support system regardless of where they go to school, succeeding or failing at the same rates regardless of whether they attend a high school with a long list of distinguished alumni. As usual, “school quality” just doesn’t have much effect, if any.

And isn’t that great? My position on these things is often represented as a doomer take, an embrace of “edunihilism” or similar, but I think that’s an odd way of looking at it. This particular research seems quite hopeful to me, albeit in a somewhat roundabout fashion: if we assume that attending the “best schools” is always going to be gated and rationed, if we’re realistic in assuming that elite education is always going to be reserved for the elite, then the fact that those schools just don’t make any particular educational difference is reason for optimism. It reminds me of the intense progressive criticism of the SAT and the notion that it perpetuates inequality; as some have argued based on personal experience, the test in fact can function as one of the only means through which poor students from nondescript public schools can distinguish themselves.

Exam schools fit perfectly into liberal visions of our education problems in that they suggest a world where everyone has equal ability but where effective teaching is sequestered behind a velvet rope. The conventional response to that vision of schooling is to try and remove the velvet rope and give the best teaching to everyone. Trouble is, if we’re going to take any lesson at all from the last twenty-five years or so of ed reform, it should be that just scaling up “what works”… doesn’t work. “Scaling excellence” has proven to be a false god. Looking at specific educational contexts that produce high-achieving students and then porting practices from those context to other places has failed again, and again, and again. You know my take on why: those schools and programs that appear to have “the good stuff” are so perceived because the students within those schools and programs have been nonrandomly selected into them and are systematically inclined to perform well. In many contexts, most notably charter schools, this systematically nonrandom nature is a dirty secret; with exam schools, it’s right out in the open. As I’ve said many times, having student applicants take an exam, selecting the best performers, and celebrating the fact that those students go on to academic excellence is like having a height minimum for your school and then bragging about how tall your students are. Of course the exam schools have impressive alumni! They only take the kids who are most likely to be impressive!

So: is all of this doomerism? Not if you understand that there’s no plausible world where access to schools perceived to be elite is universal. If we deal with reality, here on Planet Earth, where schools are sites of elitism and patronage and nepotism and socioeconomic inequality - well, the fact that schools have little influence on student outcomes is something of a win for real egalitarianism.

Of course, Jessica Winter writes for The New Yorker and I write a Substack, and as professional media circles the drain, the people within it cling more and more tightly to the prestige hierarchy. So I don’t think we’re going to see a correction anytime soon. Right now, after a half decade or so of a fallow period, the neoliberal ed reform movement is furiously fighting back, and because our system has no ability to provide for those who don’t have monetizable skills or abilities under late capitalism, they’re finding a receptive audience in elite publications. The new Substack the Argument is a reflection of this retrenchment; so are a mountain of recent opinion pieces like this comprehensively misleading one published today in the NYT. They’re throwing their back behind the effort to take us back to 2012 and the height of the failed bipartisan reform consensus. The question, as usual, is whether prestigious publications like The New Yorker are willing to be corrected when they get the facts wrong.

1. Regression discontinuity designs aren’t complicated, if you’re interested. Here’s a visual to help you understand. (Fake charts I made with fake data, btw.)

No Treatment Effect

Significant Treatment Effect

You already get it, right?

Typically, when we perform some sort of an experiment, we want to look at how a particular number responds to the treatment – how blood pressure reacts to a new drug, say, or how students improve on a reading test when they’re given a new kind of lesson. We want to make sure that the observed differences are really the product of the treatment and not some underlying difference in observed groups. That’s what random controlled trials are for. So we randomly assign subjects to test and control groups, look at what the different averages are for the two different groups, note the size of the effect, and determine whether it is statistically significant.

But sometimes we have real-world conditions that dictate that subjects get sorted into one group or another non-randomly. If we then look at how different groups perform after some treatment, we know that we’re potentially facing severe selection effects thanks to that non-random assignment. But consider if we have assignment based purely on some quantitative metric, with a cutoff score that sorts people into one group or another. (Like, for example, when students only became eligible for a gifted student program or selective high school if they score above a cut score on some test, as discussed in this post.) Here we have a non-random distribution that we can actually exploit for research purposes. A regression discontinuity design allows us to explore the impact of such a program because, so long as students aren’t able to impact their assignment beyond their score on that test, we can be confident that students just above or just below the cutoff score are very similar on the construct of interest.

Regression analyses will be run on all of the data, with subjects below and above the cut score combined but flagged into different groups. Researchers will then run statistical models to determine whether there is a difference between groups who receive the treatment and those who don’t. As you can see in the scatterplots above, a large effect will be readily apparent in how the data looks. (Of course, there are statistical thresholds; they’re not just eyeballing it.) In the above scenario, the X axis represents the score students received on the test, the cut score is 15, and the Y axis represents performance on some later educational metric, like GPA or college attendance rates or the SAT. In the top scatterplot, there is no meaningful effect from the gift students program, as the relationship between these two metrics is the same above and below the cut score. But in the bottom graph, there’s a significant jump at the cut score. Note that even after the intervention, the relationship is still linear – students who did better on the initial test do better on the later metric. But the scores of everyone have jumped right at the cut score.

There are, as you’d probably imagine, a number of potential pitfalls here, and assumption checks and quality controls are essential. All of the people tested would have to be able to be sorted into the gifted program solely on the basis of the test, the cutoff score has to be near the mean, and you need sufficient numbers to see the relationship on either side of the cut score, among other things. But if you have the right conditions, regression discontinuity design is a great way to get near-random experimental design quality in situations where you can’t do that for pragmatic or ethical reasons.