Tutoring Undermines the Validity of GPA More Than the Validity of SAT Scores

GPA, a metric with holes big enough to drive an ideology through

“everyone gets all A+s” is not really an exaggeration at many high schools

Though there have been a few green shoots, represented by some high-profile schools continuing to accept SAT results or choosing to accept them again, the reality is that the post-pandemic era still represents a major win for opponents of objective educational assessment. The anti-SAT people saw an opportunity in COVID-19 and they took it.

A lot of schools that once considered SAT scores no longer do; some are “test optional,” some simply won’t look at them. This is not something that has been imposed politically on the colleges and universities, but rather represents the schools taking advantage of shifting public mores to get what they’ve always wanted. One of the core ways people casually misunderstand elite universities lies in assuming that they always want to pursue the most academically accomplished students possible. In fact, though they’d never admit to it, the most exclusive schools often feel hamstrung by the need to let in nerds who lack the “it” factor and would prefer to admit the students whose presence can most improve the perceived desirability of their school. This is why an absolute dullard of an A-list celebrity can get into any college they want to. And, yes, it can be as ugly of a concept as you’d assume; the University of California’s dogged efforts to reduce the Asian population on its campuses, which has been an open secret in the system since I was in high school, stem from just this desire to have “the right kind” of student population. In general, admissions departments long for control, the right to admit anyone they want to. And since the anti-test movement looked at Covid and said “never let a good crisis go to waste,” these schools have had the justification they need to make rigor even less important in their determinations.

Of course, this points to just how incredibly naive the whole anti-SAT, anti-objectivity movement is about their own basic project: they seem to think that if you remove inflexible performance standards, colleges will let in a rainbow coalition of the marginalized and deserving. But of course competitive colleges are relentlessly self-interested institutions, and holding extraordinary skepticism towards the test-development industry and not towards colleges or their admissions departments is bizarre. For many years a group of Harvard faculty have pressured the university to publicly explain how it chooses who receives entry into the college thanks to diversity concerns; the school has declined to do so, and still holds its cards close to its vest following the hugely embarrassing Supreme Court case that revealed its systematic discrimination against Asian applicants. Why would they not want to reveal this information? I’m guessing it’s because, as suspected for decades, elite schools use diversity slots as just another opportunity to farm rich applicants. In general, the way the “fair admissions” people ignore the fickle and profoundly unmeritocratic self-interest of the institutions amazes me.

Today I want to take a look at one of the most common arguments against using the SAT or ACT or other entrance examination for college admissions: the claim that they can be gamed by rich students with expensive tutoring and are thus not fair. There are a variety of responses to that, the most important of which is that the best evidence continues to be that such tutoring is actually consistently ineffective. Most of the relevant studies demonstrate some modest gains, emphasis on the modest, and even then there’s an obvious confound in that just retaking the test tends to increase scores. This improvement is intuitively understandable, as there’s an advantage in experience and comfort, but the returns appear to be quickly diminishing if you take the test more than twice. You can complain that taking the SAT multiple times is itself expensive, but the College Board has offered fee waivers for most of its history, there are cities and states that offer assistance in some form or another, and there is the odd nonprofit out there that will help with registration fees. (I also would suggest turning to social media to crowdfund the $68 registration fee if you need to.) Of course, the anti-SAT crowd interprets the data differently and says that the tutoring does provide some advantage and thus the test is unfair and biased towards the rich. Tutoring works!, they insist, and rich people have more access to it, so you can’t fairly include the SAT in your application process.

Ah, but you see, now I gotcha.

I’m afraid that this logic falls apart, as an anti-SAT argument, when you realize that the removal of the SAT results in increased focus on grades, and many more American students get tutoring to address their grades, tutoring that is no doubt far more accessible and potentially more effective. Tutoring for K-12 students is ubiquitous, unregulated, and largely ad hoc, so I’m not super confident in any of the numbers that are out there. But there’s still no doubt that the number of students getting sustained tutoring for their grades dwarfs the number getting SAT tutoring. At least 5 million students currently receive “high-dosage” tutoring for K-12 schools, to use the current popular lingo for high-quality; the number receiving tutoring that does not meet the high-dosage standard is no doubt vastly higher. Compare this to the fewer than 2 million students who take the SAT each year total, very many of whom receive no tutoring (according to the anti-SAT crowd!) and you’ll see that tutoring to improve K-12 grades necessarily dwarfs tutoring for the SAT. Even if we wanted to restrict ourselves to high school students - and I don’t know why we would, given that K-8 tutoring would presumably carry over into high school performance - the number who receive tutoring for grades must vastly outstrip those who receive tutoring for the SAT.

And the research record for tutoring for grades is definitely more encouraging than that of tutoring for the SAT. I am, as you’d expect, skeptical of just how effective this tutoring is; it’s precisely the sort of magic bullet approach that the ed policy world relentlessly fixates on, and the kind of effort they’re engaged in usually entails putting wishing over thinking. (Optimism bias never leaves us.) But I will say that regular small-group tutoring has more empirical justification for optimism than any other intervention - class size reductions, pre-K, edtech, any of it. Do I want every kid to have access to tutoring? Definitely. And even if overblown, the research record suggests that tutoring for grades is more effective than tutoring to increase SAT scores, likely for reasons we’ll discuss below.

In other words

1. Removing the SAT from college admissions almost always results in increasing emphasis on GPA; most of the time, GPA soaks up the removed variance once attributable to SAT scores

2. The impact of tutoring on SAT results is frequently argued to be disqualifying, among critics of the test; tutoring is unequally distributed between students and does not reflect “real” ability, in this telling

3. But tutoring has far greater aggregate impact on GPA than on the SATs, given scope and efficacy differences, meaning that the problems with SAT tutoring are more deeply associated with the very metric that would replace the SAT!

There are still some misguided advocates who think that the SATs should be replaced not with more emphasis on GPA but by becoming more “holistic.” Happily, more and more people seem to understand why holistic admissions are a nightmare: all of the activities and clubs and interests that the most “holistically impressive” applicants take part in are deeply bound up with economic class, home stability, and parent availability. Going to Guatemala to build houses for the summer, fencing competitively rather than playing basketball, starting your own literary website - all that fancy shit is far more influenced by uncontrollable familial factors than SAT scores. The admissions essay is a good example of a criterion that I wouldn’t get rid of, but which we should never mistake as being free from income associations. There’s also this whole big conversation to be had about what people see as legitimate vs. illegitimate sources of ability; it’s not at all clear to me why getting SAT tutoring is a devious practice but sending your kid to a school you perceive to be better isn’t. But that’s a big chunk I’ll have to bite off some other time. For now I think it’s worth thinking about grades, and in particular the way that they’ve come to be seen as the noble admissions metric despite being race and class stratified.

In general, I find it powerfully difficult to understand how people have convinced themselves that grades are less subject to the corruptions of money and privilege, when every ounce of common sense suggests the opposite. Do you really think those parents paying $60K/year for private high school aren’t pressuring the principal and teachers in an effort to manipulate grades?

Grades are, to use a term often deployed by one of my grad school mentors, squishy. Their validity and reliability have a natural ceiling, thanks to a host of factors but most prominently the subjective influence of a given instructor. Grades still reflect overall ability pretty well, particularly at scale - and we know that because, somewhat ironically given current debates, they correlate strongly with more objective indicators like SAT tests.¹ Grades are used to cross-validate standardized tests all the time, and the results are encouraging if you believe that there is such a thing as better and worse in evaluating academic performance. For example, Grade 12 NAEP scores are useful predictors for how students will perform in college; 26 percent of students who performed at the Basic level, 58 percent of students who performed at the Proficient level, and 87 percent of students who performed at the NAEP Advanced level are considered college ready according to a National High School Longitudinal Study standard. This kind of research is the bedrock of assessment, and it demonstrates that grades are useful. But grades also have far more natural squish than tests that are built using psychometric theory and modern educational assessment techniques.

Since ~70% of students have SATs that are highly correlated with their high school GPA, you might wonder what the hullabaloo is about, at scale. But for me, a core part of the purpose of the test is to find diamonds in the rough, the portion of the 30% where the test score suggests someone who has strengths that are hidden by a low GPA. And this is why the anti-SAT argument feels so fundamentally toxic to me: it’s an argument for less information rather than more, literally, a call to throw out data. Literally no one who supports the value of SAT scores wants to do away with GPA. Why would anyone throw away useful information? Well….

The squishiness of grades, though directly contrary to the purpose of gathering the best information about students that we can, offers a great boon to those who oppose educational testing and promote blank slatism: it allows them to bend the world into alignment with their cheery fantasies. Grades are noisy and subject to a ton of confounds; high school grade inflation has caused the GPA signal to peak out at so many schools that it’s a useless indicator for sorting students within those schools; the distribution of GPAs between schools are so wildly inconsistent that competitive colleges go to remarkable lengths to learn how to make decent comparisons. (And a huge contributor to these problems is the influence of aggressive and well-resourced parents, precisely the kind of corrupting factor that progressives usually decry.) All of this is shitty, as a matter of choosing who gets into which college and as a matter of social science. But precisely because of all that wiggle, because of that squishiness, a certain kind of critic of quantitative educational assessment loves grades; they’re squishy enough to permit people to deny not just the outcomes of specific evaluations but the very concept of fair academic evaluation entirely. And that, ultimately, is the whole point of the campaign against the SAT, rejection of the very idea of any kind of objective evaluation of academic ability in college admissions and society writ large. Which, fine - if that’s how you feel, fine. But just advocate for that. Reject evidence-based admissions. Acting like grades are pure as the driven snow, as a matter of convenience for bashing the SAT, is just weird and disingenuous.

Critics of the SAT are fond of pointing out that the correlation between the SATs and family income is stronger than that between GPA and family income. Which is true. But then, this is merely a function of the fact that grades are a hideously noisy indicator, so noisy that in many real-world scenarios there’s a ceiling on the correlations you’re going to find with anything. But if you’re chasing a shitty indicator precisely because its shittiness increases the likelihood that a less-qualified candidate will be admitted based on it… what are we even doing here?

1. The .55-.65 r values typically found when correlating high school GPA with SAT scores might not seem that high, but it’s here that we must again remember how low correlations tend to be in education specifically and the social sciences generally. The r value for class rank and GPA in that study is .719/.735, despite the fact that class rank and GPA are both determined by the same variable. Real life is noisy.