One other factor is the progressive belief that expensive test prep is effective when the preponderance of the evidence says it adds little if any value. It's basically a scam and doesn't improve scores any more than a $20 test prep book that includes a few practice tests.
Yes, and even more frustrating from my perspective, people who complain about testing in this way tend to favor grades/GPA as an indicator - but the vast majority of dollars spent on tutoring are spend on improving grades, not test scores, and the research record suggests that tutoring is more effective than tutoring towards tests. It makes no sense.
My thought is that both the right and the left have a real problem with everyone not having the same level of ability. On the right they think is someone doesn't do well in school it's because they aren't working hard enough. On the left they think some social structure is thwarting their natural ability. The idea of an innate lack of ability just doesn't' fit into their world view.
You are dead right. But this in turn is because credentials, qualifications, bits of paper that say "I am schooled" are so dramatically important that people don't want to admit that some people just aren't going to meet the goals society has set to allow them into the middle class.
I saw an absolute nightmarish chart a few days ago, and I wish I could find it now, showing the median, top-10%, and top-1% personal income for people aged 31 to 40. For the top 10% and top percentile, there was a great, Matterhornian climb over the years, culminating in a very comfortable income. The median income remained practically in a straight line. People who believe in the current social structure are of course going to argue that, yes, everyone can be above-average, if only they'd pull themselves up by their bootstraps or if only schools had more funding (delete as left/right appropriate.)
Instead a society that had less wildly skewed incentives and recognized that a lack of booksmarts isn't a sentence to penury would be ideal here. (Even some - a minority, admittedly - on the Right are starting to come around to this. C.f. Donald Trump in 2016: "I love the poorly educated!") But as long as we fetishize degrees, and elite degrees at that, this argument will not end, despite the evidence. All must have prizes.
There's an interesting mix of components of what you describe as people 'fetishizing degrees'.
It would be nice/best/ideal if people could live comfortably _regardless_ of their credentials (e.g. degrees) – and I agree with that and I identify as (mostly/somewhat) 'libertarian'.
_But_ ... there are probably _some_ times (if strictly a MUCH smaller number than now) when it seems reasonable to 'insist' on credentials/degrees for some jobs/positions/offices, e.g. doctors, engineers.
But I definitely sympathize with everyone's anxiety regarding the status quo!
I mean, I think part of the SAT hate is coming from exactly the fact that you can't beat it by prepping. Modern parenting doctrine is built fundamentally around the idea that if you do the right things as a parent, and provide the right resources, you can ensure the success of your child. The flip side of believing that all disadvantages are structural is the reassuring belief that your own structural advantage is secure.
From this perspective, the LAST thing a modern bougie parent wants is a test that they can't throw resources at to trivialize for their child. So of course they're going to prefer a "holistic" assessment that lets them pad out their kid's resume.
There is also a faction that thinks grades and other rewards should be based on effort not results. If Alvin can learn a chapter's worth of Algebra in 45 min he should get a worse grade that Kevin who spent hours studying.
I think these idea also ties into the thesis of Freddie's book in that some people think the world actually works this way. It's totally fair that Alvin the electrical engineer makes 4x as much as the Walmart cashier - he worked harder.
I'm friends with a very smart, very honest guy who used to work for Kaplan. I told him, based on Freddie's assertions, that there's no good evidence that test-prep works. He swore up and down that he improved several of his tutees' SAT scores by a couple of hundred points. So here's my question:
Is it just that test prep doesn't work *on average*, but that there are some people for whom it works really well? (And, along these lines, are there some people whose performance it worsens?)
My understanding is that test prep works well enough. If those kids had bought any number of SAT prep books and did the work, their scores would have increased dramatically as well.
The question is does elaborate and expensive test prep work better than a cheap or in some cases free book/website? The answer is no as far as we can tell.
Right. On average the increase from test of any kind prep is fairly modest. Even simply taking the test again explains most of the difference. But presumably, if the guy from Kaplan was seeing dramatic increases, these stemmed from misunderstanding that would have been addressed in the cheaper test prep options. Or even more simply been revealed when reviewing their answers to the original practice questions.
Freddie, did you link to the research literature on this in one of your posts? (If you did, you needn't post it; I'll just do ctrl+f for "test prep" in your education posts.) I ask because I'm confused as to how the practitioners could be overly optimistic if they actually saw the score-increases.
I mean, anecdotes aren't data. People self report all kinds of things. That's why we have research, because those affirmations have proven unreliable in all manner of contexts.
Fair enough. I could tell him that, I guess. He's the kind of guy who would certainly be open to that possibility, which is why I trusted his assessment of what happened in the first place.
But I'll look for the literature! Maybe it's in your book?
This is a rare education issue I actually know anything about! I've worked in test prep (as a side job) for over 15 years. To use the most idiotic expression of our time, I know what my "lived experience" tells me. I've had too many students get into the law/medical/grad school of their dreams to think "it's a scam." Anyone who thinks that probably either has little experience with it or a bad personal one.
I also know enough to know that it's probably something that's exceedingly difficult to actually study. At the high school level - from what little I've seen - you have a high percentage of people who are doing test prep for a simple reason: their parents told them to. You end up with extremely low effort students doing the absolute minimum (if that) and who get about the same results as someone who does the minimum on anything that involves practice.
At the grad level you get the opposite: you're essentially self-selecting out the best and the brightest. I didn't take a test prep class for a simple reason: I didn't need to. So you're already dealing with the people who are least likely to do well. There's also serious issues in terms of effort here: I'd estimate that at most one third of all the students I've ever had have done all the work I've asked them to do.
That said, this isn't a defense of test prep, it is definitely NOT effective enough (largely for those reasons I've mentioned) to be something that has a significant impact. In my experience it seems to be incredibly valuable for a small subset of students: those who are hardworking but not autodidacts and have a a certain level of talent in whatever is being tested. For them, it's quite valuable. They weren't going to sit there and puzzle it out based off some "$20 test prep book that includes a few practice tests." They needed someone to push them to do their work. And they did very well. But for most students, they're missing one of those ingredients and it's a waste of time and money.
I can also say that when it comes down to it, the most frustrating part is that most of these students just aren't smart enough. At a certain point, I just can't explain to somehow how to do what they need to do, because no matter how much effort they put in they won't be smart enough to make the connections in their mind they need. You can't teach smart, you can only help people who are already smart but aren't using that correctly.
Test prep is, in the end, a product like any other: it's only going to be beneficial for a small percentage of the people who use it, but there's a profit motive in not acknowledging that, and a lot of customers are happy just having the appearance of having done something. It's like a yoga class or guitar lessons. It's dumb to call it a "scam" and it's dumb and dangerous to consider it enough of a factor that it's worth discounting the value of these tests.
As far as I know MCAT test review is valuable as it’s a subject matter test vs. an aptitude test. It’s sort of like BarBri and prepping for the bar exam.
This is very well put and probably explains in part why motivated Chinese students (others, too, but these are the ones I've seen do this) gain 200-400 points on the SAT after a cram class.
That's very interesting and I had never heard that before. I think that certainly plays into the idea that motivation has something to do with it, which makes sense because so much is practice based.
My second grader was on the borderline for qualifying for academically and intellectually gifted services at his public school based on testing (i think he was in like the 80th or 81st percentile). So they offered to rescreen him. Well after a final week of school
doing various end of year assessments and finishing up assignments (he got extremely bored with online school during Covid and getting him to finish his work was a task in itself) he then had a week of summer break before the rescreening. He sped through that in an hour and ranked in the 65th percentile.
I’m honestly not particularly concerned; I’m really pleased with our school and the teachers and I’m confident he’ll be challenged enough and then one day if he qualifies he’ll get the extra services. But motivation at this juncture was absolutely crucial. And both immeasurable and un-measurable—though his percentile drop in the span of a month was 16 points.
I used to believe this. But students in China do cram courses (2-4 weeks of ten to 16 hours a day) and raise their score on the SAT 200-400 points. Really, this happens. Chinese teens just look down with a very faint smile when you say this kind of stuff.
As someone who this happened to (I went from ~1700 to 2390 between middle and high school) I would qualify this a bit. From my experience, test prep is quite effective at raising performance up to a potential point for each person, past which it makes very little difference. That potential point is where you have a sense of how to use your time, how to answer questions out of order efficiently, how to exploit givens in question text to cross-reference your own uncertainties - the kind of general test-taking strategy which makes some people much better or worse at taking tests than their peers of roughly equal scholastic potential. The thing is that these skills are relatively quite easy to learn for students of all levels even from basic test prep material and do not necessitate multi-week intensive courses. On top of these strategic skills is basic familiarity with all the topics in the math section and a reasonable vocabulary, which prep courses can genuinely address, but only for students who are already underprepared - for students who already have that familiarity by regular performance in secondary education it won't significantly improve their ability. And for those underprepared students it's no longer a design flaw of the test - the test is genuinely measuring the kind of tertiary education preparedness it purports to measure! Which I think generally matches the position that Freddie and the linked literature takes: a large amount of the variability between your theoretical potential and your score on your first SAT comes from either (lack of) basic familiarity with the testing format, time usage strategy, and confidence, which is why a retake with no intervening intensive prep can often produce gains of several hundred points, or from genuine underpreparedness, which means the test is doing its job; this can be addressed by regular free/cheap test prep but not significantly better by expensive intensive prep. And some necessary prep that is accessible to students at all socioeconomic classes seems okay, because I don't think the SAT purports to be in the same category as things like the referenced Raven's Matrices which are by design not significantly improvable by any prep. It's got "Scholastic" in the name, after all. This of course skips over the genuine problem that knowledge of need/access to that low level prep is SES-correlated and unfair in a certain sense, but the core of the SAT fight doesn't really include that topic.
So I think the more qualified position is "test prep at the level available for cheap/free is helpful to all, but expensive intensive test prep over and above the level available from cheap/free test prep produces minimal improvement".
Finally I do agree that this is all a bit definitionally muddy. I think a big part of the population variance is which students are able to self-learn these "strategy" items and which students benefit from test prep teaching them these. It is clearly possible for students to perform to their theoretical potential without intensive test prep by just learning the "strategy" bits through practice tests and workbooks, but it's also clear that it's not universally possible. General ability at taking tests separate from knowledge of the subject matter (or the ability to learn such) is arguably a form of proxy IQ test, and there is a good example in the essay titled "Examsmanship and the Liberal Arts" which is a fun piece of reading I'd recommend regardless!
I feel really unclear on this because year in and year out I see students underperform on standardized tests, and a smaller number who over perform their daily show of understanding. I’m certainly biased but it doesn’t seem nearly as predictive as the other 179 days they’re with me.
And none of these students go to fourth grade and suddenly start performing like the standardized test said they would, they continue to be odd cases of luck or test anxiety.
The evidence is very clear, per the post. Perhaps what you're discussing is content validity? If the students' daily show of understanding is low, and their test performance is high, perhaps the content of lessons and the measurement of tests are not aligned. It is also possible that students who underperform on tests are able to engage with a teacher in a positive way but not actually master the material. (Office analogy: the yes-man versus the technocrat.)
I graduated 9th from the bottom of my high school class with a 1.2 GPA and got a 32 on my ACT. When the results came back I got called into the counselors office asked repeatedly if I cheated and I believe they had the proctor come in a verify it was really me that took the test. My sense then was it was mostly a reading test, especially when the answer is right there in front of you it’s just a matter of answering what they’re asking. What was weird for me and I still try to understand, is that while I did well in college, especially my first couple jobs after college were a repeat of high school where I couldn’t get the hang of it and performed terribly. I guess different situations cause underperformance. For different people.
"who over perform their daily show of understanding."
I think what you're talking about is IQ vs. conscientiousness/diligence. You can have a very smart kid who is not really into third grade. On the other hand to can have kids who like getting good grades, like doing the work assigned to them, like doing the extra credit, etc. I think that might be what you're thinking of when you say, "who over perform their daily show of understanding."
N is the magic number. There are a lot of system-wide trends that are nearly impossible to notice at the classroom level, especially when your school is probably feeding you a similar set of kids year in and year out.
Maybe I read this too fast, but I have a question. Let's see if I can try to make it clear. I never really put any thought to these tests since I had to take them, but I totally believe your claims about what people say about these tests (though there is no support, at least here, for the 'suddenly' claim at the outset). And I think your case here about their validity, reliability, etc is pretty straightforward.
What I don’t understand is what you think their actual current utility is. Say we find some group-level inequalities with these tests, is our society set up to do anything about that? It kinda reminds me of that MMT stuff, which for all its (for all I know) precision and correctness, seems to forget how power works in our society. I think the real task before one in favor of these tests is to show how they could be used as a lever to create a more just society. If we can’t convincingly make that claim, why are we doing them? Just to know more true things? Things not worth doing are not worth doing well.
Are these tests intended to be part of a larger sorting machine that creates fair job outcomes based on ability? Like some free-market style means of getting the right people into the right sort of jobs? That seems false to me. And even if it were true, it’s not clear to me that we would want such a thing. Do we want an elaborate apparatus as part of a technocratic attempt to optimize human potential? I certainly don’t want to live in such a society. But maybe you do and I think that case would be harder to make, but I’d love to hear it!
Certain jobs require a high level of cognitive ability - oncologist, software developer, structural engineer, geneticist. We need a system to sort people on to the educational paths where they are most likely to succeed. It doesn't do any good to pretend that anyone can do anything if they just put their mind to it.
Where does this worry about success or failure come from? Seems like a defensive position in a society that functions as a meat-grinder for failure. Maybe in light of the horrors of capitalism, we're trying to come up with a way to shield people from the grinder, I wish we were more upfront about that. Scholastic tests as palliative care in a world where we aren't allowed to turn off the meat-grinder
Idle curiosity: Do these tests make such fine grain distinctions between who should be an oncologist versus a geneticist? Do we aspire to such granularity?
This doesn't seem like a capitalist problem, this seems like an efficient-distribution-of-resources problem (I'd imagine a centrally-planned economic system like communism would do *more* sorting). If Tommy doesn't have the intellectual capability to be a doctor, then it seems only prudent to steer him in directions where his talents lie.
Exactly, the Soviet Union certainly didn't have a system where everyone who wanted to go to college could go to college. If they thought you were smart enough to benefit from college you could go. If they didn't think you were smart enough you'd be sent for training to be a factory worker or mechanic or whatever they thought you'd be good at.
Layered on top of this, though, was Party loyalty: if you were perceived as insufficiently loyal, a la Havel's greengrocer, your kids might get shut off from college, just as you might get demoted. I personally know a number of FSU scientists who fled to the West as soon as they could who tell many horrifying stories about the Soviet system.
Self-identified Marxists like Freddie really need to tell us how they plan to avoid these pitfalls of communism if they want the rest of us to take them seriously.
yeah, I agree. By the same token, for real efficiency, be it in a capitalist or communist system, there needs to be a way to prevent 'wasted' talent. In capitalism, well, smart people avoid low-skill jobs because life sucks if you have one (broadly speaking). In communism, you need some other means of coercing smart people not to take low-skill jobs (force? dunno)
Either way, these seem to presuppose the necessity of sorting. Both seem to require a form of coercion for efficiency's sake. Both rub me the wrong way as someone more skeptical of hierarchy, however technocratic
"In communism, you need some other means of coercing smart people not to take low-skill jobs"
From what my Russian friend said as a college student they gave you list of jobs and you ranked them. Then those doing the hiring would rank the students and the highest ranked one you matched for would be your job. I believe a similar system existed for those on a vocational track. But in either case your only options were what was on the list they gave you.
I don't think test today measure at that level of granularity. But it would be nice if they did. I think many people are held back because there are jobs they'd be good at that they didn't pursue because they didn't know they existed.
thanks for engaging, this stuff is really helping me with my own thinking, which I guess I'm sort of exploring here. hopefully not too much of a non sequitur:
It seems that we have a lot of discussion about 1) how to match people up with the right jobs, but there's no discussion about 2) how we determine what jobs we have, what jobs need to be done, etc. which seems so intimately related to the first question that one hardly makes sense without the other.
this reminds me of FDB's talk of why there's so much left/liberal focus on college-related concerns: they have no power anywhere else. Maybe that's the same here, we can squabble over school stuff qua job-prep because there's some modicum of popular control over that sphere. But when it comes to questions like 2) we have much less say. dunno, interesting stuff
I'm well aware that this is the current ideology. I just think discussion of schools and testing etc needs to include discussion of what we think schooling is for, which I think is lacking in freddie's post.
I think you're entirely correct that there is some tension between freddie's professed beliefs with respect to economics and his apparent beliefs with respect to the utility of testing. I imagine he thinks that our current way of determining "what jobs we have, what jobs need to be done, etc." is bad. What I'd really like to hear from him is how testing and all that supports/mitigates/relates to our current job set-up
I'd add to this in a few ways. One is to note that the liberal consensus was decidedly pro-testing in the 90s and 2000s, but that the way in which large scale state testing rolled out under NCLB was modeled on Texas - which, FDB pointed out, may not have been the success story it claimed to be.
Another is that the testing became a tool in the hands of various reformers. Clearly it's the teachers unions who are to blame, I mean, look at these test scores! Enter Scott Walker, Chris Christie, Mike Bloomberg. Right along side of those reformers came expanded voucher programs, more charter schools, and pushes against collective bargaining. So, while the testing was finding various inequalities - something that liberals thought would provide evidence for locating and improving instructional practices, thus outcomes - the policies that seemed to result from testing's evidence base threatened teachers and schools systems' historic power as well as local control (charters and vouchers being very much outside of local board control in most places). So the liberal intent behind testing was perceived as being turned to somewhat libertarian and market-oriented purposes popular until recently on the right.
Finally, testing came with a veneer of corporate control. As FDB noted in the post kicking off edu week, Gates brought Common Core into being almost single handedly and with surprisingly little resistance (imo this is because, even then, lots of educators were still feeling like testing revealed a problem, so curriculum goals could solve it). The other big thing the Gates Foundation did was roll out curriculum aligned to Common Core in districts around the country and try to implement evidence-based instruction to, well, end educational inequality. It didn't work and Gates more or less lit money on fire for about a decade. But teachers and districts felt burned (especially when the CSAIL report said teachers and districts were the reason Gates' project failed).
So, at least coming from the perspective of the schools and their teachers/aides/admins, nearly two decades of constant changes, instability, and ruthless pay/benefits/job security cuts seemed to be the results of these tests. And the supposed upside, the renaissance of data-driven best-practices and all the other hyphenated buzzwords failed to materialize. Anyone remember teachers being forced to kneel under desks because their test scores didn't go up and then getting fired when the scores didn't go up again? How about the ATL scantron parties where district leadership just falsified the scores?
The tests themselves are not to blame, obviously. And they are not some warped instrument of white supremacy wielding western colonialist mathematics against black and brown bodies (to borrow the parlance). But they were the justification that schools heard again and again for what turned out to be a whole package of reforms that didn't work but made life in many schools much worse. This was especially true in the schools that were the lowest performers.
So you can see why testing gets lumped in with all the bad stuff. Justified or not, it comes from somewhere and it's not only unpopular because of what the testing reveals but also because of what the testing was used for.
Following yesterday's post, is there a chance that some of this anti-testing spittle on the liberal left is partly redirected anxiety over high-end college admissions? Or even bad-faith attempts to transition towards "softer" metrics like GPA, club membership, and letters of rec because they're easier for mediocre rich kids to game than the SAT?
I have this hunch, purely anecdotal, that the most vociferous anti-testers are rich white parents who publicly scream about helping poor black kids but privately worry more about their bumbling failson with SAT scores far below the median for incoming freshmen at their elite alma mater.
I think there's some of that. I also think that there's a lot of genuinely well-meaning anxiety about race going on here, and there's a quiet belief among a lot of white progressives that (despite their BLM rhetoric) real progress for Black America is probably impossible. So they agitate for lowering standards and eliminating indicators because they do want best for Black children but secretly don't think they're capable of meeting contemporary standards.
For my part, I don't think that there are any likely short-term major positive jumps in Black academic performance coming. But I do think that there are major improvements to Black quality of life and political power that could be achieved with strategic economic policy. But a lot of liberals have kind of given up on the possibility of large-scale economic policy change, or don't want it because they are economically comfortable themselves. I realize that this is a lot of armchair psychologizing.
That makes sense, thanks! Elite reproduction anxiety is a hobbyhorse or mine, so I tend to see it everywhere.
I agree with you on the impossibility of short-term jumps, too. It's depressing, looking at the data, to think how little we can do to meaningfully change outcomes for the class of 2024, but overwhelming to think about the dismal chances of getting the slow-and-steady infrastructure changes we need *right now* to give the class of 2039 a fair shot.
Speaking of anxiety among elites I have to think about some of the anecdotal hostility I've witnessed directed towards Asian students: namely that tiger parenting results in humorless drones who may be good at test taking but are completely incapable of creative thinking. I have to wonder if it's any coincidence that there seems to be a significant overlap with the criteria that Harvard uses to discriminate against Asian applicants (personality traits such as likeability, personal courage, etc.).
"Few people would doubt, however, that any assessment regime should involve the kind of basic numeracy and literacy testing common to virtually all K-12 standardized exams, the NAEP, and entrance exams like the SAT and similar."
I was a child when first introduced to the criticism that standardized tests are biased. It was in an episode of "Good Times" (DyNoMiiiiiiite!), which we watched as foreigners unfamiliar with life in America. The youngest son had taken a standardized test and done poorly, and was frustrated by the test's unfairness. The example he gave was that he was asked to pick the correct pairing with "cup." The answer was "saucer," but he had answered "table," because in his household, the cup went on the table, and they didn't own saucers.
This scene from a sitcom eons ago stuck in my head. It was the first time I'd encountered the novel (to me) "systemic bias" concept, though I was somewhat unimpressed by the example given to demonstrate it. I didn't think a 10 or 11 year old needed to have grown up with cups and saucers to know of their existence, and I was skeptical that such a question would even be on tests, at least in other than a multiple-choice format, eliminating the excuse for picking table over saucer. But this example stuck with me because, though it was silly, it made its point: standardized tests should not ask questions that test the knowledge, culture, or personal experiences predominantly of a particular race, culture, or income level.
We have gone from this undeniable premise, to arguments that 1 + 1 can equal 3, if the test taker thought the equation represented animal breeding, which apparently only non-whites unburdened by the oppressive rigors of math would do. The fictional "cup," "saucer," and "table" example was at least concrete. The criticisms today that standardized test are racially stratified don't offer concrete examples. The proof supporting this conclusion is the test performance statistics themselves, not individual questions. It would help if there was some explanation on why a particular test question is racially biased. I'm open to that possibility and I'd like to know what bias creeps into testing. Maybe there are already gobs of papers demonstrating this point. I just don't know what the substantive arguments are, other than the predictable ranking of test results by race.
One well-defined meaning of "bias" is that it's the difference between groups in the group-averaged test-predicted performance compared to the actual performance difference. This definition only has meaning if there is something that the test is being used to predict, e.g. completion of a degree program. I believe that the standard tests used for school-related purposes have little or no demographic bias in this sense.
I stumbled on some relevant evidence in the process of helping my wife assess the effects of using only online lectures for her intro stats course. Since the students couldn't be randomized to alternative treatments, we had to adjust for predictive covariates. Math ACT turned out to be the most predictive of objective exam scores, but high school GPA added some incremental predictive power. The typical math ACT of the student's major also added significantly to those, which surprised us a bit. For homework score, HSGPA beat math ACT, just as you'd expect for a more effort-weighted results. Overall, it's pretty much just what common sense would say.
BTW, on another topic, the study outcome was that in this course purely online lectures did at least as well as in-person +online. And yes, we did everything we could to adjust for confounders using four different statistical techniques, which agreed very closely. We don't think that important other confounders are lurking around.
Whoops, I think that was typical total ACT of the major. For the individual students we had separate components, and only math was predictive. SATs were converted t ACT equivalents.
Does anyone have a link to actual SAT questions that have been flagged as being biased? I know that there have been great efforts within the last 30 years+ to eliminate biased questions.
BTW, a friend who works as admissions counselor told me that in his high school, various teachers compared SATs given in the 1960s and 1970s to more recent (last 20 years) SATs. My friend said that the test from the 60s and 70s were far more difficult than ones recently given.
Such great writing: lucid, engaging, familiar, and - as usual - provocative in an equal opportunity way. As one of those “conservative[s] who’s certain that the minimum wage kills jobs because his understanding of Econ 101 tells him so” I have to ask, even though it’s somewhat off-topic, am I mistaken? I’m willing to be wrong, but I’d like to see and understand your take on the evidence we have. I just recently subscribed to your substack so I might have missed your past writing on the topic. But if you haven’t yet written about it, would you at some point? It’s an important and consequential subject.
You might be interested in Matt Yglesias's writing on this subject, since in many ways he is the ultimate reasonable-center-left representative, with pragmatic essays about things like minimum wage mostly untainted by political wishful thinking. Not saying that he's right about everything, but he does present genuinely reasoned arguments instead of ultra-progressive propaganda.
UC (and indeed all competitive-admission colleges) have been discriminating against asians for decades, they didn't need to toss the SAT to do that.
One other factor is the progressive belief that expensive test prep is effective when the preponderance of the evidence says it adds little if any value. It's basically a scam and doesn't improve scores any more than a $20 test prep book that includes a few practice tests.
Yes, and even more frustrating from my perspective, people who complain about testing in this way tend to favor grades/GPA as an indicator - but the vast majority of dollars spent on tutoring are spend on improving grades, not test scores, and the research record suggests that tutoring is more effective than tutoring towards tests. It makes no sense.
My thought is that both the right and the left have a real problem with everyone not having the same level of ability. On the right they think is someone doesn't do well in school it's because they aren't working hard enough. On the left they think some social structure is thwarting their natural ability. The idea of an innate lack of ability just doesn't' fit into their world view.
You are dead right. But this in turn is because credentials, qualifications, bits of paper that say "I am schooled" are so dramatically important that people don't want to admit that some people just aren't going to meet the goals society has set to allow them into the middle class.
I saw an absolute nightmarish chart a few days ago, and I wish I could find it now, showing the median, top-10%, and top-1% personal income for people aged 31 to 40. For the top 10% and top percentile, there was a great, Matterhornian climb over the years, culminating in a very comfortable income. The median income remained practically in a straight line. People who believe in the current social structure are of course going to argue that, yes, everyone can be above-average, if only they'd pull themselves up by their bootstraps or if only schools had more funding (delete as left/right appropriate.)
Instead a society that had less wildly skewed incentives and recognized that a lack of booksmarts isn't a sentence to penury would be ideal here. (Even some - a minority, admittedly - on the Right are starting to come around to this. C.f. Donald Trump in 2016: "I love the poorly educated!") But as long as we fetishize degrees, and elite degrees at that, this argument will not end, despite the evidence. All must have prizes.
There's an interesting mix of components of what you describe as people 'fetishizing degrees'.
It would be nice/best/ideal if people could live comfortably _regardless_ of their credentials (e.g. degrees) – and I agree with that and I identify as (mostly/somewhat) 'libertarian'.
_But_ ... there are probably _some_ times (if strictly a MUCH smaller number than now) when it seems reasonable to 'insist' on credentials/degrees for some jobs/positions/offices, e.g. doctors, engineers.
But I definitely sympathize with everyone's anxiety regarding the status quo!
I mean, I think part of the SAT hate is coming from exactly the fact that you can't beat it by prepping. Modern parenting doctrine is built fundamentally around the idea that if you do the right things as a parent, and provide the right resources, you can ensure the success of your child. The flip side of believing that all disadvantages are structural is the reassuring belief that your own structural advantage is secure.
From this perspective, the LAST thing a modern bougie parent wants is a test that they can't throw resources at to trivialize for their child. So of course they're going to prefer a "holistic" assessment that lets them pad out their kid's resume.
There is also a faction that thinks grades and other rewards should be based on effort not results. If Alvin can learn a chapter's worth of Algebra in 45 min he should get a worse grade that Kevin who spent hours studying.
I think these idea also ties into the thesis of Freddie's book in that some people think the world actually works this way. It's totally fair that Alvin the electrical engineer makes 4x as much as the Walmart cashier - he worked harder.
I'm friends with a very smart, very honest guy who used to work for Kaplan. I told him, based on Freddie's assertions, that there's no good evidence that test-prep works. He swore up and down that he improved several of his tutees' SAT scores by a couple of hundred points. So here's my question:
Is it just that test prep doesn't work *on average*, but that there are some people for whom it works really well? (And, along these lines, are there some people whose performance it worsens?)
My understanding is that test prep works well enough. If those kids had bought any number of SAT prep books and did the work, their scores would have increased dramatically as well.
The question is does elaborate and expensive test prep work better than a cheap or in some cases free book/website? The answer is no as far as we can tell.
The research literature on test prep is in fact far more pessimistic about its efficacy than practitioner assumptions.
Right. On average the increase from test of any kind prep is fairly modest. Even simply taking the test again explains most of the difference. But presumably, if the guy from Kaplan was seeing dramatic increases, these stemmed from misunderstanding that would have been addressed in the cheaper test prep options. Or even more simply been revealed when reviewing their answers to the original practice questions.
Freddie, did you link to the research literature on this in one of your posts? (If you did, you needn't post it; I'll just do ctrl+f for "test prep" in your education posts.) I ask because I'm confused as to how the practitioners could be overly optimistic if they actually saw the score-increases.
I mean, anecdotes aren't data. People self report all kinds of things. That's why we have research, because those affirmations have proven unreliable in all manner of contexts.
Fair enough. I could tell him that, I guess. He's the kind of guy who would certainly be open to that possibility, which is why I trusted his assessment of what happened in the first place.
But I'll look for the literature! Maybe it's in your book?
This is a rare education issue I actually know anything about! I've worked in test prep (as a side job) for over 15 years. To use the most idiotic expression of our time, I know what my "lived experience" tells me. I've had too many students get into the law/medical/grad school of their dreams to think "it's a scam." Anyone who thinks that probably either has little experience with it or a bad personal one.
I also know enough to know that it's probably something that's exceedingly difficult to actually study. At the high school level - from what little I've seen - you have a high percentage of people who are doing test prep for a simple reason: their parents told them to. You end up with extremely low effort students doing the absolute minimum (if that) and who get about the same results as someone who does the minimum on anything that involves practice.
At the grad level you get the opposite: you're essentially self-selecting out the best and the brightest. I didn't take a test prep class for a simple reason: I didn't need to. So you're already dealing with the people who are least likely to do well. There's also serious issues in terms of effort here: I'd estimate that at most one third of all the students I've ever had have done all the work I've asked them to do.
That said, this isn't a defense of test prep, it is definitely NOT effective enough (largely for those reasons I've mentioned) to be something that has a significant impact. In my experience it seems to be incredibly valuable for a small subset of students: those who are hardworking but not autodidacts and have a a certain level of talent in whatever is being tested. For them, it's quite valuable. They weren't going to sit there and puzzle it out based off some "$20 test prep book that includes a few practice tests." They needed someone to push them to do their work. And they did very well. But for most students, they're missing one of those ingredients and it's a waste of time and money.
I can also say that when it comes down to it, the most frustrating part is that most of these students just aren't smart enough. At a certain point, I just can't explain to somehow how to do what they need to do, because no matter how much effort they put in they won't be smart enough to make the connections in their mind they need. You can't teach smart, you can only help people who are already smart but aren't using that correctly.
Test prep is, in the end, a product like any other: it's only going to be beneficial for a small percentage of the people who use it, but there's a profit motive in not acknowledging that, and a lot of customers are happy just having the appearance of having done something. It's like a yoga class or guitar lessons. It's dumb to call it a "scam" and it's dumb and dangerous to consider it enough of a factor that it's worth discounting the value of these tests.
“ law/medical/grad”
As far as I know MCAT test review is valuable as it’s a subject matter test vs. an aptitude test. It’s sort of like BarBri and prepping for the bar exam.
This is very well put and probably explains in part why motivated Chinese students (others, too, but these are the ones I've seen do this) gain 200-400 points on the SAT after a cram class.
You often get 200 point gains from just taking the test a second time.
That's very interesting and I had never heard that before. I think that certainly plays into the idea that motivation has something to do with it, which makes sense because so much is practice based.
My second grader was on the borderline for qualifying for academically and intellectually gifted services at his public school based on testing (i think he was in like the 80th or 81st percentile). So they offered to rescreen him. Well after a final week of school
doing various end of year assessments and finishing up assignments (he got extremely bored with online school during Covid and getting him to finish his work was a task in itself) he then had a week of summer break before the rescreening. He sped through that in an hour and ranked in the 65th percentile.
I’m honestly not particularly concerned; I’m really pleased with our school and the teachers and I’m confident he’ll be challenged enough and then one day if he qualifies he’ll get the extra services. But motivation at this juncture was absolutely crucial. And both immeasurable and un-measurable—though his percentile drop in the span of a month was 16 points.
I used to believe this. But students in China do cram courses (2-4 weeks of ten to 16 hours a day) and raise their score on the SAT 200-400 points. Really, this happens. Chinese teens just look down with a very faint smile when you say this kind of stuff.
As someone who this happened to (I went from ~1700 to 2390 between middle and high school) I would qualify this a bit. From my experience, test prep is quite effective at raising performance up to a potential point for each person, past which it makes very little difference. That potential point is where you have a sense of how to use your time, how to answer questions out of order efficiently, how to exploit givens in question text to cross-reference your own uncertainties - the kind of general test-taking strategy which makes some people much better or worse at taking tests than their peers of roughly equal scholastic potential. The thing is that these skills are relatively quite easy to learn for students of all levels even from basic test prep material and do not necessitate multi-week intensive courses. On top of these strategic skills is basic familiarity with all the topics in the math section and a reasonable vocabulary, which prep courses can genuinely address, but only for students who are already underprepared - for students who already have that familiarity by regular performance in secondary education it won't significantly improve their ability. And for those underprepared students it's no longer a design flaw of the test - the test is genuinely measuring the kind of tertiary education preparedness it purports to measure! Which I think generally matches the position that Freddie and the linked literature takes: a large amount of the variability between your theoretical potential and your score on your first SAT comes from either (lack of) basic familiarity with the testing format, time usage strategy, and confidence, which is why a retake with no intervening intensive prep can often produce gains of several hundred points, or from genuine underpreparedness, which means the test is doing its job; this can be addressed by regular free/cheap test prep but not significantly better by expensive intensive prep. And some necessary prep that is accessible to students at all socioeconomic classes seems okay, because I don't think the SAT purports to be in the same category as things like the referenced Raven's Matrices which are by design not significantly improvable by any prep. It's got "Scholastic" in the name, after all. This of course skips over the genuine problem that knowledge of need/access to that low level prep is SES-correlated and unfair in a certain sense, but the core of the SAT fight doesn't really include that topic.
So I think the more qualified position is "test prep at the level available for cheap/free is helpful to all, but expensive intensive test prep over and above the level available from cheap/free test prep produces minimal improvement".
Finally I do agree that this is all a bit definitionally muddy. I think a big part of the population variance is which students are able to self-learn these "strategy" items and which students benefit from test prep teaching them these. It is clearly possible for students to perform to their theoretical potential without intensive test prep by just learning the "strategy" bits through practice tests and workbooks, but it's also clear that it's not universally possible. General ability at taking tests separate from knowledge of the subject matter (or the ability to learn such) is arguably a form of proxy IQ test, and there is a good example in the essay titled "Examsmanship and the Liberal Arts" which is a fun piece of reading I'd recommend regardless!
I feel really unclear on this because year in and year out I see students underperform on standardized tests, and a smaller number who over perform their daily show of understanding. I’m certainly biased but it doesn’t seem nearly as predictive as the other 179 days they’re with me.
And none of these students go to fourth grade and suddenly start performing like the standardized test said they would, they continue to be odd cases of luck or test anxiety.
The evidence is very clear, per the post. Perhaps what you're discussing is content validity? If the students' daily show of understanding is low, and their test performance is high, perhaps the content of lessons and the measurement of tests are not aligned. It is also possible that students who underperform on tests are able to engage with a teacher in a positive way but not actually master the material. (Office analogy: the yes-man versus the technocrat.)
I graduated 9th from the bottom of my high school class with a 1.2 GPA and got a 32 on my ACT. When the results came back I got called into the counselors office asked repeatedly if I cheated and I believe they had the proctor come in a verify it was really me that took the test. My sense then was it was mostly a reading test, especially when the answer is right there in front of you it’s just a matter of answering what they’re asking. What was weird for me and I still try to understand, is that while I did well in college, especially my first couple jobs after college were a repeat of high school where I couldn’t get the hang of it and performed terribly. I guess different situations cause underperformance. For different people.
"who over perform their daily show of understanding."
I think what you're talking about is IQ vs. conscientiousness/diligence. You can have a very smart kid who is not really into third grade. On the other hand to can have kids who like getting good grades, like doing the work assigned to them, like doing the extra credit, etc. I think that might be what you're thinking of when you say, "who over perform their daily show of understanding."
N is the magic number. There are a lot of system-wide trends that are nearly impossible to notice at the classroom level, especially when your school is probably feeding you a similar set of kids year in and year out.
Maybe I read this too fast, but I have a question. Let's see if I can try to make it clear. I never really put any thought to these tests since I had to take them, but I totally believe your claims about what people say about these tests (though there is no support, at least here, for the 'suddenly' claim at the outset). And I think your case here about their validity, reliability, etc is pretty straightforward.
What I don’t understand is what you think their actual current utility is. Say we find some group-level inequalities with these tests, is our society set up to do anything about that? It kinda reminds me of that MMT stuff, which for all its (for all I know) precision and correctness, seems to forget how power works in our society. I think the real task before one in favor of these tests is to show how they could be used as a lever to create a more just society. If we can’t convincingly make that claim, why are we doing them? Just to know more true things? Things not worth doing are not worth doing well.
Are these tests intended to be part of a larger sorting machine that creates fair job outcomes based on ability? Like some free-market style means of getting the right people into the right sort of jobs? That seems false to me. And even if it were true, it’s not clear to me that we would want such a thing. Do we want an elaborate apparatus as part of a technocratic attempt to optimize human potential? I certainly don’t want to live in such a society. But maybe you do and I think that case would be harder to make, but I’d love to hear it!
Certain jobs require a high level of cognitive ability - oncologist, software developer, structural engineer, geneticist. We need a system to sort people on to the educational paths where they are most likely to succeed. It doesn't do any good to pretend that anyone can do anything if they just put their mind to it.
Where does this worry about success or failure come from? Seems like a defensive position in a society that functions as a meat-grinder for failure. Maybe in light of the horrors of capitalism, we're trying to come up with a way to shield people from the grinder, I wish we were more upfront about that. Scholastic tests as palliative care in a world where we aren't allowed to turn off the meat-grinder
Idle curiosity: Do these tests make such fine grain distinctions between who should be an oncologist versus a geneticist? Do we aspire to such granularity?
This doesn't seem like a capitalist problem, this seems like an efficient-distribution-of-resources problem (I'd imagine a centrally-planned economic system like communism would do *more* sorting). If Tommy doesn't have the intellectual capability to be a doctor, then it seems only prudent to steer him in directions where his talents lie.
Exactly, the Soviet Union certainly didn't have a system where everyone who wanted to go to college could go to college. If they thought you were smart enough to benefit from college you could go. If they didn't think you were smart enough you'd be sent for training to be a factory worker or mechanic or whatever they thought you'd be good at.
Layered on top of this, though, was Party loyalty: if you were perceived as insufficiently loyal, a la Havel's greengrocer, your kids might get shut off from college, just as you might get demoted. I personally know a number of FSU scientists who fled to the West as soon as they could who tell many horrifying stories about the Soviet system.
Self-identified Marxists like Freddie really need to tell us how they plan to avoid these pitfalls of communism if they want the rest of us to take them seriously.
What's that also the case with repressive right wing regimes?
yeah, I agree. By the same token, for real efficiency, be it in a capitalist or communist system, there needs to be a way to prevent 'wasted' talent. In capitalism, well, smart people avoid low-skill jobs because life sucks if you have one (broadly speaking). In communism, you need some other means of coercing smart people not to take low-skill jobs (force? dunno)
Either way, these seem to presuppose the necessity of sorting. Both seem to require a form of coercion for efficiency's sake. Both rub me the wrong way as someone more skeptical of hierarchy, however technocratic
"In communism, you need some other means of coercing smart people not to take low-skill jobs"
From what my Russian friend said as a college student they gave you list of jobs and you ranked them. Then those doing the hiring would rank the students and the highest ranked one you matched for would be your job. I believe a similar system existed for those on a vocational track. But in either case your only options were what was on the list they gave you.
"Do we aspire to such granularity?"
I don't think test today measure at that level of granularity. But it would be nice if they did. I think many people are held back because there are jobs they'd be good at that they didn't pursue because they didn't know they existed.
thanks for engaging, this stuff is really helping me with my own thinking, which I guess I'm sort of exploring here. hopefully not too much of a non sequitur:
It seems that we have a lot of discussion about 1) how to match people up with the right jobs, but there's no discussion about 2) how we determine what jobs we have, what jobs need to be done, etc. which seems so intimately related to the first question that one hardly makes sense without the other.
this reminds me of FDB's talk of why there's so much left/liberal focus on college-related concerns: they have no power anywhere else. Maybe that's the same here, we can squabble over school stuff qua job-prep because there's some modicum of popular control over that sphere. But when it comes to questions like 2) we have much less say. dunno, interesting stuff
Markets determine "what jobs we have, what jobs need to be done, etc."
https://www.polk.k12.ga.us/userfiles/555/Classes/180285/Naked%20Econ%20Chapter%201.pdf
I'm well aware that this is the current ideology. I just think discussion of schools and testing etc needs to include discussion of what we think schooling is for, which I think is lacking in freddie's post.
I think you're entirely correct that there is some tension between freddie's professed beliefs with respect to economics and his apparent beliefs with respect to the utility of testing. I imagine he thinks that our current way of determining "what jobs we have, what jobs need to be done, etc." is bad. What I'd really like to hear from him is how testing and all that supports/mitigates/relates to our current job set-up
I'd add to this in a few ways. One is to note that the liberal consensus was decidedly pro-testing in the 90s and 2000s, but that the way in which large scale state testing rolled out under NCLB was modeled on Texas - which, FDB pointed out, may not have been the success story it claimed to be.
Another is that the testing became a tool in the hands of various reformers. Clearly it's the teachers unions who are to blame, I mean, look at these test scores! Enter Scott Walker, Chris Christie, Mike Bloomberg. Right along side of those reformers came expanded voucher programs, more charter schools, and pushes against collective bargaining. So, while the testing was finding various inequalities - something that liberals thought would provide evidence for locating and improving instructional practices, thus outcomes - the policies that seemed to result from testing's evidence base threatened teachers and schools systems' historic power as well as local control (charters and vouchers being very much outside of local board control in most places). So the liberal intent behind testing was perceived as being turned to somewhat libertarian and market-oriented purposes popular until recently on the right.
Finally, testing came with a veneer of corporate control. As FDB noted in the post kicking off edu week, Gates brought Common Core into being almost single handedly and with surprisingly little resistance (imo this is because, even then, lots of educators were still feeling like testing revealed a problem, so curriculum goals could solve it). The other big thing the Gates Foundation did was roll out curriculum aligned to Common Core in districts around the country and try to implement evidence-based instruction to, well, end educational inequality. It didn't work and Gates more or less lit money on fire for about a decade. But teachers and districts felt burned (especially when the CSAIL report said teachers and districts were the reason Gates' project failed).
So, at least coming from the perspective of the schools and their teachers/aides/admins, nearly two decades of constant changes, instability, and ruthless pay/benefits/job security cuts seemed to be the results of these tests. And the supposed upside, the renaissance of data-driven best-practices and all the other hyphenated buzzwords failed to materialize. Anyone remember teachers being forced to kneel under desks because their test scores didn't go up and then getting fired when the scores didn't go up again? How about the ATL scantron parties where district leadership just falsified the scores?
The tests themselves are not to blame, obviously. And they are not some warped instrument of white supremacy wielding western colonialist mathematics against black and brown bodies (to borrow the parlance). But they were the justification that schools heard again and again for what turned out to be a whole package of reforms that didn't work but made life in many schools much worse. This was especially true in the schools that were the lowest performers.
So you can see why testing gets lumped in with all the bad stuff. Justified or not, it comes from somewhere and it's not only unpopular because of what the testing reveals but also because of what the testing was used for.
Following yesterday's post, is there a chance that some of this anti-testing spittle on the liberal left is partly redirected anxiety over high-end college admissions? Or even bad-faith attempts to transition towards "softer" metrics like GPA, club membership, and letters of rec because they're easier for mediocre rich kids to game than the SAT?
I have this hunch, purely anecdotal, that the most vociferous anti-testers are rich white parents who publicly scream about helping poor black kids but privately worry more about their bumbling failson with SAT scores far below the median for incoming freshmen at their elite alma mater.
I think there's some of that. I also think that there's a lot of genuinely well-meaning anxiety about race going on here, and there's a quiet belief among a lot of white progressives that (despite their BLM rhetoric) real progress for Black America is probably impossible. So they agitate for lowering standards and eliminating indicators because they do want best for Black children but secretly don't think they're capable of meeting contemporary standards.
For my part, I don't think that there are any likely short-term major positive jumps in Black academic performance coming. But I do think that there are major improvements to Black quality of life and political power that could be achieved with strategic economic policy. But a lot of liberals have kind of given up on the possibility of large-scale economic policy change, or don't want it because they are economically comfortable themselves. I realize that this is a lot of armchair psychologizing.
That makes sense, thanks! Elite reproduction anxiety is a hobbyhorse or mine, so I tend to see it everywhere.
I agree with you on the impossibility of short-term jumps, too. It's depressing, looking at the data, to think how little we can do to meaningfully change outcomes for the class of 2024, but overwhelming to think about the dismal chances of getting the slow-and-steady infrastructure changes we need *right now* to give the class of 2039 a fair shot.
Speaking of anxiety among elites I have to think about some of the anecdotal hostility I've witnessed directed towards Asian students: namely that tiger parenting results in humorless drones who may be good at test taking but are completely incapable of creative thinking. I have to wonder if it's any coincidence that there seems to be a significant overlap with the criteria that Harvard uses to discriminate against Asian applicants (personality traits such as likeability, personal courage, etc.).
"Few people would doubt, however, that any assessment regime should involve the kind of basic numeracy and literacy testing common to virtually all K-12 standardized exams, the NAEP, and entrance exams like the SAT and similar."
should not involve?
Should involve, I think - few would doubt that testing should involve basic numeracy and literacy. Yeah?
hmmm, showing my lack of reading comprehension there
I was a child when first introduced to the criticism that standardized tests are biased. It was in an episode of "Good Times" (DyNoMiiiiiiite!), which we watched as foreigners unfamiliar with life in America. The youngest son had taken a standardized test and done poorly, and was frustrated by the test's unfairness. The example he gave was that he was asked to pick the correct pairing with "cup." The answer was "saucer," but he had answered "table," because in his household, the cup went on the table, and they didn't own saucers.
This scene from a sitcom eons ago stuck in my head. It was the first time I'd encountered the novel (to me) "systemic bias" concept, though I was somewhat unimpressed by the example given to demonstrate it. I didn't think a 10 or 11 year old needed to have grown up with cups and saucers to know of their existence, and I was skeptical that such a question would even be on tests, at least in other than a multiple-choice format, eliminating the excuse for picking table over saucer. But this example stuck with me because, though it was silly, it made its point: standardized tests should not ask questions that test the knowledge, culture, or personal experiences predominantly of a particular race, culture, or income level.
We have gone from this undeniable premise, to arguments that 1 + 1 can equal 3, if the test taker thought the equation represented animal breeding, which apparently only non-whites unburdened by the oppressive rigors of math would do. The fictional "cup," "saucer," and "table" example was at least concrete. The criticisms today that standardized test are racially stratified don't offer concrete examples. The proof supporting this conclusion is the test performance statistics themselves, not individual questions. It would help if there was some explanation on why a particular test question is racially biased. I'm open to that possibility and I'd like to know what bias creeps into testing. Maybe there are already gobs of papers demonstrating this point. I just don't know what the substantive arguments are, other than the predictable ranking of test results by race.
One well-defined meaning of "bias" is that it's the difference between groups in the group-averaged test-predicted performance compared to the actual performance difference. This definition only has meaning if there is something that the test is being used to predict, e.g. completion of a degree program. I believe that the standard tests used for school-related purposes have little or no demographic bias in this sense.
I stumbled on some relevant evidence in the process of helping my wife assess the effects of using only online lectures for her intro stats course. Since the students couldn't be randomized to alternative treatments, we had to adjust for predictive covariates. Math ACT turned out to be the most predictive of objective exam scores, but high school GPA added some incremental predictive power. The typical math ACT of the student's major also added significantly to those, which surprised us a bit. For homework score, HSGPA beat math ACT, just as you'd expect for a more effort-weighted results. Overall, it's pretty much just what common sense would say.
BTW, on another topic, the study outcome was that in this course purely online lectures did at least as well as in-person +online. And yes, we did everything we could to adjust for confounders using four different statistical techniques, which agreed very closely. We don't think that important other confounders are lurking around.
https://arxiv.org/abs/2101.06755
In "The typical math ACT of the student's major also added significantly", was "math" intended? I'm unfamiliar with the ACT.
Whoops, I think that was typical total ACT of the major. For the individual students we had separate components, and only math was predictive. SATs were converted t ACT equivalents.
Does anyone have a link to actual SAT questions that have been flagged as being biased? I know that there have been great efforts within the last 30 years+ to eliminate biased questions.
BTW, a friend who works as admissions counselor told me that in his high school, various teachers compared SATs given in the 1960s and 1970s to more recent (last 20 years) SATs. My friend said that the test from the 60s and 70s were far more difficult than ones recently given.
Such great writing: lucid, engaging, familiar, and - as usual - provocative in an equal opportunity way. As one of those “conservative[s] who’s certain that the minimum wage kills jobs because his understanding of Econ 101 tells him so” I have to ask, even though it’s somewhat off-topic, am I mistaken? I’m willing to be wrong, but I’d like to see and understand your take on the evidence we have. I just recently subscribed to your substack so I might have missed your past writing on the topic. But if you haven’t yet written about it, would you at some point? It’s an important and consequential subject.
You might be interested in Matt Yglesias's writing on this subject, since in many ways he is the ultimate reasonable-center-left representative, with pragmatic essays about things like minimum wage mostly untainted by political wishful thinking. Not saying that he's right about everything, but he does present genuinely reasoned arguments instead of ultra-progressive propaganda.