The above graphic, from an old Wall Street Journal article, is just one of many similar graphs that get floated around on social media when people rail against the SAT. “Look!” they shout. “It’s just an income test! The SATs just tell you who the rich kids are.” Well, elite colleges are very invested in harvesting rich kids, yes, but the SAT is not the problem. This particular graphic is specifically misleading in the scale of its Y-axis - these are 800-point tests, and yet the total range of the Y-axis is 180 points, making the effect seem misleadingly large. But it’s also just weird to try and represent a simple correlation with a graph in this context. We have specific statistics that reflect what these graphs try to depict, the amount of variation in one number (in this case SAT scores) that is predictable from another number (family income). In particular, the
The information captured by r^2 is completely hidden in this chart; we don't even get error bars. It's compatible with nearly any strength of association between the variables. A scatterplot would have been more informative on variance but probably concealed the association from visual inspection.
Variance in often omitted (I doubt it's intentional hiding) from popular reporting, which is fine for most readers because few have learned about it to begin with. Nearly everyone can get this gist of a simple chart or graph like the one above. Never mind that variance is one of the most important concepts you'll need to understand and evaluate anything beyond summary statistics.
Box plot would be another option to show this data better than the bar plot, since it distills information about quantiles in an easy to digest way. The bar chart misses the "within-group variation" that you're referring to, and it would be useful to see what % of people in each income group score above a certain target score.
Even more broadly than your second point - our society, broadly speaking, tends to reward smart people who go to good colleges with higher-paying jobs - not all the time, not consistently, but as a broad trend. If there is even a tiny bit of genetic OR home environmental cause for those traits we should expect people with high incomes to have smarter and/or better-at-studying kids - not because the income itself magically causes it, but because, well, it would be weird if two mathematicians had no higher chance of having a kid who was good at math than two assembly line workers. Again, broadly speaking, general trends, stereotyping, etc, etc., but that's why it's a 6% r^2 like you said and not 100. And again, it's not necessarily even genetic, it's also just two college-educated professionals being able to help a student plan for and prioritize academics and having houses full of books and so on compared to two parents juggling fast-food jobs. It would be surprising if that didn't have any effect.
Ha, but you didn't consider it's actually the two assembly line workers who are more likely to have the intelligent kid who's good at math than the two highly educated mathematicians, because they are the only ones with a probability of having kids that's greater than zero!
"And across 150,000 students, they observe an r² of .0625 when regressing SAT scores and family income."
- might be inaccurate. They regress *freshman GPA* on SES (and SAT and HS GPA) and find r² of 0.068.
You write:
"We have specific statistics that reflect what these graphs try to depict, the amount of variation in one number (in this case SAT scores) that is predictable from another number (family income)"
Sackett et al write: "In the 2006 national population of test takers, the correlation between SES and composite SAT score was .46. Therefore, 21.2% of variance in SAT scores is shared with SES, as measured here as a composite of mother’s education, father’s education, and parental income. Thus, SAT scores are by no means isomorphic with SES, although the source of the SES-SAT relationship is likely due to some combination of educational opportunity, school quality, peer effects, and other social factors."
So I think r² might be 0.212, not 0.0625, in the original sentence.
Would be interesting to see data actually comparing the correlation by splitting those SES measures separately into income on one hand and parental education (or even parent SAT or IQ scores if available) on the other hand. I know what I would bet would have the higher correlation.
People talk as if attending a selective college makes the students smarter and earn more.
It's pretty obviously the other way around - smart people win. They win everything - exclusive college admittance, top jobs, high incomes, social status.
Of course they do - they're smart. If "smart" doesn't mean "the ability to find ways to win", then whatever does it mean?
Intelligent non-academics are among the most overlooked and undeservedly unlistened to. I think they're more likely to have novel and original views unrestrained by the intense in-group pressures of scientific peer groups, as well.
Far as I can tell, the evidence all suggests smart people don't win, they lose, and that's why there's fewer and fewer intelligent people: genes for educational attainment and cranial volume are being actively selected against, and we are seeing the reversal of the Flynn effect. You must be noticing the examples of success more, either that or your ideas of success are subjective, since biologically they are losers without a doubt.
The things you list as "wins" like exclusive college admittance are ironic because of how educational attainment correlates with reproductive success. So those people are clearly not future oriented as they select themselves out of the gene pool—which if they're smart, they are harming society and future generations as a whole by ensuring the next generation is less intelligent—so how smart are they, anyway, if they are making gene-culture co-evolution with intelligence fall into a negative feedback loop?
For the above claims, I can cite Valge, M., P. Horak, and J.M. Henshaw. (2020) and Kong, A., M.L. Frigge, G. Thorleifsson, H. Stefansson, A.I. Young, F. Zink, G.A. Jonsdottir, A. Okbay, P. Sulem, G. Masson, D.F. Gudbjartsson, A. Helgason, G. Bjornsdottir, U. Thorsteinsdottir, and K. Stefansson. (2017) as a couple examples of further reading.
People with slightly above average intelligence not too far out of the normal range who possess certain personality traits are the obvious winners, not smart people in general, and especially not if their intelligence isn't the verbal kind or they lack personality traits essential to educational and career success such as agreeableness. Intelligence helps, but clearly is secondary to other traits. Just look at how the best paid jobs are all in areas like business, if you want income intelligence is secondary.
I take issue defining intelligence as meaning "the ability to find ways to win" in the way you mean as well, since to "win" can mean anything even if counterproductive biologically and socially. For most a "win" might be material wealth, for many it's not, and for outliers it varies, e.g if you're Ted Kaczynski a "win" would be destruction of modern technological society (better yet civilisation too), and if you're kind of machine that was his worst enemy you may decide a "win" is to (with complex instrumental, strategic reasoning) find the optimal way of converting all matter on Earth including human beings into paperclips. What is more correct? See Bostrom, N. (2012): intelligence and value are orthogonal.
In short: intelligence ranges from secondary to counterproductive for conventional definitions of success in modern society, which fits with how rarely it's selected for in nature at all. Other traits are more important.
Broadly speaking, I think you're correct and it's a fascinating (and worrying) trend. I wonder if it's new (or if high intelligence people also had few children in the past).
But the post was narrowly about income and SAT scores - in that domain smart people "win". Here "win" means "get what they want". Modern society rewards things other than reproductive success - so "what they want" isn't successful children and proliferation of genes, but income, status, etc.
“ And, given that SAT scores are not easily modified with tutoring, it’s truly strange that the very-real phenomenon of rich students having a leg up in college admissions is blamed on the test.”
Income tracks with intelligence so it should be unsurprising that children of wealthier people do better on standardized tests.
In Ontario we have a test called the EQAO. Passing this test is mandatory for high school graduation; except if you look closely, it really isn’t. Which makes the test a multimillion dollar boondoggle.
My read is that a lot of the critics of standardized tests understand perfectly well that they aren’t that gameable. That’s exactly why they don’t like the tests - they’re too accurate for too comfort. PMC parents aren’t too keen on finding out their kids aren’t budding geniuses.
There’s something simmering below the testing/college/inequality discourse that never seems to get explicitly articulated, and it’s one of those weird, seemingly paradoxical moral attitudes of the professional managerial class. To wit: for as long as any real class mobility has been possible under a capitalist, even-pseudo-meritocratic social order, a big part of amassing wealth and social status and adopting bourgeois norms has been so you could pass these tangibles and intangibles along to your children - so that they could have a better or equal quality of life. The fact that highly meritocratic, driven, ambitious, basically “Protestant work ethic” people then turn around and blast the very system that rewarded them, and from which their children will surely benefit, is unwittingly or not undermining basic assumptions about human behavior. It’s probably at root biological: don’t fuck with people’s children’s opportunities or they will turn on you right quick (and their behavior will be manifest in all kinds of macroeconomic trends, migration patterns, stuff that only a much more authoritarian society could control for). So there’s this bizarre inconsistency at the heart of the supposed social contract: work hard and adopt certain norms, and you’ll be rewarded to a greater or lesser extent, but then you *must* at least publicly attack the very idea that the system is at all efficient or just, must pretend not to know that smart couples have smart children, must ignore the (fuzzy but very real) association between academic intelligence and income. The collision here is between an older critical attitude toward inherited wealth (like actual capital) and aristocratic privileges, and a newer bourgeois self-loathing which doesn’t map as sensibly onto what seems to be more targeted now, which is upper-middle class achievement, professional managerial class and petite bourgeois level social formations. Income inequality and the furtherance of basic social-democratic lifestyle expectations is a huge, festering problem, but you can’t even begin to think about solutions if the scapegoat for all this is going to be the daughter of of hardworking dry cleaners who becomes a pediatrician or McKinsey consultant or something.
"This particular graphic is specifically misleading in the scale of its Y-axis - these are 800-point tests, and yet the total range of the Y-axis is 180 points, making the effect seem misleadingly large."
This is annoyingly common across a range of topics in the media.
I'm not sure why this is a valid critique on Freddie's part?
The chart shows deviation from the average, so add the average in and you get raw score vs. income. What's important seems to be exactly the deviation from the mean, though, but we require further context about the point spreads. For example, how does +/-100 points affect percentile performance?
Agree with JQXVN that the problem is a lack of confidence intervals.
It isn't like the scores go from 0 to 1600 with equal distribution. What score do you get with random guessing? 400? So 180 point variance is a fair amount more than he seems to be admitting here.
I think this entire field is pretty suspect. But regardless, I don't think you've really captured what is motivating the push-back to SAT statistics.
We have statistics showing:
Genetics -> IQ -> SAT -> Educational Achievement -> All sorts of good things.
These are easily and accurately broken down by race. If you are following the statistics, I see no reason as to why you would stop with SAT being a reasonable predictor. It is probably the weakest link in the chain.
Driven people with high IQ's will be much better suited to certain types of work that society prizes. I haven't seen much else in this field that is convincing.
"This measuring tool/report is unflattering to my desired reality, bring me back a better one" is the feeling I often get wrt anti-testing...that, and confusing The Thing and The Symbolic Representation Of The Thing. If every SAT score were magically inflated to 800, this wouldn't solve the education gap
(except on paper along this one dimension)...because the actual desired outcomes of getting degrees, landing remunerative jobs, etc. wouldn't change. And a generic War On Episteme is just as pointless in the other direction; as Freddie often writes about, this just makes colleges/employers/whatever choose worse proxies for the same intended candidate pool. The key thing is to change the actual underlying reality, not the measurements thereof. (How would we know we're doing a better job of educating the disadvantaged if we couldn't measure their progress, anyway?)
I'd also like to say thanks for your various posts containing statistics. It's an invaluable science, and as a math-illiterate who's nonetheless interested in policy areas heavily reliant on such data, I really value having someone explain what the arcanum means in layman's terms. Wish you'd been one of my math teachers.
I agree with your main point, but the fact is that the research on SAT prep is not really granular enough to tell if private test prep makes a difference, much less if 8 weeks of 40 hours of prep with 3 hours of homework a night and 10 practice tests has more of an impact than, you know, picking up a Kaplan manual.
Asian test prep is insane and the SAT has become singularly less discriminating on the high end since 1995 for math and 2005 for English. The recent test is an entirely different animal, with a ridiculously simple verbal section and a math section that is simultaneously more reliant on higher math while asking easier questions about that math. And I don't think there's any research on whether or not prep matters in the most recent test. Most of the research was done on the earlier versions, which were much harder. And, as Briggs points out in your link, 20-30 points means a lot more at 1550 than it does at 450.
Finally, most of the research showed the *average* improvement was 20-30 points, with some kids getting a much bigger boost. So it always makes sense to send your kid to prep with those odds.
I've long since stopped believing in the tests as meaningful past broad tiers. It's hard to fake a 1200, but pretty easy to fake a 1400 given a 1200.
I agree it's not an income test. Just skeptical of the broader claims about the futility of test prep.
Yeah, I’m skeptical of this claim that SAT prep doesn’t work. I took the test itself 3 times, and did Princeton Review, and went up about 250 points.
Prep varies a lot, but I’m sure I did 2-3 full-length practice exams. I’m not sure how this research was conducted, but I assume many prep exercises could be DIY’d almost as effectively. And taking the test repeatedly is probably very impactful by itself (as my score went up from test to test even with no prep in between). Plus there are maturation effects from 10th to 11th grade.
I would guess that selection effects swamp everything else, and that different segments of students benefit differently from prep.
I can’t imagine it isn’t helpful to everyone, but there’s also a hard ceiling imposed by innate ability and actually knowing the content that prep can’t break through.
I also think prep is a “you get out of it what you put into it” situation, and a lot of kids are sent to SAT prep programs by their parents, with absolutely no intention of putting forth any effort.
I was shocked that no one in my Princeton Review class seemed to be trying, and I’d be shocked if their scores went up much.
Education in pretty much all of its forms is a "you get out of it what you put into it" situation. It shouldn't be much of a surprise that it correlates with grades and financial / status success.
SAT discourse is insane because the reality is this: the SAT is not a hard test. I took a practice test from Khan during the pandemic to make sure my memory wasn’t playing tricks on me. The math is just algebra and basic geometry. The verbal is mostly reading comprehension (what’s the main point of the passage? What’s the most logical conclusion one could draw?) and some grammar. People act like it’s Samson’s riddle, but any reasonably smart person who went to a decent (not even good) high school should be fine.
I think a lot of this discourse is just anxiety and cope. There’s resistance to acknowledging that plenty of students can show up basically cold and nail the test, including many kids of scientists and lawyers who are certainly privileged but also legitimately cognitively gifted. No, it has to be rigged in some way.
At the risk of slightly removing some anonymity from myself, I can offer my anecdata here: I went to an above-average public high school on the suburban East Coast with a small but non-insignificant sliver of students who were the kids of educated immigrants, and the rest mostly white.
The first time I took a SAT practice test cold, I got the equivalent of a 2100-something out of the 2400 scale. When I told my non-immigrant friends this, they genuinely reacted as if I'd won a MacArthur genius grant. Through some questioning, I found out that most of their scores ranged from 1800-1900, and they'd been studying for multiple weeks. (These scores were highly biased towards the verbal/writing sections, and away from the quantitative ones.)
When I told my immigrant friends about my 2100 score, they told me I better crack open those Kaplan/Princeton Review books and start studying.
I completely agree that the SAT is definitely not a hard test for some people, and maybe this is just an American thing, but the number of otherwise privileged kids I've known with access to competent education who struggled with comprehending algebra and basic geometry is not trivial. And these were kids that, frankly, would be in the top half of high school students across the country.
(For the record, more anecdata on whether access to SAT tutoring/practice material actually helps or not: After my 2100 "debacle", I bought a Barron's book from Barnes and Noble and studied it for about two and a half weeks before the test. My final score ended up being in the 2300's.)
The information captured by r^2 is completely hidden in this chart; we don't even get error bars. It's compatible with nearly any strength of association between the variables. A scatterplot would have been more informative on variance but probably concealed the association from visual inspection.
Variance in often omitted (I doubt it's intentional hiding) from popular reporting, which is fine for most readers because few have learned about it to begin with. Nearly everyone can get this gist of a simple chart or graph like the one above. Never mind that variance is one of the most important concepts you'll need to understand and evaluate anything beyond summary statistics.
Box plot would be another option to show this data better than the bar plot, since it distills information about quantiles in an easy to digest way. The bar chart misses the "within-group variation" that you're referring to, and it would be useful to see what % of people in each income group score above a certain target score.
Even more broadly than your second point - our society, broadly speaking, tends to reward smart people who go to good colleges with higher-paying jobs - not all the time, not consistently, but as a broad trend. If there is even a tiny bit of genetic OR home environmental cause for those traits we should expect people with high incomes to have smarter and/or better-at-studying kids - not because the income itself magically causes it, but because, well, it would be weird if two mathematicians had no higher chance of having a kid who was good at math than two assembly line workers. Again, broadly speaking, general trends, stereotyping, etc, etc., but that's why it's a 6% r^2 like you said and not 100. And again, it's not necessarily even genetic, it's also just two college-educated professionals being able to help a student plan for and prioritize academics and having houses full of books and so on compared to two parents juggling fast-food jobs. It would be surprising if that didn't have any effect.
Ha, but you didn't consider it's actually the two assembly line workers who are more likely to have the intelligent kid who's good at math than the two highly educated mathematicians, because they are the only ones with a probability of having kids that's greater than zero!
(Funniest part is I'm only half-joking.)
"The actual process is trivial and left as an exercise to the reader..." :-D
I think this -
"And across 150,000 students, they observe an r² of .0625 when regressing SAT scores and family income."
- might be inaccurate. They regress *freshman GPA* on SES (and SAT and HS GPA) and find r² of 0.068.
You write:
"We have specific statistics that reflect what these graphs try to depict, the amount of variation in one number (in this case SAT scores) that is predictable from another number (family income)"
Sackett et al write: "In the 2006 national population of test takers, the correlation between SES and composite SAT score was .46. Therefore, 21.2% of variance in SAT scores is shared with SES, as measured here as a composite of mother’s education, father’s education, and parental income. Thus, SAT scores are by no means isomorphic with SES, although the source of the SES-SAT relationship is likely due to some combination of educational opportunity, school quality, peer effects, and other social factors."
So I think r² might be 0.212, not 0.0625, in the original sentence.
https://citeseerx.ist.psu.edu/document?repid=rep1&type=pdf&doi=316b3e2f17c886d44700b835118965767cf897ce
Thanks, will update!
Would be interesting to see data actually comparing the correlation by splitting those SES measures separately into income on one hand and parental education (or even parent SAT or IQ scores if available) on the other hand. I know what I would bet would have the higher correlation.
People talk as if attending a selective college makes the students smarter and earn more.
It's pretty obviously the other way around - smart people win. They win everything - exclusive college admittance, top jobs, high incomes, social status.
Of course they do - they're smart. If "smart" doesn't mean "the ability to find ways to win", then whatever does it mean?
Well there is something in successful intelligence that does not always shine in the narrow measures of academics.
Intelligent non-academics are among the most overlooked and undeservedly unlistened to. I think they're more likely to have novel and original views unrestrained by the intense in-group pressures of scientific peer groups, as well.
Far as I can tell, the evidence all suggests smart people don't win, they lose, and that's why there's fewer and fewer intelligent people: genes for educational attainment and cranial volume are being actively selected against, and we are seeing the reversal of the Flynn effect. You must be noticing the examples of success more, either that or your ideas of success are subjective, since biologically they are losers without a doubt.
The things you list as "wins" like exclusive college admittance are ironic because of how educational attainment correlates with reproductive success. So those people are clearly not future oriented as they select themselves out of the gene pool—which if they're smart, they are harming society and future generations as a whole by ensuring the next generation is less intelligent—so how smart are they, anyway, if they are making gene-culture co-evolution with intelligence fall into a negative feedback loop?
For the above claims, I can cite Valge, M., P. Horak, and J.M. Henshaw. (2020) and Kong, A., M.L. Frigge, G. Thorleifsson, H. Stefansson, A.I. Young, F. Zink, G.A. Jonsdottir, A. Okbay, P. Sulem, G. Masson, D.F. Gudbjartsson, A. Helgason, G. Bjornsdottir, U. Thorsteinsdottir, and K. Stefansson. (2017) as a couple examples of further reading.
People with slightly above average intelligence not too far out of the normal range who possess certain personality traits are the obvious winners, not smart people in general, and especially not if their intelligence isn't the verbal kind or they lack personality traits essential to educational and career success such as agreeableness. Intelligence helps, but clearly is secondary to other traits. Just look at how the best paid jobs are all in areas like business, if you want income intelligence is secondary.
I take issue defining intelligence as meaning "the ability to find ways to win" in the way you mean as well, since to "win" can mean anything even if counterproductive biologically and socially. For most a "win" might be material wealth, for many it's not, and for outliers it varies, e.g if you're Ted Kaczynski a "win" would be destruction of modern technological society (better yet civilisation too), and if you're kind of machine that was his worst enemy you may decide a "win" is to (with complex instrumental, strategic reasoning) find the optimal way of converting all matter on Earth including human beings into paperclips. What is more correct? See Bostrom, N. (2012): intelligence and value are orthogonal.
In short: intelligence ranges from secondary to counterproductive for conventional definitions of success in modern society, which fits with how rarely it's selected for in nature at all. Other traits are more important.
Broadly speaking, I think you're correct and it's a fascinating (and worrying) trend. I wonder if it's new (or if high intelligence people also had few children in the past).
But the post was narrowly about income and SAT scores - in that domain smart people "win". Here "win" means "get what they want". Modern society rewards things other than reproductive success - so "what they want" isn't successful children and proliferation of genes, but income, status, etc.
“ And, given that SAT scores are not easily modified with tutoring, it’s truly strange that the very-real phenomenon of rich students having a leg up in college admissions is blamed on the test.”
Income tracks with intelligence so it should be unsurprising that children of wealthier people do better on standardized tests.
In Ontario we have a test called the EQAO. Passing this test is mandatory for high school graduation; except if you look closely, it really isn’t. Which makes the test a multimillion dollar boondoggle.
> because so many people harbor personal resentments for the SAT because of the stress of taking it themselves
Aha! My cope analysis from the last post proves correct. Cope Analysis will henceforce be shortened to "Copanal" and designated as useful.
Your new word read like some weird police fetish and now I can't unsee it...
Can't make an omlette without breaking a few eggs
My read is that a lot of the critics of standardized tests understand perfectly well that they aren’t that gameable. That’s exactly why they don’t like the tests - they’re too accurate for too comfort. PMC parents aren’t too keen on finding out their kids aren’t budding geniuses.
There’s something simmering below the testing/college/inequality discourse that never seems to get explicitly articulated, and it’s one of those weird, seemingly paradoxical moral attitudes of the professional managerial class. To wit: for as long as any real class mobility has been possible under a capitalist, even-pseudo-meritocratic social order, a big part of amassing wealth and social status and adopting bourgeois norms has been so you could pass these tangibles and intangibles along to your children - so that they could have a better or equal quality of life. The fact that highly meritocratic, driven, ambitious, basically “Protestant work ethic” people then turn around and blast the very system that rewarded them, and from which their children will surely benefit, is unwittingly or not undermining basic assumptions about human behavior. It’s probably at root biological: don’t fuck with people’s children’s opportunities or they will turn on you right quick (and their behavior will be manifest in all kinds of macroeconomic trends, migration patterns, stuff that only a much more authoritarian society could control for). So there’s this bizarre inconsistency at the heart of the supposed social contract: work hard and adopt certain norms, and you’ll be rewarded to a greater or lesser extent, but then you *must* at least publicly attack the very idea that the system is at all efficient or just, must pretend not to know that smart couples have smart children, must ignore the (fuzzy but very real) association between academic intelligence and income. The collision here is between an older critical attitude toward inherited wealth (like actual capital) and aristocratic privileges, and a newer bourgeois self-loathing which doesn’t map as sensibly onto what seems to be more targeted now, which is upper-middle class achievement, professional managerial class and petite bourgeois level social formations. Income inequality and the furtherance of basic social-democratic lifestyle expectations is a huge, festering problem, but you can’t even begin to think about solutions if the scapegoat for all this is going to be the daughter of of hardworking dry cleaners who becomes a pediatrician or McKinsey consultant or something.
"This particular graphic is specifically misleading in the scale of its Y-axis - these are 800-point tests, and yet the total range of the Y-axis is 180 points, making the effect seem misleadingly large."
This is annoyingly common across a range of topics in the media.
I'm not sure why this is a valid critique on Freddie's part?
The chart shows deviation from the average, so add the average in and you get raw score vs. income. What's important seems to be exactly the deviation from the mean, though, but we require further context about the point spreads. For example, how does +/-100 points affect percentile performance?
Agree with JQXVN that the problem is a lack of confidence intervals.
It isn't like the scores go from 0 to 1600 with equal distribution. What score do you get with random guessing? 400? So 180 point variance is a fair amount more than he seems to be admitting here.
I think this entire field is pretty suspect. But regardless, I don't think you've really captured what is motivating the push-back to SAT statistics.
We have statistics showing:
Genetics -> IQ -> SAT -> Educational Achievement -> All sorts of good things.
These are easily and accurately broken down by race. If you are following the statistics, I see no reason as to why you would stop with SAT being a reasonable predictor. It is probably the weakest link in the chain.
Driven people with high IQ's will be much better suited to certain types of work that society prizes. I haven't seen much else in this field that is convincing.
"This measuring tool/report is unflattering to my desired reality, bring me back a better one" is the feeling I often get wrt anti-testing...that, and confusing The Thing and The Symbolic Representation Of The Thing. If every SAT score were magically inflated to 800, this wouldn't solve the education gap
(except on paper along this one dimension)...because the actual desired outcomes of getting degrees, landing remunerative jobs, etc. wouldn't change. And a generic War On Episteme is just as pointless in the other direction; as Freddie often writes about, this just makes colleges/employers/whatever choose worse proxies for the same intended candidate pool. The key thing is to change the actual underlying reality, not the measurements thereof. (How would we know we're doing a better job of educating the disadvantaged if we couldn't measure their progress, anyway?)
I'd also like to say thanks for your various posts containing statistics. It's an invaluable science, and as a math-illiterate who's nonetheless interested in policy areas heavily reliant on such data, I really value having someone explain what the arcanum means in layman's terms. Wish you'd been one of my math teachers.
I agree with your main point, but the fact is that the research on SAT prep is not really granular enough to tell if private test prep makes a difference, much less if 8 weeks of 40 hours of prep with 3 hours of homework a night and 10 practice tests has more of an impact than, you know, picking up a Kaplan manual.
Asian test prep is insane and the SAT has become singularly less discriminating on the high end since 1995 for math and 2005 for English. The recent test is an entirely different animal, with a ridiculously simple verbal section and a math section that is simultaneously more reliant on higher math while asking easier questions about that math. And I don't think there's any research on whether or not prep matters in the most recent test. Most of the research was done on the earlier versions, which were much harder. And, as Briggs points out in your link, 20-30 points means a lot more at 1550 than it does at 450.
Finally, most of the research showed the *average* improvement was 20-30 points, with some kids getting a much bigger boost. So it always makes sense to send your kid to prep with those odds.
I've long since stopped believing in the tests as meaningful past broad tiers. It's hard to fake a 1200, but pretty easy to fake a 1400 given a 1200.
I agree it's not an income test. Just skeptical of the broader claims about the futility of test prep.
Yeah, I’m skeptical of this claim that SAT prep doesn’t work. I took the test itself 3 times, and did Princeton Review, and went up about 250 points.
Prep varies a lot, but I’m sure I did 2-3 full-length practice exams. I’m not sure how this research was conducted, but I assume many prep exercises could be DIY’d almost as effectively. And taking the test repeatedly is probably very impactful by itself (as my score went up from test to test even with no prep in between). Plus there are maturation effects from 10th to 11th grade.
I would guess that selection effects swamp everything else, and that different segments of students benefit differently from prep.
I can’t imagine it isn’t helpful to everyone, but there’s also a hard ceiling imposed by innate ability and actually knowing the content that prep can’t break through.
I also think prep is a “you get out of it what you put into it” situation, and a lot of kids are sent to SAT prep programs by their parents, with absolutely no intention of putting forth any effort.
I was shocked that no one in my Princeton Review class seemed to be trying, and I’d be shocked if their scores went up much.
Education in pretty much all of its forms is a "you get out of it what you put into it" situation. It shouldn't be much of a surprise that it correlates with grades and financial / status success.
Grades are worthless, and nothing infuriates me more than using them for college admission.
I grew up poor as shit and had a great score when I took the SAT in 7th grade (for the TIP program) and in 11th grade for college admission.
https://www.nakedcapitalism.com/2023/07/the-admissions-game.html
Tl;Dr controlling for test scores, poor kids get a slight bump in admissions. Rich kids get a huge bump.
SAT discourse is insane because the reality is this: the SAT is not a hard test. I took a practice test from Khan during the pandemic to make sure my memory wasn’t playing tricks on me. The math is just algebra and basic geometry. The verbal is mostly reading comprehension (what’s the main point of the passage? What’s the most logical conclusion one could draw?) and some grammar. People act like it’s Samson’s riddle, but any reasonably smart person who went to a decent (not even good) high school should be fine.
I think a lot of this discourse is just anxiety and cope. There’s resistance to acknowledging that plenty of students can show up basically cold and nail the test, including many kids of scientists and lawyers who are certainly privileged but also legitimately cognitively gifted. No, it has to be rigged in some way.
At the risk of slightly removing some anonymity from myself, I can offer my anecdata here: I went to an above-average public high school on the suburban East Coast with a small but non-insignificant sliver of students who were the kids of educated immigrants, and the rest mostly white.
The first time I took a SAT practice test cold, I got the equivalent of a 2100-something out of the 2400 scale. When I told my non-immigrant friends this, they genuinely reacted as if I'd won a MacArthur genius grant. Through some questioning, I found out that most of their scores ranged from 1800-1900, and they'd been studying for multiple weeks. (These scores were highly biased towards the verbal/writing sections, and away from the quantitative ones.)
When I told my immigrant friends about my 2100 score, they told me I better crack open those Kaplan/Princeton Review books and start studying.
I completely agree that the SAT is definitely not a hard test for some people, and maybe this is just an American thing, but the number of otherwise privileged kids I've known with access to competent education who struggled with comprehending algebra and basic geometry is not trivial. And these were kids that, frankly, would be in the top half of high school students across the country.
(For the record, more anecdata on whether access to SAT tutoring/practice material actually helps or not: After my 2100 "debacle", I bought a Barron's book from Barnes and Noble and studied it for about two and a half weeks before the test. My final score ended up being in the 2300's.)