One other factor is the progressive belief that expensive test prep is effective when the preponderance of the evidence says it adds little if any value. It's basically a scam and doesn't improve scores any more than a $20 test prep book that includes a few practice tests.
I feel really unclear on this because year in and year out I see students underperform on standardized tests, and a smaller number who over perform their daily show of understanding. I’m certainly biased but it doesn’t seem nearly as predictive as the other 179 days they’re with me.
And none of these students go to fourth grade and suddenly start performing like the standardized test said they would, they continue to be odd cases of luck or test anxiety.
Maybe I read this too fast, but I have a question. Let's see if I can try to make it clear. I never really put any thought to these tests since I had to take them, but I totally believe your claims about what people say about these tests (though there is no support, at least here, for the 'suddenly' claim at the outset). And I think your case here about their validity, reliability, etc is pretty straightforward.
What I don’t understand is what you think their actual current utility is. Say we find some group-level inequalities with these tests, is our society set up to do anything about that? It kinda reminds me of that MMT stuff, which for all its (for all I know) precision and correctness, seems to forget how power works in our society. I think the real task before one in favor of these tests is to show how they could be used as a lever to create a more just society. If we can’t convincingly make that claim, why are we doing them? Just to know more true things? Things not worth doing are not worth doing well.
Are these tests intended to be part of a larger sorting machine that creates fair job outcomes based on ability? Like some free-market style means of getting the right people into the right sort of jobs? That seems false to me. And even if it were true, it’s not clear to me that we would want such a thing. Do we want an elaborate apparatus as part of a technocratic attempt to optimize human potential? I certainly don’t want to live in such a society. But maybe you do and I think that case would be harder to make, but I’d love to hear it!
I'd add to this in a few ways. One is to note that the liberal consensus was decidedly pro-testing in the 90s and 2000s, but that the way in which large scale state testing rolled out under NCLB was modeled on Texas - which, FDB pointed out, may not have been the success story it claimed to be.
Another is that the testing became a tool in the hands of various reformers. Clearly it's the teachers unions who are to blame, I mean, look at these test scores! Enter Scott Walker, Chris Christie, Mike Bloomberg. Right along side of those reformers came expanded voucher programs, more charter schools, and pushes against collective bargaining. So, while the testing was finding various inequalities - something that liberals thought would provide evidence for locating and improving instructional practices, thus outcomes - the policies that seemed to result from testing's evidence base threatened teachers and schools systems' historic power as well as local control (charters and vouchers being very much outside of local board control in most places). So the liberal intent behind testing was perceived as being turned to somewhat libertarian and market-oriented purposes popular until recently on the right.
Finally, testing came with a veneer of corporate control. As FDB noted in the post kicking off edu week, Gates brought Common Core into being almost single handedly and with surprisingly little resistance (imo this is because, even then, lots of educators were still feeling like testing revealed a problem, so curriculum goals could solve it). The other big thing the Gates Foundation did was roll out curriculum aligned to Common Core in districts around the country and try to implement evidence-based instruction to, well, end educational inequality. It didn't work and Gates more or less lit money on fire for about a decade. But teachers and districts felt burned (especially when the CSAIL report said teachers and districts were the reason Gates' project failed).
So, at least coming from the perspective of the schools and their teachers/aides/admins, nearly two decades of constant changes, instability, and ruthless pay/benefits/job security cuts seemed to be the results of these tests. And the supposed upside, the renaissance of data-driven best-practices and all the other hyphenated buzzwords failed to materialize. Anyone remember teachers being forced to kneel under desks because their test scores didn't go up and then getting fired when the scores didn't go up again? How about the ATL scantron parties where district leadership just falsified the scores?
The tests themselves are not to blame, obviously. And they are not some warped instrument of white supremacy wielding western colonialist mathematics against black and brown bodies (to borrow the parlance). But they were the justification that schools heard again and again for what turned out to be a whole package of reforms that didn't work but made life in many schools much worse. This was especially true in the schools that were the lowest performers.
So you can see why testing gets lumped in with all the bad stuff. Justified or not, it comes from somewhere and it's not only unpopular because of what the testing reveals but also because of what the testing was used for.
Following yesterday's post, is there a chance that some of this anti-testing spittle on the liberal left is partly redirected anxiety over high-end college admissions? Or even bad-faith attempts to transition towards "softer" metrics like GPA, club membership, and letters of rec because they're easier for mediocre rich kids to game than the SAT?
I have this hunch, purely anecdotal, that the most vociferous anti-testers are rich white parents who publicly scream about helping poor black kids but privately worry more about their bumbling failson with SAT scores far below the median for incoming freshmen at their elite alma mater.
"Few people would doubt, however, that any assessment regime should involve the kind of basic numeracy and literacy testing common to virtually all K-12 standardized exams, the NAEP, and entrance exams like the SAT and similar."
I was a child when first introduced to the criticism that standardized tests are biased. It was in an episode of "Good Times" (DyNoMiiiiiiite!), which we watched as foreigners unfamiliar with life in America. The youngest son had taken a standardized test and done poorly, and was frustrated by the test's unfairness. The example he gave was that he was asked to pick the correct pairing with "cup." The answer was "saucer," but he had answered "table," because in his household, the cup went on the table, and they didn't own saucers.
This scene from a sitcom eons ago stuck in my head. It was the first time I'd encountered the novel (to me) "systemic bias" concept, though I was somewhat unimpressed by the example given to demonstrate it. I didn't think a 10 or 11 year old needed to have grown up with cups and saucers to know of their existence, and I was skeptical that such a question would even be on tests, at least in other than a multiple-choice format, eliminating the excuse for picking table over saucer. But this example stuck with me because, though it was silly, it made its point: standardized tests should not ask questions that test the knowledge, culture, or personal experiences predominantly of a particular race, culture, or income level.
We have gone from this undeniable premise, to arguments that 1 + 1 can equal 3, if the test taker thought the equation represented animal breeding, which apparently only non-whites unburdened by the oppressive rigors of math would do. The fictional "cup," "saucer," and "table" example was at least concrete. The criticisms today that standardized test are racially stratified don't offer concrete examples. The proof supporting this conclusion is the test performance statistics themselves, not individual questions. It would help if there was some explanation on why a particular test question is racially biased. I'm open to that possibility and I'd like to know what bias creeps into testing. Maybe there are already gobs of papers demonstrating this point. I just don't know what the substantive arguments are, other than the predictable ranking of test results by race.
I stumbled on some relevant evidence in the process of helping my wife assess the effects of using only online lectures for her intro stats course. Since the students couldn't be randomized to alternative treatments, we had to adjust for predictive covariates. Math ACT turned out to be the most predictive of objective exam scores, but high school GPA added some incremental predictive power. The typical math ACT of the student's major also added significantly to those, which surprised us a bit. For homework score, HSGPA beat math ACT, just as you'd expect for a more effort-weighted results. Overall, it's pretty much just what common sense would say.
BTW, on another topic, the study outcome was that in this course purely online lectures did at least as well as in-person +online. And yes, we did everything we could to adjust for confounders using four different statistical techniques, which agreed very closely. We don't think that important other confounders are lurking around.
Does anyone have a link to actual SAT questions that have been flagged as being biased? I know that there have been great efforts within the last 30 years+ to eliminate biased questions.
BTW, a friend who works as admissions counselor told me that in his high school, various teachers compared SATs given in the 1960s and 1970s to more recent (last 20 years) SATs. My friend said that the test from the 60s and 70s were far more difficult than ones recently given.
Such great writing: lucid, engaging, familiar, and - as usual - provocative in an equal opportunity way. As one of those “conservative[s] who’s certain that the minimum wage kills jobs because his understanding of Econ 101 tells him so” I have to ask, even though it’s somewhat off-topic, am I mistaken? I’m willing to be wrong, but I’d like to see and understand your take on the evidence we have. I just recently subscribed to your substack so I might have missed your past writing on the topic. But if you haven’t yet written about it, would you at some point? It’s an important and consequential subject.
One other factor is the progressive belief that expensive test prep is effective when the preponderance of the evidence says it adds little if any value. It's basically a scam and doesn't improve scores any more than a $20 test prep book that includes a few practice tests.
I feel really unclear on this because year in and year out I see students underperform on standardized tests, and a smaller number who over perform their daily show of understanding. I’m certainly biased but it doesn’t seem nearly as predictive as the other 179 days they’re with me.
And none of these students go to fourth grade and suddenly start performing like the standardized test said they would, they continue to be odd cases of luck or test anxiety.
Maybe I read this too fast, but I have a question. Let's see if I can try to make it clear. I never really put any thought to these tests since I had to take them, but I totally believe your claims about what people say about these tests (though there is no support, at least here, for the 'suddenly' claim at the outset). And I think your case here about their validity, reliability, etc is pretty straightforward.
What I don’t understand is what you think their actual current utility is. Say we find some group-level inequalities with these tests, is our society set up to do anything about that? It kinda reminds me of that MMT stuff, which for all its (for all I know) precision and correctness, seems to forget how power works in our society. I think the real task before one in favor of these tests is to show how they could be used as a lever to create a more just society. If we can’t convincingly make that claim, why are we doing them? Just to know more true things? Things not worth doing are not worth doing well.
Are these tests intended to be part of a larger sorting machine that creates fair job outcomes based on ability? Like some free-market style means of getting the right people into the right sort of jobs? That seems false to me. And even if it were true, it’s not clear to me that we would want such a thing. Do we want an elaborate apparatus as part of a technocratic attempt to optimize human potential? I certainly don’t want to live in such a society. But maybe you do and I think that case would be harder to make, but I’d love to hear it!
I'd add to this in a few ways. One is to note that the liberal consensus was decidedly pro-testing in the 90s and 2000s, but that the way in which large scale state testing rolled out under NCLB was modeled on Texas - which, FDB pointed out, may not have been the success story it claimed to be.
Another is that the testing became a tool in the hands of various reformers. Clearly it's the teachers unions who are to blame, I mean, look at these test scores! Enter Scott Walker, Chris Christie, Mike Bloomberg. Right along side of those reformers came expanded voucher programs, more charter schools, and pushes against collective bargaining. So, while the testing was finding various inequalities - something that liberals thought would provide evidence for locating and improving instructional practices, thus outcomes - the policies that seemed to result from testing's evidence base threatened teachers and schools systems' historic power as well as local control (charters and vouchers being very much outside of local board control in most places). So the liberal intent behind testing was perceived as being turned to somewhat libertarian and market-oriented purposes popular until recently on the right.
Finally, testing came with a veneer of corporate control. As FDB noted in the post kicking off edu week, Gates brought Common Core into being almost single handedly and with surprisingly little resistance (imo this is because, even then, lots of educators were still feeling like testing revealed a problem, so curriculum goals could solve it). The other big thing the Gates Foundation did was roll out curriculum aligned to Common Core in districts around the country and try to implement evidence-based instruction to, well, end educational inequality. It didn't work and Gates more or less lit money on fire for about a decade. But teachers and districts felt burned (especially when the CSAIL report said teachers and districts were the reason Gates' project failed).
So, at least coming from the perspective of the schools and their teachers/aides/admins, nearly two decades of constant changes, instability, and ruthless pay/benefits/job security cuts seemed to be the results of these tests. And the supposed upside, the renaissance of data-driven best-practices and all the other hyphenated buzzwords failed to materialize. Anyone remember teachers being forced to kneel under desks because their test scores didn't go up and then getting fired when the scores didn't go up again? How about the ATL scantron parties where district leadership just falsified the scores?
The tests themselves are not to blame, obviously. And they are not some warped instrument of white supremacy wielding western colonialist mathematics against black and brown bodies (to borrow the parlance). But they were the justification that schools heard again and again for what turned out to be a whole package of reforms that didn't work but made life in many schools much worse. This was especially true in the schools that were the lowest performers.
So you can see why testing gets lumped in with all the bad stuff. Justified or not, it comes from somewhere and it's not only unpopular because of what the testing reveals but also because of what the testing was used for.
Following yesterday's post, is there a chance that some of this anti-testing spittle on the liberal left is partly redirected anxiety over high-end college admissions? Or even bad-faith attempts to transition towards "softer" metrics like GPA, club membership, and letters of rec because they're easier for mediocre rich kids to game than the SAT?
I have this hunch, purely anecdotal, that the most vociferous anti-testers are rich white parents who publicly scream about helping poor black kids but privately worry more about their bumbling failson with SAT scores far below the median for incoming freshmen at their elite alma mater.
"Few people would doubt, however, that any assessment regime should involve the kind of basic numeracy and literacy testing common to virtually all K-12 standardized exams, the NAEP, and entrance exams like the SAT and similar."
should not involve?
I was a child when first introduced to the criticism that standardized tests are biased. It was in an episode of "Good Times" (DyNoMiiiiiiite!), which we watched as foreigners unfamiliar with life in America. The youngest son had taken a standardized test and done poorly, and was frustrated by the test's unfairness. The example he gave was that he was asked to pick the correct pairing with "cup." The answer was "saucer," but he had answered "table," because in his household, the cup went on the table, and they didn't own saucers.
This scene from a sitcom eons ago stuck in my head. It was the first time I'd encountered the novel (to me) "systemic bias" concept, though I was somewhat unimpressed by the example given to demonstrate it. I didn't think a 10 or 11 year old needed to have grown up with cups and saucers to know of their existence, and I was skeptical that such a question would even be on tests, at least in other than a multiple-choice format, eliminating the excuse for picking table over saucer. But this example stuck with me because, though it was silly, it made its point: standardized tests should not ask questions that test the knowledge, culture, or personal experiences predominantly of a particular race, culture, or income level.
We have gone from this undeniable premise, to arguments that 1 + 1 can equal 3, if the test taker thought the equation represented animal breeding, which apparently only non-whites unburdened by the oppressive rigors of math would do. The fictional "cup," "saucer," and "table" example was at least concrete. The criticisms today that standardized test are racially stratified don't offer concrete examples. The proof supporting this conclusion is the test performance statistics themselves, not individual questions. It would help if there was some explanation on why a particular test question is racially biased. I'm open to that possibility and I'd like to know what bias creeps into testing. Maybe there are already gobs of papers demonstrating this point. I just don't know what the substantive arguments are, other than the predictable ranking of test results by race.
I stumbled on some relevant evidence in the process of helping my wife assess the effects of using only online lectures for her intro stats course. Since the students couldn't be randomized to alternative treatments, we had to adjust for predictive covariates. Math ACT turned out to be the most predictive of objective exam scores, but high school GPA added some incremental predictive power. The typical math ACT of the student's major also added significantly to those, which surprised us a bit. For homework score, HSGPA beat math ACT, just as you'd expect for a more effort-weighted results. Overall, it's pretty much just what common sense would say.
BTW, on another topic, the study outcome was that in this course purely online lectures did at least as well as in-person +online. And yes, we did everything we could to adjust for confounders using four different statistical techniques, which agreed very closely. We don't think that important other confounders are lurking around.
https://arxiv.org/abs/2101.06755
Does anyone have a link to actual SAT questions that have been flagged as being biased? I know that there have been great efforts within the last 30 years+ to eliminate biased questions.
BTW, a friend who works as admissions counselor told me that in his high school, various teachers compared SATs given in the 1960s and 1970s to more recent (last 20 years) SATs. My friend said that the test from the 60s and 70s were far more difficult than ones recently given.
Such great writing: lucid, engaging, familiar, and - as usual - provocative in an equal opportunity way. As one of those “conservative[s] who’s certain that the minimum wage kills jobs because his understanding of Econ 101 tells him so” I have to ask, even though it’s somewhat off-topic, am I mistaken? I’m willing to be wrong, but I’d like to see and understand your take on the evidence we have. I just recently subscribed to your substack so I might have missed your past writing on the topic. But if you haven’t yet written about it, would you at some point? It’s an important and consequential subject.