What Does the Research Really Say About the Phonics Advantage?

and a few other Mississippi Miracle thoughts

The Argument responds to my recent skepticism about a supposed “Mississippi Miracle” or “Southern Surge.” In particular, they complain that

DeBoer characterizes the reforms in Mississippi as “minor administrative and pedagogical changes,” but spends zero time on what even happened in the state. The words “reading”¹ and “phonics” are absent from his piece.

There’s a lot more to be said about all of this, and I’ll try to address some general issues at the end, but this question of what exactly the research says about phonics and its supposedly immense advantage is what really matters. (Two can play the “challenge accepted” game!) I get that they’re saying that various administrative shuffling and (sigh) ACCOUNTABILITY are crucial in the Mississippi Miracle story, but there’s no DV there - we simply have no research to address such changes, which are always highly contextual and subject to great scaling difficulties. (“Let’s scale up the implausibly optimistic outcomes in this one learning context” is famous last words in ed policy.) What we do have is a lot of research about phonics, its advantages, and its limits. So let’s go deep on those details consider the recent conviction that phonics is some sort of secret magic enjoyed only by the rich and privileged. In fact, phonics is basic pedagogy, freely available to everyone, used extensively even in “whole language” reading classrooms, taught in many schools that are failure factories, enjoys real but quite modest effect-size advantages relative to placeholder reading instruction, and in general, not remotely worthy of all this hype.

In education, few things are more alluring than the claim that “phonics” (which is not actually a unified thing at all) is the magic key to fixing reading. If we just force phonics into every school (whatever that means), install an evidence-based program (whatever that means), mandate a rigorous curriculum (whatever that means), and demand accountability (whatever that means), then all children will not just read but excel at reading (whatever that means). It sounds… tidy. It also sounds like something people who want a clean policy solution can sell; never underestimate the degree to which political expediency determines educational orthodoxy. But when you dig into the research, phonics turns out to be helpful, sure, but far from transformational in all the ways people claim. “Phonics instruction, at the lowest levels and for basic word decoding, is probably better than other approaches” is defensible. Go ahead, defend it! You won’t get an argument from me. The far broader, more messianic, more revolutionary, emotionalist claims that are being bandied about in our media are in contrast not well supported by research, to put it mildly. Take away the relentless “if you ask any hard questions you hate children” rhetoric, and phonics becomes just another false god for the ed reform crowd, destined to disappoint.

The gains claimed by the Mississippi Miracle people aren't a matter of realistic, modest growth like you might expect from the research record; they’re extraordinary, never-before-seen gains of a type not remotely suggested by that research record. The effect sizes of phonics instruction in the relevant contexts are simply not sufficient to explain the explosive growth that Mississippi Miracle proponents are citing. So let's take a look.

The National Reading Panel

First, let’s look at what the most promising research actually demonstrates. We’ll start with the National Reading Panel research from 2000, which was a first mover when it came to the now-congealed conventional wisdom that phonics is the key to spreading literacy and that “whole language” and similar methods are a villain. The NRP findings are, to me, the definition of a real but modest finding. It’s essential to always contrast the effect size advantage of any particular approach with that of general instruction regardless of approach. As I’ve said for a long time, of course simply restricting students from having any exposure to formal teaching (such as happened, in effect, during COVID) will artificially restrict learning gains. What’s important however is that a) such restriction might hide but does not eliminate the existence of underlying talent differences between individual students and b) the fact that any instruction is generically better than none does not mean that a given approach is preferable than any other. If you just take raw NRP effect size results, or indeed pretty much any raw effect size results, you might be tempted to overinterpret them.¹ A common “generic” effect size often cited in this context is .34SD,² chosen here because of its presence in the phonics literature.

For children from low socioeconomic status backgrounds, a group targeted for all manner of educational interventions - that’s the “I’m here to save the children!” messianic policy apparatus at play - the NRP reported that systematic phonics resulted in an effect size of .45SD for overall reading performance. While a superficially moderate effect, this figure is still below most conventional thresholds for a medium effect and, crucially, represents only a slight elevation above the often-cited general educational intervention average of .34SD. That difference isn’t nothing! I think it’s worth thinking about, I think it’s worth researching more deeply, I think it’s good reason to attempt some of the policy interventions that proponents favor. But if something like a .10 to .15SD advantage over other reading instruction methods is what we’re really talking about here, that’s on the order of one-ninth or so of the generic racial achievement gap. An effect of that size would move a given learner something like 4% to 6% of the way up the performance distribution. Set aside for now my constant questions about how this all works if everyone receives the same advantage, meaning the whole distribution would move, leaving everyone running in place - that is worth fighting for, but not a major difference. A kid who goes from being a 20th percentile reader to a 25th percentile reader thanks to the advantage of phonics instruction over other methods is going to receive real but very modest practical benefits from doing so.

Even more concerning in the NRP data are the results for students struggling with reading difficulties, the natural research subject in context. Phonics instruction improved the reading performance of disabled readers (defined as children with average IQs but poor reading skills), but the resultant effect size was .32SD. An effect of 0.32SD is, obviously, lower than (and more importantly statistically indistinguishable from) the average educational intervention effect of .34SD. That difference simply isn’t practically meaningfully, and this result indicates that for individuals with the greatest need for reading support, systematic phonics instruction is not an intervention generating gains any bigger than any other kind of instruction. The suggestion is that phonics instruction is simply replicating the same gains to be expected from receiving any education at all. Again, this is the “not being a kid locked in a closet” advantage: if you artificially restrict students from any instruction, they will of course not learn anything. But if you do expose different students to instruction, their outcomes will vary widely thanks to their profoundly different individual academic potentials. This is a very simple concept that a lot of people profess not to understand because it’s argumentatively convenient for them not to understand.

Furthermore, the general applicability of systematic phonics is severely limited by age and achievement level. NRP meta-analysis demonstrates that phonics instruction failed to exert any significant impact on the reading performance of low-achieving readers in 2nd through 6th grades. This finding restricts the claim of effective intervention primarily to early, beginning readers. The implication is clear: once students move beyond the initial stages of reading development, the isolated teaching of phonics loses its effectiveness for generalized reading improvement. We also have the classic “decoding vs. actually understanding” debates at hand with this research. Yes, phonics research shows real gains when it comes to decoding words. But when measuring the impact of the same intervention on the ultimate outcome, that is, reading for meaning meaning, the effect size shrinks dramatically. When it comes to acutally comprehending texts among students in grades 2 through 6, the effect size was found to be only .12SD. That result is practically negligible, indicating that for the most crucial measure of reading proficiency, phonics instruction just doesn’t really offer much direct benefit beyond first grade. The difference between a moderate effect on decoding (.5SD) and a negligible effect on comprehension (.12SD) lends some support to the long-standing criticism that intense phonics instruction can produce “word callers,” who possess the mechanical skill to pronounce words but lack the ability to construct meaning.

Of course, decoding is a prerequisite to understanding, and I would never understate the importance of decoding. Walk before run, etc. But this is a crucial, crucial set of facts - phonics really only works for certain ages and certain levels of students at the pre-understanding stage of decoding words. Have you heard that from any of the many aggressive Mississippi Miracle proponents lately? Do the phonics partisans ever concede that the meaningful quantitative advantages are restricted to particular age groups and particular performance bands? No, they’re too busy hanging banners to actually parse the details.

Overall, the NRP results provoked many disagreement among researchers, who responded with warring meta-analyses. Researchers like Hammill and Swanson in 2006 argued that the magnitude of the effects reported by the NRP was modest. From my perspective, the consistent statistical outcome from the NRP data is that systematic phonics is effective, but not overwhelmingly superior to the typical gains expected from any well-designed educational approach. Which is my point exactly: these are real, durable gains, but ultimately fairly small ones, restricted to a particular age cohort and performance band, with very limited applicability to broader questions of reading comprehension, which is what we care about. Consider the abstract to this piece from phonics defenders responding to skepticism about the reported effects:

their findings do not contradict the NRP findings of effect sizes in the small to moderate range favoring systematic phonics… binomial effect size displays show that effect sizes of the magnitude found for systematic phonics by the NRP are meaningful and could result in significant improvement for many students depending on the base rate of struggling readers and the size of the effect. [emphasis added]

That’s the defensive position, that the effects are generally smallish and only potentially meaningful and largely dependent on the relative underlying performance of a given learning cohort. That’s not revolutionary stuff! Guys… the effect sizes just aren’t that big. The effect sizes just aren’t that big. It says a lot about this debate that simply accurately reporting on the basic details of the most influential research prompts people to make all sorts of personal accusations.

The Cochrane Review and Related Research

The widely-cited 2018 Cochrane review of phonics training for English-speaking poor readers (whew, that’s a mouthful) suggests that phonics pedagogy likely improves “non-word reading fluency, word reading fluency, irregular word reading accuracy, and letter-sound knowledge” compared to baseline. Please note the modesty of the identified advantages here! These are all pre-comprehension skills, skills that contribute to actually understanding complex texts but which can’t guarantee that understanding, anymore than knowing your times tables ensures that you can do algebra. The findings suggest that phonics might improve spelling and reading comprehension, as opposed to the more robust pure decoding advantages, but the identified effects are small and, given the underlying variability, uncertain. The summary statistic for reading comprehension in that review - a review which, I stress, is a core piece of evidence for the “phonics is god” crowd - is in fact quite modest, while the confidence intervals are large enough that we can’t be totally confident that the benefit is large. These are inconvenient facts but facts nonetheless.

Second, and crucially, long-term effects of phonics instruction are much weaker than immediate effects. Consider this cited 2014 meta-analysis that looked at interventions in reading, including phonics and various interventions related to comprehension, fluency, and similar approaches. At immediate post-test, the average effect size for those interventions is reasonable, but by follow-up (one-ish years later), most phonics and fluency interventions lose much of their advantage. This is, as I will not stop pointing out, a remarkably consistent finding in educational research - initial encouraging returns fade out as individual learners regress to their general talent level. The meta-analysis finds that “comprehension” and phonemic awareness approaches have better longitudinal outcomes, while phonics drags behind. In other words, phonics looks good at first analysis but fails to maintain such optimism over time; again, this is a generic finding in research, one frequently ignored by reform zealots. Similarly, a randomized Swedish study that gave children daily training in phoneme/grapheme mapping - phonics, more or less, although please note again that “phonics” as a unified whole is not a thing - reading comprehension, and reading speed in third grade showed at post-test better performance across several reading-related skills… but five years later, the only remaining significant advantage was in word decoding. Comprehension and speed gains largely faded. Once again, this is a generic, even unavoidable tendency in ed research.

Third, the benefits of phonics instruction do not generalize automatically into the things people care most about, such as understanding texts, reading across subjects, thinking from texts, and comprehension in general. Schools (and society) don’t just want children to sound out words, after all; they want students to read meaningfully, to learn history via text, science via text, etc. The literature on older struggling readers shows that while interventions that include decoding/phonics help, they typically show much larger effects on word-level tasks than comprehension. One synthesis of interventions for older students yielded a large average effect on comprehension outcomes when interventions were well rounded (i.e. decoding + fluency + vocabulary + comprehension), but when the intervention is mostly decoding (phonics) the effect on comprehension is smaller than that of much-demonized word-level interventions.

So what does all this mean politically and in policy terms? If phonics were truly magic, you’d expect a huge jump in national/comparative reading comprehension scores wherever phonics gets mandated, durable gains that persist into higher grades without massive extra instruction, uniform benefits across students (rich and poor, strong and weak) in comprehension, and minimal need for other reading supports. But the empirical record does not robustly deliver on any of that. Quite the contrary; once you move from a very reasonable “phonics instruction is probably a good place to start to give all students the best chance at eventually developing superior literacy ability” to a considerably less reasonable “phonics and shouting about accountability and such will deliver us a nation of super-geniuses,” you end up with a research record that can’t possibly backstop the rhetoric. You stay modest and reserved in your identified potential benefits, and you stay within the boundaries of the actual research record. You start talking about how we can just choose for every student to be an excellent reader, and you’re wandering well outside of those boundaries. Again, this is a generic condition in modern education debates.

There’s plenty of research to back up this need to stay modest. Many studies show that the biggest gains from phonics are for the weakest readers, or those behind in fluency or decoding. (This is also true in math and all manner of other domains btw.) Children who already have decent oral vocabulary, background knowledge, home reading, etc., often get less additional benefit. And when follow-ups happen after control and intervention groups go back to “normal instruction,” the phonics advantage shrinks or disappears for many of the non-decoding/read-word outcomes. That means phonics training is not self-sustaining without being embedded in a fuller reading environment. All of which points to my broader stance on education: you can make marginal improvements at the bottom of the performance spectrum, but as you move further and further along in educational life, students lock into their relative performance bands.

The Bowers Review

A very useful text that looks at exactly the kind of claims I’m interest in is a 2020 review article from British researcher Jeffrey Bowers. Bowers argues that despite the prevailing orthodoxy that holds that early reading instruction must begin with systematic phonics to be rigorous or “research-based,” the empirical support for that claim is weaker than people typically admit. He doesn’t defend whole language approaches as superior, nor does he dismiss the idea that phonics is an important tool, but rather questions the confidence around phonics as a research-derived position, which is my stance exactly. Because the more outlandish the claims about phonics grow, the more we should engage in healthy skepticism.

To make his case, Bowers re-examines 12 meta-analyses often cited in favor of phonics, assessing whether they genuinely support “systematic phonics beats all alternatives. Additionally, Bowers looks at England’s large-scale shift since 2007 to mandated phonics as a kind of natural experiment to see whether the national data support strong claims about phonics. His conclusion? Yes, phonics instruction generally helps in some measures (especially decoding and “nonword reading”), but the evidence for the superiority of phonics is far from the clear and overpowering case often claimed by phonics advocates and our credulous media. Many of the stronger claims about phonics and its relationship to higher-order skills like reading comprehension and the durability of perceived superiority to other methods don’t survive scrutiny. Meanwhile, the generic observation that there’s a normal distribution of academic talent and that students tend to populate themselves along that distribution endures.

Bowers’s review finds multiple recurring problems in the literature used to prop up the phonics superiority narrative:

1. Mischaracterization & overgeneralization. Many reviews and policy arguments take weak or narrow findings and present them as sweeping wins. Bowers argues authors sometimes stretch what their data justify; for instance, claiming “phonics is superior to whole language” even when the comparisons in the meta-analysis were not strictly between phonics vs. whole language.

2. Heterogeneous control groups. A recurring methodological flaw is that “control” conditions include a mixture of no phonics, unsystematic phonics, or other literacy practices. If the control group sometimes has unsystematic phonics, then finding a phonics vs. control difference doesn’t cleanly test systematic vs. unsystematic phonics, which is the precise claim being made by many advocates. There’s a great deal of inconsistent terminology and idiosyncratic classifications in the research in general.

3. Effects drop over time; weak long-term evidence. In many meta-analyses, the effect sizes measured immediately post-intervention are larger; but when follow-ups occur (4–12 months later or more), the gains often shrink or vanish, especially in outcomes like comprehension, spelling, or text reading (as opposed to decoding). To repeat myself once again, this is an outcome found in countless education studies, such as in the notorious case of pre-K research - initial outcomes seem very encouraging, but students inevitably regress over time to the level of their natural talent and stay there.

4. Bias, outliers, and over-reliance on small studies of dubiously large effect. Some meta-analyses count or give weight to studies with extreme effect sizes or questionable designs, which disproportionately inflate estimates. Bowers points out that some of the gains in word reading or accuracy are heavily driven by individual studies with methodological confounds. This, too, is a bad habit of education research; because the optimism bias in this field is so overwhelming, researchers cling to implausibly large positive results in an attempt to craft an optimistic narrative.

5. Lack of strong evidence on comprehension, broader reading outcomes. Of the 12 meta-analyses reviewed, only one reports a statistically significant effect on comprehension. (Again, phonics is about decoding on the word level, AKA grapheme-phoneme correspondence, not about understanding what’s been read, AKA comprehension.) Most of the observed strong, consistent effects are on simpler tasks (nonword reading, again, plus word decoding). Thus the leap from “phonics helps decoding” to “phonics ensures comprehension and advanced reading” is unjustified in relationship to the actual research record - but of course in the real world, outside of education debates, no one actually cares about decoding as such, only the practical (and monetizable) abilities decoding enables.

6. The difference between “helpful extra instruction” and superiority claims. Some meta-analyses compare phonics instruction against no additional reading intervention at all. Which… yeah. In those settings, anything (some structured instruction) might outdo nothing. But that’s not the same as showing that systematic phonics is superior to the “formative practices already in use” (that is… less systematic phonics, I guess?, AKA balanced approaches). Bowers calls this a subtle but crucial interpretive error.

Taken together, Bowers contends, these problems erode confidence in the stronger claims made for systematic phonics. And then there’s the case of English adoption of phonics as a policy mandate. Since 2007, English state primary schools have been legally required to adopt “systematic synthetic phonics,” along with a related standardized assessment. Many proponents point to this as a grand, real-world test that affirmed phonics. Bowers scrutinizes whether the national data actually back those claims.

• Short-term gains fade (again!) In phased rollouts, early benefits in early literacy assessments are visible, particularly in regards to decoding (ie “sounding out”) specifically, but when students progress to later standardized reading tests, the advantages disappear. For example, in an Early Reading Development Pilot that was attached to the UK adoption of phonics, immediate effects on foundational literacy are visible, but by key stage 1 and 2 tests (second and sixth grade equivalents, more or less), the effect sizes shrink to near zero or negative.

• No real evidence of a uniform lift in reading comprehension. Bowers notes that claims attributing England’s rise in PIRLS rankings to phonics are lacking in evidence. For one thing, England’s performance is mixed overall; state vs private school disaggregation (and all of the demographic differences that underlie such distinctions) undercuts the clean narrative. Also, England’s standing before the phonics mandate was already strong, muddying perceived improvements. Meanwhile, the gains are demographically selective.

• Subgroup differences complicate claims. In the data, some positive long-term effects appear for nonnative speakers or economically disadvantaged children in some programs, but these are small and inconsistent, and some other groups (conspicuously, advanced native English students, that is, the already high-performing) showed weak or even negative associations. Bowers cautions that these mixed subgroup findings make any simple “phonics = social justice gains” arguments harder to sustain.

• Confounds in inference. Because the policy change came with massive investment in teacher training, monitoring, materials, and exposure time, you can’t isolate phonics as the sole causal driver. Bowers argues that the evidence is compatible with multiple contributing factors, making the claim “we introduced phonics, reading soared” too simplistic.

Bowers doesn’t claim that letter-sound knowledge or phonics is unimportant. On the contrary, he accepts that grapheme-phoneme correspondence - which is kind of the core deal with phonics, which is why I semi-seriously wish we would just refer to it as “sounding out” instead of phonics - is a real part of reading. What he resists is the oft-heard policy narrative that systematic phonics instruction is a one-size-fits-all, overwhelmingly superior approach over all meaningful alternatives. His perspective is that the evidence base simply does not support the strong claims made by many advocates. The effect sizes are modest, many of them may be inflated by bias or outlier studies, long-term evidence is weak, comparisons to realistic control practices are murky, and large system-level policy rollouts do not deliver the clean, dramatic gains often promised. Bowers ends by urging researchers and policymakers to “temper confidence,” resist overselling, and seriously explore and test “alternative approaches to reading instruction” or more hybrid/multi-component literacy models.

---

So yes, phonics is useful. It should be part of every reading curriculum. The trouble is twofold: first, phonics is already part of essentially every reading curriculum. The idea that phonics is some secret language that only rich kids have access to is an absurd misrepresentation of the reality. The emphasis on sounding out over sight reading/whole language/etc. can certainly be meaningful, and we should pursue the most efficient and effective policies. But the effect size differences just aren’t big enough for the question to be existential for most students, and it just isn’t the case that a room of students are getting zero phonics thanks to woke teachers. Second, the case for phonics being the defining or sole reform needed is weak. The ridiculous rhetoric sometimes heard (“phonics is everything / phonics solves reading”) is not backed by evidence. And yet we have this absurd, politicized, “anti-woke” policy effort right now that insists that “the science of reading” proves that phonics is some sort of reading panacea. It’s not. You can give the less talented students phonics instruction and they’ll still lag far behind the more talented students. You can give the more talented students whole language instruction (the horror! the horror!) and not only will they still learn to read, they’ll probably read better than the former group. And the most talented students never needed any formal reading instruction to learn to read anyway. This is the reality we live in.

If we treat phonics as a panacea, we misallocate effort. We may hire more “phonics specialists” and adopt regimented phonics programs, but if vocabulary, comprehension, reading culture, content knowledge are ignored, the comprehension gaps will remain. Also, expectations become distorted: when students fail to learn to read, people blame teachers or schools for improper implementation of phonics, rather than acknowledge that phonics can’t do everything, especially under resource constraints. I must stress this point again and again: there are hundreds of American schools that have used phonics for decades, and there is no reason to believe that they have anything other than the typical performance distribution we expect in American schooling in general. There are plenty of failing schools and districts that have used phonics extensively throughout their modern history, and they still churn out students who can’t read. Because the idea of phonics as some sort of prophylactic against illiteracy is absurd overreach.

In short, phonics is an important tool, not a magical one. The evidence gives it credit for moving certain levers (decoding, accuracy, nonword reading), but it repeatedly shows that phonics power for comprehension, reading speed, long-term retention, and transfer to content reading is limited. Phonics gives students a better opportunity to move to meaningful reading comprehension, but it guarantees nothing, and indeed the weak relationship between phonics instruction and comprehension metrics shows that plenty of kids who learn to sound out words nevertheless fail to ever develop as mature readers. Meanwhile, the restrictions to the conclusions we can draw based on learner age, learner relative ability, and the specific developed skills are what they are. Policy that treats phonics as more than what it is - phonics is a part, not the whole, and not in any way a magic balm for those students who already fail at incredible levels - invites both overpromising and under-serving students.

---

So sure - I agree there’s a strong evidence-based argument for using phonics to teach reading to students who need formal instruction in order to read, and that such instruction can offer real if modest advantages compared to some other approaches for young, struggling readers who need help decoding at the word level. Beyond the earliest grades, outside of lower-level learners, and for tasks beyond decoding and into real comprehension, the phonics advantage collapses. Now please look at the missionary zeal in that original Argument piece about the so-called “Mississippi Miracle” and the notion of phonics instruction (plus the true god of ed reform, “““accountability”””) as a heal-all. Guys! Phonics as a pedagogical method goes back to the 17th century! There’s nothing remotely revolutionary about phonics, and the idea that phonics is some sort of pedagogical secret sauce that the rich white schools have been hoarding is an absolute absurdity. The same dynamic persists: I’d much rather be a highly-talented student learning under the less-empirically-robust reading systems than a less-talented student learning with phonics. That’s not an argument against using phonics! But most students already learn phonics in some form, and it’s no magic bullet. I beg you, calm down. Just calm down.

For the record: it strikes me as deeply suspicious how phonics is being recast, almost overnight, as a kind of pedagogical talisman in certain wonk circles, as though once you install “strong phonics instruction,” the rest of the reading crisis just vanishes. I’m perfectly open to a phonics-heavy approach - indeed, systematic phonics is one of the more defensible components in the “science of education” literature - but I am bewildered by the sudden elevation of phonics to near-religious status. The empirical record simply doesn’t support the claim that phonics is a panacea for literacy woes. And you already know how little I think of “accountability,” which was a policy obsession for the first couple of decades of this century, to no discernible advantage.

I also find the Mississippi conversation frustrating because I keep referring to education scandals where the problem was missing students or missing data and being hit back with “but look at the student data.” The whole point is that the data walks away! That’s been true again and again in various failed educational miracles - eventually, the effect has been revealed to be an artifact of hiding the football, of complicated and multisite efforts to push poorly-performing students off the books. That’s exactly what happened in Texas, and the lengths that were gone to there demonstrate the extremes that people are willing to go to in order to manufacture miracles. No, of course I can’t prove that the effect is illusory, and for the record I’m prepared to be proven wrong. But it’s going to take a ton of confirmatory evidence, longitudinal investigation, independent assessment, and time to confirm that this is really happening. Because as I will not stop pointing out, we’ve seen this movie before.

That points to one more essential point: the demise of local journalism makes this kind of large-bore educational debate so much harder to sort through. The unraveling of the so-called “Texas miracle,” the fable of miraculous test score gains and plummeting dropout rates in Texas schools, didn’t come from federal investigators or think-tank audits; it came from dogged local reporters and an initial courageous whistleblower. It was The Houston Chronicle and the local CBS affiliate KHOU-TV, not some national policy outfit, that painstakingly revealed the cooked numbers and the pressure on principals to falsify data. That kind of accountability journalism is intensely local, slow, and unglamorous, and it’s exactly the kind of work being extinguished by the collapse of local newspapers. When a city loses its newsroom, it loses not just coverage of zoning boards and school board meetings, but the basic capacity to notice when the official story is a lie. The next “Texas miracle,” if it’s happening right now, may never be exposed simply because there’s no one left to dig. What local journalistic enterprise in Mississippi, do you think, is fulfilling this function right now? I don’t know.

I’m confident that the supposed miracle in Mississippi is in fact not what it seems, probably a matter of some sort of data manipulation, likely in part due to some degree of systemic fraud and partially due to grey-area self-interest, institutional inertia, just-following-orders, etc. Could be wrong, but that’s my strong suspicion. Still, I’m not at all convinced that we’ll ever expose this fraud. Too many people are invested in finding the cheery, false outcome. This is precisely what I predicted at the end of my book The Cult of Smart:

I think, in the words of the television show The Wire, that we’ll juke the stats, that as a society we will communally realize that we are sick of our educational quagmire and decide that we have actually solved our problems. Some new narrative will emerge that will allow us to rest easy, some cheery story about how, if you look the right way, we’ve already solved our educational “crisis.” I have no doubt that this narrative will serve the interests of the charter school scam or some other form of educational profiteering. Those who look at education as a means to siphon public money into private hands are better funded, better organized, and better connected than those of us who defend our traditional schools. Public education— the radical idea that we can guarantee an education for all children, funded through taxation, instituted through government-run schools, and subject to the oversight of voters, taxpayers, and parents— is mortally threatened in this country. But however we choose to lie to ourselves, the basic brokenness of our system will endure.

1. A common site of this kind of error can be seen in research that attempts to measure the impact of test prep on scores of standardized entrance exams like the SAT. Such effects are commonly dramatically overstated because they don’t account for test-retest improvement; when you give test takers a second crack at an exam like the SAT, there is a generic (albeit small) advantage relative to the first time, simply from an increase in comfort and familiarity with the process. Gains commonly ascribed to test prep classes and tutoring typically fail to account for this reality and thus overstate the impact of these interventions, which are generally minimal. For the record, there are not usually any additional gains to be had from taking such tests additional times past two.

2. That paper (Lipsey and Wilson) is a pretty remarkable document that seeks to create contextual effect sizes for comparing different interventions based on real meta-analytic data; I think it’s easy to have profound misgivings with this kind of approach while still recognizing the immense effort here. More to the point, this kind of comparison is common in this kind of analysis and so useful here - several of the most commonly-cited papers in this literature make reference to this kind of generic advantage, so it’s important to include in my analysis.