Charter School Lotteries Are a Black Box. Why Don't Charter School Advocates Care?

charter schools are only as good as their lotteries, which we have no reason to trust

City on a Hill Charter School student volunteers check names of future students drawn from a list of hopefuls on lottery day, March 12, 2014

In education the great enemy of effective analysis and fair decisions is selection bias, as I wrote about at length here. Selection bias refers to research outcomes that are the product of systematic differences in the way studied groups are assembled rather than of intrinsic differences between the groups. The classic example of falling for selection bias in education is to look at outcomes from two different school districts and conclude that because one district has better test scores than another that district must have better schools than the other. Public school districts are the product of zoning and municipal borders; zoning and municipal borders sort children into different districts non-randomly. The districts aren’t teaching children with the same underlying characteristics and so any conclusions drawn about them or their methods will be unsound. Hartford (median income $22,266) is right next to West Hartford (median income $52,472). Comparing their schools by their raw outcomes would be unwise as well as unkind.

You’d like students to be sorted into schools as randomly as possible for research and equity purposes, but for obvious political reasons this is not feasible. Differences in student distribution into different schools are often cited by reformers as a key source of injustice, and many want to do away with traditional districting¹, but that’s not going to happen anytime soon. Regardless of whether the comparison is across districts, between schools within districts, or between private and public, I advise parents all the time that trying to discern the quality of a school (whatever that might be) is much, much harder than they assume. In my experience it’s nearly impossible for people to get past “look at the colleges the private school’s graduates go to!,” which is the single dumbest way to evaluate a school.

Comparisons between charter schools and traditional public schools are potentially as subject to these corruptions as any other. The mechanism through which we are supposedly avoiding these are admissions lotteries. Typically parents sign their kids up (which is perhaps sufficient to break randomization itself, given systematic differences in how income groups select schools) and there is a lottery system where students are awarded seats or not, with the losers left to go back to the public school system. These lotteries are the way we ensure different publicly-funded entities are treated equitably, they are the way that we create true diversity in student bodies, and they are often the way that academic researchers establish that the effects they are investigating are the product of random selection. Because of this importance, charter school people are often fiercely defensive about them.

Almost all states formally require lotteries. As a (rather defensive) charter school advocate explains in the Washington Post

Nearly all of the 43 states and Washington, D.C. that have charter schools require that random lotteries be used to select students if there is not room for all that apply. Three states, Arizona, Colorado and Wyoming, don’t require lotteries, but even there officials said charters are using either lotteries or other impartial ways of admitting students.

This is not inaccurate, but it demonstrates what is frustrating about the way that charter advocates talk about the lotteries. They simply refer to the fact that schools have them, or that state law requires them, without reference to evidence that there is consistency, transparency, or lawfulness in the processes themselves. Many will dispute this statement, but since I began investigating this question in earnest in late 2017 I have found this conclusion to be inescapable: we simply do not have good public information on the real-world conditions of charter school lotteries. They are hugely important for the children who are being sorted and for the research that informs our future policy, but I can tell you from researching my book that getting solid information that goes beyond parroted lines about the official process is quite difficult. Repeated efforts to research accountability mechanisms have mostly turned up a lot of blanks. What’s really going on in these vitally-important events? I don’t know.

Not only is there a lack of consistency between different states in charter school lottery practices, from what I’ve gathered there is also a lack of consistency within some states and even within some municipalities. There is also a profound lack of transparency in many of these lotteries; they occur behind closed doors, often undertaken by charter employees who operate under less regulation than their public school peers. That really is the worst of all - in many cases the schools themselves are handling key parts of the lottery process². They have direct financial incentives to cheat, and yet in many cases they have multiple opportunities to influence the results. When you give people incentives to break the rules and you trust them with enforcing those rules themselves, they’ll break them. (Certainly including public school educators.) The honor system is not a check on cheating.

And cheat they do. At least 253 charter schools in California were found in 2016 to be illegally manipulating their student bodies to exclude those who were least likely to succeed and enroll those who were most. A 2013 investigation by Reuters reported that

across the United States, charters aggressively screen student applicants, assessing their academic records, parental support, disciplinary history, motivation, special needs and even their citizenship, sometimes in violation of state and federal law.

There are many individual cases of charter school admissions misconduct that you can investigate, along with other types of fraud. In fact the charter system is so rife with graft and mismanagement that it was reported in 2019 that the federal Department of Education has wasted $1 billion on various shady or incompetent charter schools, including many demonstrable cases of charter schools deliberately manipulating their student bodies to exclude undesirable students, in exactly the way that I fear.

If these problems have cropped up repeatedly in the past, and there’s been no systematic attempt to address them - and there has not - there’s no reason to think these problems don’t persist. And anecdotal evidence of entirely unrepresentative charter schools can be found easily. AstralCodexTen recently ran a review of my book and several commenters shared that they had experience with shady lotteries. Here’s a quote from one of the comments, from a recent charter school student:

I attended a charter school all 4 years quite recently. Admissions was entirely by lottery, open to everyone in the district. I can tell you that even in freshman year, the student body was not even remotely close to representative of normal kids; it was basically an entire school of the kids who would normally be in gifted / accelerated programs. And by graduation, it was even more refined to super talented & smart people, because the students who left to go back to their local normal schools were mostly from the rear of the pack….

[Despite the lottery, admission was selected by] whose parents were involved enough, interested enough in education, valued education enough. Simple as that, I think.

And from a book review by Michael Pershan linked in that same post:

The common criticism leveled at Moskowitz and her schools is that they cherry-pick students, attracting bright children and shedding the poorly behaved and hardest to teach. This misses the mark entirely. Success Academy is cherry-picking parents. Parents who are not put off by uniforms, homework, reading logs and constant demands on their time, but who view those things as evidence that here, at last, is a school that has its act together. Parents who are not upset by tight discipline and suspensions but who are grateful for them, viewing Success Academy as a safe haven from disorderly streets and schools. Charter schools cannot screen parents to ensure culture fit, but the last hour in the auditorium is a close proxy for such an effort, galvanizing disciplines and warning off the indifferent and uncommitted. A the same time, there is something undeniably exclusionary about it. If you don’t have the resources to get your child to school by 7:30 and pick her up at 3:45  — at 12:30 on Wednesdays — Success Academy is not for you. Literally.

Since I wrote a book about, among other things, the folly of expecting adjustments to school-side factors to fix what are student-side problems, I’ve heard from a lot of career educators. These emails have been the most meaningful reward from a book that didn’t sell very well and was barely reviewed. (Well, OK, and the advance, obviously.) I wasn’t surprised to hear from a lot of public school teachers, who as a class are exhausted from weathering the constant attacks against them for little financial reward. (The median teacher in Mississippi takes home $630 a week.) But you’d expect them to like my book, right? It’s exonerating them.

What has surprised me though is the number of charter school teachers who have reached out since the book was released, over a dozen. They demonstrate a range of opinions about the charter school model, as you’d expect, but they are almost universally more critical than I’d expect. (Of course, since we’re talking about selection bias, I should note that dissatisfied charter school teachers are much more likely to read my book and contact me than the average charter school teacher.) A consistent source of sheepish criticism of their schools from these teachers is that their job is too easy. That is, they enjoy working for charter schools because their kids are overwhelmingly diligent students with committed parents. But this personal advantage demonstrates the dishonesty in the system: they are perfectly aware that were students truly randomly selected into their schools they would have more problematic students with problematic parents, as such students are a fact of life. None of the teachers discussed the lottery system as such, but most reflected on the fact that their students are not the same as those at most schools, including two who had previously worked in public school systems and noted that the difference was night and day.

This is all to say nothing of grey-area student body manipulation practices, such as through charter school applications themselves. A classic way to manipulate who applies is with application requirements that are so onerous they strain credulity. Consider this mountain applicants had to climb to apply to a Los Angeles area middle school:

The “Getting to Know You” sections require would-be students to write five short essays covering two pages (“use complete sentences”) on a variety of topics (“Tell us about your family”). Then there’s a third page calling for short responses on an additional six issues (“The qualities and strengths that I will bring to school are… .”).

Wait, wait. We’re just getting started. The parents have to write seven little essays of their own and then fill out the child’s medical history, including medications (an intrusive request that some critics say violates federal privacy law) — and remember, this isn’t for an accepted student to attend, but for a student to apply in the first place. It’s capped by the would-be student’s minimum three-page autobiography, typed, double spaced and “well constructed with varied structure.”

There’s also backfill. Backfilling refers to what you do when students drop out of the school - do you fill in those seats, or don’t you? If you don’t fill them in, you’re going to look better by the numbers. The students who drop out of K-12 schools are the more transient by definition and the more transient students tend to be the more academically marginal. Public schools generally have no choice but to backfill given their enrollment policies. Charter schools often get to choose and, unsurprisingly, frequently choose not to replace their poorly-performing dropouts with new enrollees who might prove to perform poorly themselves.

Media darling Success Academy charters are an attrition factory in part because of strict backfilling practices for grades past fourth (previously third). The first incoming Success Academy kindergarten class had 83 students. By the time those kids got to their senior year of high school there were 17 left. This success-through-attrition should not surprise us, coming from the home of the “Got to Go” list. (When Eva Moskowitz is asked about the disparity between the student body that starts and the student body that finishes at Success Academy, she dissembles.) The founder of the celebrated Boys Latin charter school in Philadelphia just straight up said that backfill is how they get the best students³. In general the sheer mass of students who start at charters but don’t finish is a scandal. KIPP schools are similarly ballyhooed, yet 40% of Black students leave before graduation. In the past Washington DC charter schools had expulsion rates more than 72 times that of their public school counterparts. Are you noticing a pattern?

In some cases we are perfectly well aware that this produces unequal student populations. We know, for example, that New Jersey charter schools enroll significantly fewer special ed students and that the students they do serve have less damaging disabilities. (The famous New Jersey CREDO study has numerous methodological issues, for the record.) The highly-regarded MaST Community Charter School of Philadelphia has 41% low-income enrollment; the city’s public schools have 91% low-income enrollment. In one Georgia town the local charter school is 73% white, while the public schools in the district are 12% white. What could be the source of these enormous disparities if not deliberate bad behavior?

(In terms of how some charters attract the kinds of students they do want, well, I’d prefer not to go for a dip in that swamp.)

Researchers are not blind to these problems, though I do find many of them bizarrely credulous towards the idea that these schools are conducting lottery procedures properly. It’s not clear though that individual research teams have the mandate necessary to really investigate the lotteries. MDRC (pretty standard issue Ford-funded neoliberal ed policy chopshop) says “In our SEED study we attended the public lottery and observed the process; we also walked through the lottery documentation with school leaders and traced students through the assignment process into enrollment the following fall (and beyond).” The problems here are that a) the act of observation changes the behavior of the observed and b) the vast majority of lotteries are not being observed by researchers. Of course no system is perfect and there will always be blind spots and the potential for fraud, but something systematic seems called for given that we are placing bets on our entire education system based on charter school outcomes.

There are statistical tests that are performed to try and ascertain the effectiveness of randomization procedures; I am not qualified to comment on them myself. But there are common methodological critiques of such techniques concerning issues like matching and dummy variables, and they can never provide an answer to the most basic questions of whether individual schools are actually following procedure and producing the outcomes they should based on that procedure.

The simple and essential question is, are the lotteries producing student bodies of charter schools that are significantly like those of their traditional public school counterparts? Do all students really have the same odds of being sorted into one school or another? I’m sure many will say that I don’t know the size or extent of charter school lottery issues, and that’s true, I don’t. And neither does anyone else. That’s the problem. What frustrates me so deeply is that charter school advocates seem totally unwilling even to entertain the question. Like I said above, there appears to be not only a dearth of information about the lotteries but a dearth of interest in them as well, despite the fact that highly consequential policy decisions are made every day based on the assumption that they are working as intended. All I am asking is for the extremely well-funded, loud and strident charter movement to open its books. What do they have to hide?

I often find myself wondering why someone like Jonathan Chait, a zealous defender of charter schools and husband to a charter school apparatchik, doesn’t carry this banner with me. After all, Chait definitely believes that the charter school advantage is real and replicable across the country. Since he does, he should have no problem calling for greater regulation and transparency in charter school lottery practices. He should be more passionate about this than I am! There’s no reason charter school people should have anything to fear if their theories about education are correct. If the charter school effect is real, then the students being deprived of its power by unfair lottery systems would be the victims of great injustice.

Look, I’m anti-charter. I’m not going to pretend otherwise. I am partly out of solidarity with public school teachers and their unions, which is common to the left, and more importantly because I believe that intrinsic academic talent is the greatest determinant of student outcomes⁴, which is most certainly not. (A piece on the latter is coming.) I believe that the observed effects are the product of the selection effects I’ve laid out here, and even if you think the effects are real, they are considerably weaker than commonly assumed. (This is their miracle.) But none of that should matter here. Proponents and critics alike should recognize the absolutely essential questions being discussed. I don’t understand why we would do another CREDO study before investing serious time and manpower into finding out if these schools are doing what they are obligated to do. The size of the effect is useless if the comparison is specious.

Since people have been talking about the importance of traditional news media lately, here’s something only they can do, something I can’t do as a newsletter writer and jamoke: start a major investigative journalism project into the actual reality on the ground of charter school lotteries. Investigate whether they’re anything like truly random. Find out just how much transparency there is about the process at as many schools/locales as you can. Interview students and parents about what the experience was like and whether they were given assurances before the lottery that their students would get in, which is common. Get charter school admins to go on background and discuss the process. Talk to state education officials and have them explain the standards for charter lotteries, then look at specific schools and see if those standards are actually enforced. If a newspaper or magazine can’t do this, maybe a deep-pocketed think tank should - if any are willing to engage in an activity that might make charters look bad. If they can’t or won’t do it, the Department of Education should.

Somebody tweet this at Chait. Somebody get ahold of Raj Chetty. If these guys are serious about actually producing better outcomes equitably rather than engaging in a massive exercise in juking the stats then they should be the first to ask whether these vitally important systems are functioning as designed. Spread the word: investigate the charter lotteries. Charter advocates should have no problem with that, if they really believe in the power of these schools.

1. I suspect it will not turn out the way they hope. Their narrative is that students are systematically excluded from the best-performing districts and perform poorly for that reason; they can’t get into the best schools and so their metrics suffer. A far simpler and more likely explanation is that the schools identified as best-performing are so identified because they systematically exclude the poorest-performing kids. The district borders are not the wall between underserved kids and better schooling; they are the instrument through which those schools become “better.”

2. I am prepared for a rigorous investigation to reveal that most charter lotteries are functioning as they should, even though this conflicts with my priors. But I will never believe that having employees of an institution handle a process (in part or in whole) that is meant to maintain the institution’s integrity and has direct and powerful influence over that institution’s prestige, finances, and continued operation is a good idea.

3. That link is worth reading in full to see how the sausage gets made at a famous charter.

4. Such talent could explain the differences in achievement between individuals, but not between groups like racial achievement gaps or gender gaps. As I will discuss in this space soon, I think the most parsimonious explanation for such intrinsic talent is genes, but I am not attached to that notion and there can be such a thing as intrinsic talent without genetic origins at all.