I Am Begging Sports Media to Stop Misusing Regression to the Mean

I quite like the Check the Mic podcast with Steve Palazzolo and Sam Monson, formerly of Pro Football Focus and now associated with the 33rd Team. You should check it out and subscribe and all that stuff. There’s a tic in this clip that drives me crazy, though, and it’s something that’s way way too common in the football media and sports media generally, whether at the Athletic or the Ringer or wherever else. Palazzolo assesses the potential impact of a new defensive coordinator on the fortunes of the Cincinnati Bengals, who last year had an explosive offense and a hapless defense. At around the 3:00 mark Palazzolo suggests that new defensive coordinators often inspire better performance from NFL defenses even when the personnel does not significantly change and so the Bengals should expect some positive regression to the mean on defense.

To which I say, no no no. That’s not what regression to the mean is! If the change in underlying conditions represented by the new defensive coordinator is the reason for the expected improvement, it’s not regression to the mean.

Again, I’m not trying to pick on Palazzolo here because this is an absolutely ubiquitous thing in football media. But it is indeed wrong to describe a specific underlying change in conditions and then say that it will drive “regression to the mean.” Definitionally, regression to the mean refers to an expected trend towards average results purely as a result of underlying variability. Expecting regression to the mean, at heart, stems from the simple fact that past results that were statistically unlikely due to underlying variability remain unlikely for the same exact reasons they were unlikely in the first place. This is fundamentally an observation about the impact of variability, sometimes referred to as a luck or chance or similar. So consider fumble luck, or the propensity of a team to recover its own fumbles rather than see the other team recover them, which is generally considered random. Random binary events are expected to break down at about 50-50, but because we live in a world of variability, outliers happen all the time. The point of the concept of regression to the mean is that outcomes that were unlikely given underlying variability are unlikely to be repeated for that very reason. So if an NFL team has unusually good/bad fumble luck one season, which can be very meaningful given the impact of turnovers on win percentage, we should not expect that good/bad luck to continue in the following season. If in fact the team’s fumble luck reverts to the more expected 50-50ish outcome, that is textbook regression to the mean.

Think about it with a classic Intro to Statistics class example. If I flip a coin 100 times the expected outcome is that I will get close to 50 heads and 50 tails. But outliers happen fairly often. (Indeed, one of the paradoxical aspects of probability is that over enough repetitions it becomes very improbable that you will only ever see the probable result.) So imagine I flip the coin and I get 80 heads. That might not sound that wild to you, but the odds of this happening are less than one in two billion. So if I again set out to flip the coin 100 times, I’m going to strongly suspect that I will get significantly fewer than 80 heads, even though the coin has not changed. My confidence in the likelihood that the coin will give me fewer than 80 heads is a product of my understanding of the underlying variability. That’s regression to the mean. Things that are unlikely remain unlikely absent changes to the underlying conditions.

If, on the other hand, between the first 100 flips and the second 100 I doctored the coin to be more likely to come up tails, it would be nonsensical to call the resulting difference in number of heads regression to the mean, because that difference would not be the product of expectations of likely outcomes given consistent underlying conditions. Regression to the mean is interesting and useful precisely because (usually, on average!) it enables predictive power without assessing any underlying change.

And so it’s strange to see, as I constantly do, NFL analysts saying things like “Their offensive line was great last year, but they lost their Pro Bowl right guard to retirement and their tackles are getting old, so we should see some regression to the mean there.” You’re not predicting regression to the mean if you’re arguing that outcomes are going to change because of changes to underlying conditions! If you think a team is going to get better or worse because of new players or coaches or players improving or getting worse or because of specific injuries or similar, that’s just definitionally not invoking regression to the mean. If you think a player of team has gotten better or worse because of some tangible change, you just… think they’ve gotten better or worse. If, on the other hand, you think that injuries are random - debatable, but just go with it for now - and you observe that a team got particularly unlucky in that regard last year, as happened with the Detroit Lions, you may justifiably say that they’ll likely be luckier next year, and that is an argument towards regression to the mean. “Detroit’s defense should outperform last year’s squad because the numbers say they are very unlikely to be as ravaged by injuries” is an invocation of regression to the mean. “Detroit’s offense should be worse than last year’s because they lost several of their starting interior offensive linemen” is not. The difference matters.

A big part of the problem here is that football people constantly talk about “regression” in a casual way, without explicitly saying that they’re talking about regression to the mean, which I find very unhelpful. Yes, it’s true that if you think something is going to get worse, you can describe that as regression even if you aren’t talking about regression to the mean. But in a context where people are often talking about regression to the mean specifically, that seems to invite unnecessary confusion, and a lot of sports media types seem to overuse the word “regression” simply because it sounds fancy, like an insider’s term.

There are other pitfalls. Regression to the mean, as a concept, is almost custom-built to inspire people to fall for the Gambler’s Fallacy, or the belief that past unlikely outcomes somehow make future unlikely outcomes even less likely. The classic example of this kind of thinking is the man who walks around the casino floor, looking for roulette tables that show runs of many reds or blacks in a row. Look, over there! A table with seven hits on red in a row! Surely that’s a highly unlikely outcome; surely eight in a row would be even more unlikely. So he rushes over to place a bet on black. But of course the table doesn’t “remember” that it’s come up heads seven times in a row, and the odds of getting black are the exact same 48% as usual. (And that’s exactly why casinos put up those signs that track recent roulette outcomes.) A lot of casual invocation of regression to the mean, in sports, implies this kind of magic ability for the past to influence the future; many people on social media talk as if regression to the mean makes it more likely that we will see an expected result rather than an outlier. But the odds of an outlier are exactly the same if the underlying conditions remain the same. Regression to the mean is not a force in the universe and cannot make anything happen. And of course we should always be aware that while a second outlier in a row is less likely than seeing the expected outcome, such repetitions can and do happen. A team can have two years in a row of great or terrible fumble luck. We live, after all, in a world of variability.

Then there’s the tendency of some people to talk about regression to the mean as if it implies reduced variability over time - as if, because of regression to the mean, we can expect all sports statistics to gradually converge to the overall mean. Which, uh, is not what happens! Again, outliers are not just possible, in a large enough system outliers become inevitable, and that will never change. You can have an outlier of a game or stretch of games or season, see that result regress to the mean, and then have an outlier of as large or greater effect immediately after. Variability has its say; if it didn’t, sports would be very boring.

Seriously, though, give Check the Mic a listen if you’re a football obsessive like me. It’s well worth your time.