The Consensus Draft Board Debate is Sports Culture War at Its Worst

the loud and uniformed vs the overconfident and aggravating

Getty Images

Some, Jesus, twelve years ago I argued that the NBA had become a league for snobs. Though people still talked about NBA culture as if it were all old-school types shouting “you have to take it to the hoop!” at the television, commentary about the league had at that point become impossibly snobby. There was indeed a divide between traditional analysis that extolled the virtues of post play and stars taking two-point shots that emerged from isos, on one side, and those who extolled the newer wisdom of heavy reliance on threes and layups, on the other. But the conversation was plagued by a dynamic in the three big American sports that has aggravated me for decades: the latter side had won an overwhelming victory among the media, in team front offices, and even within the overall NBA fanbase, and yet partisans for the analytics side constantly acted like they were a beleaguered minority, endlessly complaining and nakedly self-celebratory. As in in American society writ large, the nerds had won by any possible measure and yet still complained about how oppressed they were.

We’ve just emerged from NFL draft season, that time when America’s biggest ratings draw produces a giant ejaculation of commentary, commentary which drenches the entire sports media and leaves it sticky for weeks. The sheer amount of draft coverage is unfathomable. And because this is sports in the 21st century, where public debate lives on social networks, the commentary about the draft and draft analysis is tribal, ugly, and stupid. The same old dynamic plays out, just as it did with baseball and basketball in years past: the analytics crowd is largely correct on most of the individual points of contention, but they overstate the lessons of their own analysis, refuse to address exceptions, and (in particular) act like assholes on Twitter, making it harder to support them when they’re right. It’s a constant war between people who are mostly wrong and people who are mostly right but sometimes wrong who have made being right the entirety of their identity. And I frequently wish both sides could lose. A perfect example can be found in the recent debate about the consensus draft board and whether teams should let it guide their picks.

The Consensus Board is Like an Index Fund, For Good and Bad

Think of the consensus draft board as the index fund of the NFL Draft. Just like an index fund doesn’t try to pick winners in the stock market but rather attempts to aggregate the entire market and give you a weighted average, consensus boards, most notably the one Arif Hasan has been putting out since 2015 or so, takes scores of individual analyst rankings and averages them into one composite ranking of prospects. The individual stocks, in this analogy, are not players but analysts, and the question “which will be the most profitable” becomes “whose mock draft will prove to be most prescient?” With the consensus board, you’re not betting on any single scout’s hot take; you’re buying the whole market’s collective opinion. And like an index fund, it has the appealing property that, on average, it beats most of the active predictors it’s made up of. Data shows that the consensus board has historically outperformed the median individual analyst at predicting where players actually get drafted. So when someone says a team “reached” or got “good value,” what they usually mean is that the team deviated from the index in a meaningful way - they took a player ranked 70th on the consensus board with the 40th pick or grabbed someone who slipped well below their expected slot. It’s certainly a defensible benchmark, and that’s exactly why so many people treat it as the default.

The question is whether always buying the index is actually the right strategy in a market this inefficient. And the answer, inevitably, is no.

You see, the consensus draft board is a very good example of a generally-helpful guide that’s applied to a system (the NFL draft) that a) is inherently noisy and b) has a low basal hit rate, which leaves us with a conclusion that should be obvious: the consensus draft board’s advantage is found in general and over many repetitions. Sample size is absolutely in play here. Applying the mathematically best strategy in poker can still result in losses, after all, and even the most quantitatively-oriented poker players will often still go with their gut from time to time. Over the course of many drafts, the consensus board will tend to produce strong results relative to most of the individual boards it combines, but this advantage exists in the aggregate. Saying that an individual pick is necessarily wrong because it deviates from the consensus board is like arguing that buying an individual stock out of proportion with its weight in an index can’t ever result in superior profits; of course that can and does happen. And yet every NFL draft pick that significantly differs from the consensus board now results in a mass roasting on Twitter, the cesspool that represents the sports world’s collective id. Nothing delights a certain class of sneering Twitter addict more than bashing a team for reaching on an individual pick, no matter what the larger track record of the team or GM. That’s precisely how not to think about the consensus draft board.

The consensus board is generally useful that it captures the rough market price of a draft prospect (sometimes very rough, in divisive drafts like this past one) and tends to beat the draft boards of individual analysts in predicting where players actually get selected. But “useful as a benchmark” is not the same as “deserving of slavish adherence,” and conflating the two ignores what the board is actually measuring. The consensus board is a prediction about how NFL teams will select prospects, which is a separate question from which prospects will succeed. The difference is accounted for in most consensus models, but it’s an inherently dicey prospect in a system where draft experts always have an eye on someone else’s work. Hasan himself splits his contributors into “forecasters” (predicting where players go) and “evaluators” (assessing who is good), and the forecaster rankings cluster more tightly because they’re working off of a lot of shared information. The herd can certainly be wrong about a player’s likely NFL future. (See: Jadaveon Clowney.) Similarly, a lot of people have argued that the price of tech stocks has been screwy for many years, and a lot of people believe that the fundamentals are out of line with the current price of oil. Market prices can diverge from reality, but that’s a different question than trying to price the market. Likewise, the consensus board is closer to a market price than an oracle on quality.

That distinction matters because what we actually care about is how draft prospects perform in their NFL careers. After the draft, who went in what order is largely a matter of trivia. But predicting actual success in the NFL is obviously hugely important - and it’s brutally, brutally tough business. First round quarterbacks (the most important picks of all) hit at roughly 35–45%, depending on how you define success and the data you want to look at; regardless of where you look, a majority fail to become winning starters. RotoWire’s analysis of 800 first-round picks since 2000 found that 52% of all players taken in the top-10 ever earned Pro Bowl honors, while the first round in general saw a 29% bust rate, remarkable figures given how much teams covet first round draft selections. The fact that whether a first round pick succeeds is essentially a coin flip often blows people’s minds, given how every much high-profile prospect is hyped in the leadup to the draft. And true stars become rare the deeper you go in the draft. About 0.5% of players drafted in rounds 3 through 7 across two decades made two First-Team All-Pros. For every Brock Purdy or Terrel Davis, there’s dozens and dozens of guys who never get a second contract in the league and many who never make the active roster. There’s an obvious conclusion to draw from this: most players in most drafts are not going to be quality NFL starters, let alone stars, and so the baseline for success is low no matter how well you draft.

This has serious consequences for the consensus board debate. If the system you’re trying to predict is a coin flip at the top of the draft and essentially a lottery by the middle rounds, then a board built by aggregating people who watch the same tape, attend the same Senior Bowl and combine, and read each other’s mocks cannot meaningfully transcend the noise floor of the underlying problem. (More on that last bit below.) Refusing to deviate from a benchmark that’s wrong roughly half the time isn’t an act of discipline, or not one you’d want to achieve, anyway; you’re just guaranteeing that you replicate the median outcome in a system where the median outcome is mediocre relative to the other teams and bad relative to the hope that your prospects will mostly pan out. The whole point of having an evaluation process is to occasionally beat the market. If you anchor entirely to consensus, you’ve built an apparatus that by its basic architecture can never outperform it - and, again, the market usually fails.

To put it another way, if the median first-round pick is roughly a coin flip to become a meaningful contributor, then the consensus board inherits all of that noise. Aggregating noisy estimators reduces variance around their mean - skinnys the spread - but it doesn’t inherently move the mean closer to the truth if the underlying signal is weak, as it is with draft data. If the signal was stronger, the hit rate wouldn’t be as low as what the NFL draft produces.

To return to the index fund analogy, then, you can think of consensus draft board orthodoxy as the belief that you should never pick individual stocks, only buy into index funds… only in a system where the entire market largely fails, rather than succeeds! Telling investors not to pick and choose individual stocks is wise because the market generally and predictably goes up over time. Index funds are thus a far safer and more defensible investment compared to the notoriously low-return practice of picking individual stocks. But giving the same advice to NFL teams picking players is a much less sensible proposition, because the overall “market” of NFL draftees has seen much less success over time than (say) the S&P 500. You can of course argue that a pick that’s truly outside of the consensus is a pick you could have gotten in a later round and that you should deploy your picks more strategically. But if a team feels truly confident that a particular player is far more valuable than the consensus board implies, taking him isn’t some sort of embarrassing error, and the people who suggest that it is are showing their ass.

Like I said, the consensus model’s value lies in sample size and repetition. If you want to get mad at 49ers GM John Lynch for so consistently stepping outside of the consensus with his picks, I wouldn’t object; that’s a sustained pattern of behavior. James Gladstone of the Jaguars has shown a pattern there too, though he’s too new in the position to be sure of his long-term tendencies. Criticizing teams and executives that have a history of reaches relative to consensus is fine. Blasting a team for picking an individual guy who was “supposed” to go 30 picks later based on the consensus board simply misunderstands the whole enterprise. It’s a good idea to just buy into an index fund and watch the market slowly move up over time… unless you’re the beneficiary of insider trading and you know something the market doesn’t. It’s the same with the NFL draft; the consensus board is a good guide most of the time, but if you feel very confident that you have information the other teams don’t, you should take your swing.

You Can’t Really Justify the Consensus Board with Bayes

The defense of the consensus board usually leans on the wisdom of crowds and, well, The Wisdom of Crowds, James Surowiecki’s classic 2004 big-think book that argues that the estimates and predictions of many people are generally better than those of any expert. Aggregate enough independent estimates and the errors cancel out, leaving signal, or so the idea goes. I’m not sure that The Wisdom of Crowds is often explicitly cited by consensus draft gurus, but certainly that kind of thinking is prevalent in this debate. But here’s where I get back to the problem with everybody reading each other’s draft boards: Surowiecki is explicit that this dynamic only works under specific conditions, the most important of which is independence. For the crowd to be wise, its estimates have to be drawn from genuinely separate information, separate reasoning processes, separate intuitions…. When estimates are correlated, when everyone is reading each other’s mock drafts, insider tips, and interviews, aggregation no longer cancels error; it amplifies whatever shared bias the cluster carries. And that’s exactly what the NFL mock draft game entails, everybody looking at another kid’s test.

Hasan has acknowledged this himself; the forecaster boards show “strong evidence indicating that many of them are working off of the same information,” with similar clustering year after year. A hundred correlated boards is not a hundred independent samples but a bunch of closely-related projections derived from a lot of the same information. Crowd wisdom assumes you have something like the former, but the consensus board is structurally closer to the latter.

This is also where a Bayesian justification collapses. Bayesian updating requires the new information to be informative and (mostly) independent. NFL draft information fails both tests: it’s heavily correlated across the same info (college production, 40 time, hand and arm measurements, etc) and thus not independent, while much of it has weak predictive value to begin with. For example, Sports Info Solutions found running back combine scores correlate poorly with production despite driving major draft-capital decisions, and you can find many similar examples of data suggesting that specific draft info isn’t useful. This gets to a broader condition of modern life, where we’re drowning in information without seeing tangible benefits from it in many contexts. Teams and analysts now have access to cognition tests, GPS data, advanced college tracking, health information that previous generations would have killed for… and yet hit rates haven’t meaningfully improved. The simple Bayesian story (more data, better probability distribution) only works if your data is informative and independent. NFL draft data is often neither, which is why a board that aggregates it can make people more confident without making them more correct.

The Typing Penis Problem

I should say that, in general, the analysts aren’t the ones overstating the confidence we should feel about the consensus draft board and the actions it implies. Sometimes they are, but usually it’s this phenomenon where Twitter randos and similar hear statistical terms and try to apply them without understanding the underlying theory. (This is, to pick a common example that’s my own pet peeve, a big part of the absolute constant misunderstanding of regression to the mean in sports.) Hassan frequently refers to the consensus board as an expression of modesty, and he is better than many about accepting that there are deviations from it that are likely to prove to be the better move. So is this all just the fault of stupid eggs on social media, a problem that afflicts the entire internet? No, I don’t think so. Because NFL analysts sure do contribute to the Typing Penis problem, and I think Hassan is a good example.

Hassan seems bright and levelheaded in his longform writing and has a good newsletter on this platform. But on Twitter, where all of this discourse plays out, he’s a certified Typing Penis, spending a lot of time belittling low-follower accounts with team helmets for avis. Indeed, like a lot of new-school sports analysts, Hassan’s Twitter style almost seems designed to prove every stereotype about statheads correct - he comes across as condescending, rude, and seemingly uninterested in convincing any skeptics of anything. This has been true of the sports analytics revolution for decades, with baseball and basketball culture also giving rise to an expert class full of guys with revolutionary ambitions and a profound lack of interest in reaching out to the unconverted. It’s an archetype that goes all the way back to Bill James. It’s bad enough when a genuinely minority point of view spends all of its time dunking on people. It’s so much worse now that the nerds have won an unconditional surrender. Like Mark Zuckerberg still whining about being insufficiently adored, sports nerds engage in classic crybullying, the rage of the enfranchised.

Twitter is the crucible; every bad incentive and shitty habit on that awful site has come to permeate how we understand sports and argue about them. And it turns generally reasonable people into the worst version of themselves. Sam Monson is perfectly pleasant on his podcast Check the Mic, I remember him being unobjectionable back when he used to do video for PFF, and yet his Twitter persona is certified Typing Penis. It’s inexplicable, especially given that his cohost Steve Palazzolo embraces the much more pleasant Sleepy Italian paradigm. ESPN’s Bill Barnwell is a good analyst with a lot of insight, but his Typing Penis quotient is off the charts on Twitter. And his colleague Seth Walder, my god! The man seems interested in football only to the degree that it gives him an opportunity to tell other people how much smarter he is than they are. Classic Typing Penis stuff. (Their fellow ESPN analyst Ben Solak, however, is not a Typing Penis; his archetype is Dorkus Malorkus, a much better thing to be.) Blank derision is the lingua franca of all social media, but that doesn’t make this stuff any more justifiable.

The very structure of social media is most to blame here, but so is analytics culture, which across sports and deep into decades of success has never grown up, has never moved on from its sneering, Nerds > Jocks framing. And at this point, you have to wonder why. You guys won. Maybe enjoy it a little bit? I do understand though that there’s clearly some bunker mentality involved; no matter what your perspective, people on Twitter are going to scream at you. So maybe they adopt their manner of engaging thanks to that dynamic. But… like I said, no matter what your perspective, people on Twitter are going to scream at you. That’s just the nature of the beast. Of course Twitter randoms are ignorant dickheads; I expect smart analysts to behave better than they do! That seems like a low bar. And what you lose by being a Typing Penis is opportunities to persuade. Consensus draft discussions are a particularly loss because consensus draft boards are a) useful but b) subject to a lot of error. That makes sneering dismissal of alternative opinion unfortunate in both directions.

If you’d like an example of a better way to do football analysis generally and draft predictions specifically, consider the Ringer’s own version of the kids from the movie Good Boys, the Ringer Fantasy Football Show. They love football, they’re deeply engaged with the league, they have a lot of good insights, and they’re analytically informed, but they are also unpretentious, funny, and self-deprecating. I’ve learned real insights from the show, but I also appreciate their ethos, which holds that football shouldn’t feel like homework. Danny Kelly is a very well-informed draft guy, but he never acts like he knows everything or suggests certainty in a process that will always be filled with variance and failure. In general, the podcast’s hosts manage the feat of taking football seriously without taking themselves too seriously. And after another months-long, bruising draft season, I wonder why so many others struggle to do the same.