Is Bill Simmons a Superforecaster?

Bill Simmons is a great entertainer, but is Bill Simmons a superforecaster?

Serendipity struck when I listened to Bill Simmons and Joe House forecast their NBA Eastern Conference  over/under win totals podcast the same day I finished Philip Tetlock’s Superforecasting. That night, I wondered: Is Bill Simmons a good forecaster?

Simmons is a great podcaster. He knows a lot about NBA, he wrote a book about it. He talks a lot about gambling, so he may be good at that as well. These things seem to indicate that he’s a good forecaster, but is he really?

But before we answer that, I have to confess a near miss. 

I almost answered the wrong question.  Philip Tetlock, the author of Superforecasting, warns readers to be careful they don’t switch one question for another. Tetlock writes:

“Formally, it’s called attribute substitution, but I call it bait and switch: when faced with a hard question, we often surreptitiously replace it with an easy one. “Should I worry about the shadow in the long grass?” is a hard question. Without more data, it may be unanswerable. So we substitute an easier question: “Can I easily recall a lion attacking someone from the long grass?””

When I started thinking about Simmons and forecasting I nearly fell for this. I replaced, is Bill Simmons a good forecaster with is Bill Simmons an entertaining forecaster. The answer to the second question is yes. When I checked the iTunes store on 10/27/15, three of the top five podcasts were Simmons’.

What I needed was a scorecard. Something that would keep me focused on the right things and the right question. Something like this:

Tetlock’s book concludes with 11 commandments for aspiring forecasters. These act as a snapshot of the book. It’s just a glimpse. In the same way you wouldn’t confuse a vacation brochure (do these still exists? maybe today it’s a vacation twitter account?) for an actual vacation, you shouldn’t confuse these commandments for a thorough explanation of forecasting. 

Our plan: List each of the 11 commandments and see what Simmons or House says about the idea. The bold parts will be commandments from the book, quotes will be from the podcast, and my notes are whatever is left.

One final note: The line is the number of games the odds makers in Las Vegas say a team will win. If it’s a number like, 34.5, a bettor can choose the over (35 and above) or the under (34 wins and below).

Ready?

1 – Triage, focus on where your work will pay off. Simmons and House both name certain bets as “locks.” The guys recognize that betting on all the NBA teams is a lot harder than betting on some of them. Some of the Vegas win totals are very close to their initial guesses, while others are much farther away. Some are more enticing than others. 

When Jason Zweig added comments to The Intelligent Investor, both Zweig and Graham were quick to point out that a good company may not be a good investment. The best investments were in companies that returned the most value to their shareholders. It’s an important nuance.

House and Simmons look for this in the NBA. One example is the Atlanta Hawks, who Simmons thinks will be not-quite-as-good as last year. The Hawks might have overachieved, and them being a bit worse (mean reversion, see this Nick Murray post for more ) makes sense. That’s an easier case than figuring out if the Knicks will be better with many new players.

2 – Break intractable problems into tractable sub-problems. It’s hard to figure out if the Cleveland Cavaliers will win more than 57 of the 82 games they play. There are a lot of variables that in that problem. To make it easier, Simmons and House create a sub-problem that’s easier to figure out — will Lebron James (the best player on the team) play a lot?

Current thinking in the NBA is to not push the best players during the regular season so they will be healthy for the playoffs. Simmons and House guess the Cavaliers will do just that. Will Lebron play all 82 games? Certainly not.

Last year he played 69 games and the team won 50 of them. Now he’s a year older, has a greater chance of injury, and maybe he’s not as good as last year. But, this doesn’t answer our question. We need to avoid the bait and switch I almost fell for.

Once you break hard problems into solvable pieces, you need to put everything back together again. A smart bettor will sum the solvable sub-problems to get a cumulative answer to the intractable one.

3 – Strike the right balance between inside and outside views. Simmons is both a “Boston homer” and, well-connected analyst. House is a fan. This episode features mostly an outside view of the league.

This can be good. There are lots of cases where a team believes they will be good, but they aren’t. Teams and fans that think “this is our year,” fall into an outside view trap. Aside from when they talk about their favorite teams (Boston and Washington), House and Simmons have an impartial view on the league.  

On the other hand there’s the inside view. Inside views are ones with the most specific information, but it’s often not enough. When the Titanic shipped off, there weren’t enough lifeboats for everyone on board. The inside view (from the shipmaker) was that the ship was unsinkable, therefore they didn’t need that many boats. The outside view would have been to ask a third class passenger what they thought about not having access to a lifeboat.

The best way to forecast sports is also an illegal way. Pete Rose is banned from baseball because he used the inside view. The key is to find balance, like when buying a house. You literally should go into the house for the inside view, but also look at the school district, previous sales, etc. for an outside perspective. 

We don’t want to be the hammer that sees everything as a nail. Instead, we should use multiple views like we would multiple tools.

4 – Strike the right balance between overreacting and underreacting to evidence. A lot of the talk on the podcast is about players on new teams. For good reason. Steve Kerr is credited as the needed upgrade for the Golden State Warriors to win the title last season. Lebron won once he teamed up with Dwyane Wade. The challenge for a forecaster is to find the Goldilocks zone of information, what’s just right?

Tetlock might suggest we pay attention and not answer the wrong question. Let’s use the Boston Celtics as our example. Simmons is a huge Boston fan, as you can gather from his positivity on the team:

 “They were 20-10 last year, they were an above .500 team once they got Isiah Thomas, their biggest issue was they had bad big guys and they brought in David Lee and Amir Johnson. They’ll be better defensively. They have a lot of depth… I think Marcus Smart is going to be a lot better this year…they’re built for a 47-48 win type of season.”

Okay, a lot of new evidence. The key to superforecasting is to separate how much of this matters and how much doesn’t. Simmons weaves numerate – “they were an above .500 team” – and literate – “they have a lot of depth” – examples into his case. How do you figure out what matters?

The best forecasters, Tetlock found in his research, are ones who start with a good forecast and  make small changes. It’s like a chef who switches oil for butter in a recipe, not a chef who makes lasagna without noodles.

A good starting point (see this Michael Mauboussin post for more on base rates)  is to ask how many games the Celtics won last year (40). The over-under for this year is 43 ½.

Now, is the evidence clear enough that the team will win four more games? Simmons thinks so.

5 – Look for clashing causal forces at work. “Synthesis is an art,” Tetlock writes, “that requires reconciling irreducibly subjective judgments.” Huh? Let’s explain it in basketball terms

When Simmons and House talk about the NBA Eastern Conference, they pause at a few spots to get their bearings.

Simmons points out that not all the teams can go over the win totals. They can all improve, but they can’t all win more games. It’s the same thing if everyone became a better driver, some people would still be below average. 

When talking about the Toronto Raptors, the guys are somewhat at odds. House is confident that they’ll win more than the projected number, Simmons less so. He invokes this bit of thinking:

“It feels weird to go under, because then you have to point to at least two other teams in this division that are going to win more than you think. We know one of them, but then you’re basically saying the Knicks are going to win 40 games and I don’t feel good about saying that.”

Simmons recognizes the Raptors can’t be worse in number of games won and other teams in their division not be better. And this is key: remember what question you are answering. This podcast is not about whether teams will be better, it’s about the number of games they win. There are only 16 divisional games in the NBA. If the other divisional teams aren’t going to be better, then the chances that Toronto will win most of the 16 divisional games is high.

6- Strive to distinguish some degrees of doubt. Tetlock found  that people who refined their guesses were often the most accurate. Superforecaters avoided, “granularity as bafflegab,” Tetlock writes.

House and Simmons do this too. When they make their NBA guesses, many were to very specific numbers. The Sixers “feel like an 18-62 team,” and Chicago “feels like 46-36 to me,” says Simmons.

This is a signal that Simmons and House think at the right level. It’s detailed enough. In his research, Tetlock noticed that forecasters who guessed to the nearest 10th of percent were less accurate than those who guessed to the nearest 5th who were less accurate than those who guessed to the nearest 1 percent.

7 – Balance under and over confidence, prudence and decisiveness. Confident arrogance and decision paralysis is a balance beam that all decision makers walk. Tetlock warns not be drawn to confidence and decisiveness just because that’s what people want. There are no absolute to-dos, but there’s one do-not-do; develop hubris. A parable will help.

There once was a hedge fund that made a lot of money – and that’s in hedge fund terms. The hedge fund was full of brilliant people. Some would later win a Nobel prize. Others were less accomplished, but only barely. 

In basketball terms it was like 8 of the best 20 players got together and formed their own team. Both teams, our real hedge fund and hypothetical basketballers, were better than everyone else. By a lot. Outsiders wanted in. Investors begged to have the fund manage more of their money. Executives, trainers, and ticket sellers begged to help the players.

But as the teams succeeded, their confidence grew. Their decisiveness bloated. Their prudence shrunk. We can do this on our own, both teams thought. So, the hedge fund managers fired their clients, “we have enough of our own money,” they told the investors, “we don’t need you anymore.” The basketball players did the same. They got rid of the trainers, medical staff, ticket sellers. When you’re so dominant, do you really need help from others? 

Yes, it turns out that you do. The hedge fund, Long Term Capital Management, hit a dry spell. In 1998 they needed 2B$ to weather a momentary liquidity snafu. No one threw them a lifeline. Only once the company was dead in the water did someone pull the body out. Ironically, the amount they needed was almost exactly the amount they told to leave the fund.

Our hypothetical basketball team suffered too. Two players collided in practice. One had an emotional breakdown. One was mad he didn’t play enough and he sulked on the bench. With barely enough players, the team wore down. There were no trainers to help them get better, no ticket sellers to dream up crazy promotions, no COO to manage the team.

It would be the norm for Simmons and House to wave their hands and shout. They would boldly declare why they were correct. They don’t.

8 – Look for errors behind your mistakes, check for hindsight bias. Our ability to look back and remember things accurately is wonderful. Wonderfully wrong. We weren’t really that wrong. It takes a lot of effort to look back and remember how you really felt at the time. 

The book has some wonderful stories from Tetlock’s history that I really enjoyed. We won’t rehash those here (buy the book if you want to read those). We need action. What can we do to avoid hindsight bias?

A decision journal is one way. Write down how and why you concluded one thing rather than another. However you do it, you need to note it. Tetlock quotes Bill Gates:

“I have been struck by how important measurement is to improving the human condition,” Bill Gates wrote. “You can achieve incredible progress if you set a clear goal and find a measure that will drive progress toward that goal…. This may seem basic, but it is amazing how often it is not done and how hard it is to get right.”

Simmons does not do this. “Last year we did okay,” Simmons says, “the year before we did amazing.” That’s not good evidence. Evidence is key because we want to gauge results against process.

Mark Cuban noted that the Dallas Mavericks do this. Adam Carolla does this for his projects. Tyler Cowen works this way.

9 – Bring out the best in others, let others bring out the best in you. The conversation between House and Simmons is a case for bringing out the best in others and letting others bring out the best in you. It works between House and Simmons because they are friends and know the rules of the game – have fun.

People can be even better forecasters if they work well with others, but this isn’t easy or immediate. Tetlock writes that after some people were placed on teams, they kind of clammed up. Some noticed the credentials of the people around them and thought, maybe I shouldn’t say anything.

In those groups, the superforecasters said there was a lot of dancing around and couching statements so not to step on someone else’s toes. The best situations had accepted leaders and norms. If you know who’s in charge and what to do, you’ll do better work.

Simmons and House are old friends, and the balance is clear. Simmons is the host, House the guest. Done.

10 – Learn by doing. There’s only so much you can learn by reading about forecasting, writes Tetlock.

“Just as you can’t learn to ride a bicycle by reading a physics textbook, you can’t become a superforecaster by reading training manuals. Learning requires doing, with good feedback that leaves no ambiguity about whether you are succeeding—“ I’m rolling along smoothly!”— or whether you are failing—“ crash!” “

Betting is a great way to learn by doing because the results are clean and immediate. Once the NBA season is over, there will be no doubt about whether some bets were right or wrong. The real world though is much more ambiguous.

11 – Be willing to break any of these commandments. One key point in Superforecasting is that forecasting is never done. “It ain’t over till it’s over,” quipped Yogi Berra. It’s good advice for forecasting too. The most successful forecasters were ones that constantly evolved.

For example, sometimes superforecasters found an inside view that was so complete it correctly guided them to the right answer.

When Simmons and House talk about the NBA, they do just this. There’s no reason to “strike the right balance between overreacting and underreacting to new information” if there’s no balance at stake. If a great player hurts his ankle, the bet changes, and not evenly.

–

Is Bill Simmons a good forecaster? It appears so. Simmons marches to the same rhythm of Tetlock’s research.

Wow, you made it to the end of the post. A solid 2500 words. That’s like 10 pages in a book (and forever online). If you found a few things valuable, you can donate $2.