Not just OK OKRs

Sarah Tavel told Share Parrish:

“At Pinterest our growth team decided their OKR was monthly active users, a lowest common denominator thing. But if you choose the wrong metric you end up optimizing for the wrong thing, you’ll build the wrong features. Startups are incredibly resource constrained and you waste a lot when you focus on the wrong things. When the team realized this and changed the OKR to Weekly-Active-Pinners the entire roadmapped changed and we were able to serve the users much more successfully.” – @SarahTavel The Knowledge Project.

Tavel’s quote could be about 2000s baseball as well. The early days of baseball Moneyball were an era of what Tavel calls vanity metrics. At one point in the Michael Lewis bestseller, protagonist Billy Beane yells: We aren’t selling jeans! His point was that classic metrics like hits, home runs, or even body-type weren’t the optimizations he was looking for.

The problem that Tavel’s and Beane’s teams faced was a data collection problem. These metrics were mostly right and easy to collect.

“I have an allergy for vanity metrics. I can see a vanity metric a mile away. It comes down to intellectual rigor and being honest with yourself: what are you measuring and is it the right long term thing?” Sarah Tavel

Really wrong metrics push behavior in absurd directions. For instance, records used to earn certifications (Platinum, Gold, etc.) based on shipments not sales. Sgt. Pepper’s Lonely Hearts Club Band soundtrack (1978) was a Platinum album but was a sales bust. That’s what happens with an OKR based on shipments, not sales.

To their credit, the RIAA changed the rules for certifications in 1979. That’s what Beane did too. Tavel too. It’s a good reminder to ask: am I using this information because it is helpful or easy?


Moneyball might be the best way to win in sports but sports is a story and stories need narrative. I loved the Tim Duncan Spurs but the media didn’t. It’s why there’s only one honest sport.

Weekly active pinners? Hold my beer.

Ohio’s Vaccine Lotto

On May 12, 2021 Governor Mike DeWine of Ohio announced a one million dollar vaccination lottery. Teens were eligible for a college scholarship. Two days after the announcement Ohio doubled its vaccinations-per-day figure to thirty-three thousand people. Success!

Maybe. “States with lottery programs,” noted the Boston Globe “are not doing any better compared to states without such initiatives.”

And.

But, there are at least two reasons Ohio’s strategy was a good one. The first is the testing of new approaches. One of the beautiful things about the United States of America is the differences in states. When states do different things academics call this “heterogeneity” and “natural experiments”. While not perfect, these opportunities and observations lead to novel lessons. Part-of-the-reason there won’t be another 2020 are these learnings.

The second reason Ohio’s vaccine lottery was a good idea is an idea from Maxims for Analytical Thinking, a Michael Mauboussin recommendation:

MfAT is a book of thinking tools by Dan Levy who focuses on the ideas, information, and influence of Richard Zeckhauser. Maxim 1 is When you are having trouble getting your thinking straight, go to an extreme case. Using this lens, was the Ohio Lotto a good idea?

Imagine it this way. What if there were a Hypo-Ohio, where thanks to the industriousness, intelligence, and ingenuity of the individuals, a vaccine holiday was declared on February first. Employers gave employees the day off. Starbucks and Subway donated their stores for stick sites. Netflix was free for Ohio ISPs. Everyone that was willing and able to get a vaccine got vaccinated.

Ohio Vax

If that happened, like poker chips slid across a table, the May blip and March wave would be compressed into an early February explosion. This would have been awesome. We know from the vaccine friendship paradox that all social networks have a super-spreader. At the extreme, pulling the demand forward would be a good thing.

But what was the effect size? Here I’m over my skis. But that’s actually okay. The techniques I learned in my Ohio high school still work: remove the bad answers first. Like the 15y or 30y mortgage question, I’m looking for choosing from only the good options. At the extreme, pulling demand forward is a fantastic idea. How much effect, I don’t know, but I’m glad they tried.


Bias Warning: I thought the Ohio Lotto was a good idea from the start.