Not just OK OKRs

Sarah Tavel told Share Parrish:

“At Pinterest our growth team decided their OKR was monthly active users, a lowest common denominator thing. But if you choose the wrong metric you end up optimizing for the wrong thing, you’ll build the wrong features. Startups are incredibly resource constrained and you waste a lot when you focus on the wrong things. When the team realized this and changed the OKR to Weekly-Active-Pinners the entire roadmapped changed and we were able to serve the users much more successfully.” – @SarahTavel The Knowledge Project.

Tavel’s quote could be about 2000s baseball as well. The early days of baseball Moneyball were an era of what Tavel calls vanity metrics. At one point in the Michael Lewis bestseller, protagonist Billy Beane yells: We aren’t selling jeans! His point was that classic metrics like hits, home runs, or even body-type weren’t the optimizations he was looking for.

The problem that Tavel’s and Beane’s teams faced was a data collection problem. These metrics were mostly right and easy to collect.

“I have an allergy for vanity metrics. I can see a vanity metric a mile away. It comes down to intellectual rigor and being honest with yourself: what are you measuring and is it the right long term thing?” Sarah Tavel

Really wrong metrics push behavior in absurd directions. For instance, records used to earn certifications (Platinum, Gold, etc.) based on shipments not sales. Sgt. Pepper’s Lonely Hearts Club Band soundtrack (1978) was a Platinum album but was a sales bust. That’s what happens with an OKR based on shipments, not sales.

To their credit, the RIAA changed the rules for certifications in 1979. That’s what Beane did too. Tavel too. It’s a good reminder to ask: am I using this information because it is helpful or easy?


Moneyball might be the best way to win in sports but sports is a story and stories need narrative. I loved the Tim Duncan Spurs but the media didn’t. It’s why there’s only one honest sport.

Weekly active pinners? Hold my beer.

Incentivizing outcomes or optics?

These circles overlap to some degree in every situation. A minimal overlap situation might be the coder who wears sliders and hoodies and the banker who wears suits and slacks.

Outcome (noun): something which matters, is measured and measurable. Examples: sabermetrics, code, sales.

Optics (noun): something which might matter, however is unmeasured and unmeasurable. Examples: appearance, attention, cultural expectations..

Things which are difficult to measure will be evaluated by optics. In these conditions it’s best to maximize optics.

Things which are easy to measure will be optimized on their measured quantities. In these conditions it’s best to optimize.

Mister…Horse Racing

From, An Economist Walks into a Brothel.

“The incentives are different depending on if you plan to sell or race a horse. If breeders raced their own horses instead of selling them, they might pair two different mares and stallions to breed. Selling well takes having the right pedigree and characteristics of a sprinter, both of which encourage in-breeding, but maximizing the odds of producing a good horse is more complicated. Ideally, breeders would match the male and female characteristics, balancing out weaknesses.”

Allison Schrager

What’s remarkable about Schrager’s description is the homogeneity of breeding. If outperforming in investing is about being different and being right there’s not a lot of being different at the track.

And this kind of makes sense. There’s better ways to “WIN BIG” than racing horses, where the returns probably aren’t financial but social.

To sell well, a horse shouldn’t just be (lineage) fast but it should look fast too. Like Munger’s fisherman, sales is a question of incentives, metrics, and framing.

Tracking Tom Update.

Tom Brady passed for a paltry 196 yards this weekend and is now only ahead of pace by 36 yards. However, the combined record of the remaining teams on the scheduled is 13-24 (Lions and Falcons).

Bad teams plus a rigid playoff picture means that our framework seems to be holding: more unknowns will inhibit rather than enhance Tom Brady’s passing yards for the season.

Incentives for Marines

From Nathaniel Fick’s One Bullet Away, and a reminder that incentives function in all kinds of ways. Yes, incentives are financial but that’s just part of the whole.

In his book, No Rules Rules, Reed Hastings notes that the best performers don’t work 5% harder after a 5% raise. Instead, the best performers work harder when they work on hard problems with other hard workers.

Incentives are the ‘internal funnel’. What kind of person do we want here?

There’s no answer but there are reward options. Put on your JTBD hat, and think of the decision train. Forget the engine, what people say. Forget the caboose, what most people think. Talk and talk and talk to find the destination where people want to end up. It’ll be a mix of colleagues, money, problems, commutes (to the right person), and more. Tinkering with the mix will bring in people who best match your problem.

And if in doubt, just find a Marine.

Good luck Spencer.

What’s next to a cash register?

coffee lifestyle starbucks coffee shop
Photo by Adrianna Calvo on Pexels.com

Tyler Cowen is one of the most interesting and insightful thinkers sharing their wisdom today (and for the past decade-plus!). One of his ideas highlighted in our Twenty-Minute-Read on Cowen is to think of incentives and solving for the equilibrium.

To think like an economist, like Tyler Cowen, we should consider how things work within a market. Tim Ferriss asks Cowen what advice he would put up on a billboard? Tyler responds in an interesting, and quite different way, from what many of Tim’s other guests suggest.

Normally, this question tends to lead to something inspirational or tactical, something grand or granular. There’s also a bit of personal signaling in the answers where after an hour or so of talking to Tim, guests want to step off on the right foot.

Cowen flips the question and wonders: what works on billboards. Casinos advertise on billboards. So do lawyers and radio stations. Auto dealers advertise on the radio, which you listen to in your car, and notice how nice a new car might be. Cowen doesn’t answer Ferriss because there’s not a connection between that medium and his message, and mediums matter.

The same effect came up in the college admissions scandal book, Unacceptable. After dropping off kids, “moms in workout gear might pop into a local coffee shop, where the area near the straws and napkins was blanketed with ads for test prep services and tutoring companies.” If a college tutor, guide, or private counselor wanted to find upper-middle-class clients where better than a coffee shop?

Markets are dangerous for entrepreneurs because they lead to competition. However, markets are instructive for economists, or people who want to think like them, because they lead to understanding. During his lunch with the FT, Cowen said that he looks for Ethiopian restaurants located near other Ethiopian restaurants because “competition works.”

The biggest lesson from the short piece on Cowen is to think that through. Who is getting coffee at this kind of place at this time of day?

Unintended Consequences: bag bans

Potential bias: if bring-your-bag was equivalent to free-throw rate I’d be an all star.

Update: since drafting this post there was a related interview on Data Skeptic with Becca Taylor about this. It was good

Our evolutionary advantage was to see cause and effect. This If this then that approach to life kept us alive. It was a simple rule that worked great in a simple system. Modern life is not so simple, but that doesn’t mean we need complicated rules (see Gall’s Systems Bible).

A modern simple rule with great effect is to ask and then what when faced with an intervention. There’s always cascading effects and asking and then what is a way to look for the larger effects.

Chicago, 2016-2017, offered a chance to see this question in terms of plastic bags at the grocery store. In sequential months, there was a ban on thin plastic bags, no ban or tax, and then a tax on disposable bags.

This legislative two-step occurred because the first bag ban was a debacle. Lawmakers gave the wrong answer to the and then what question. Instead of ‘people will use reusable bags’, it was ‘stores will get around this by using a slightly different bag.’ 

Asking and then what helps us find that when schools ban soda sales households buy more, when communities ban payday lenders pawn shop foot traffic booms, when governments limit cars one day a week the total number of cars rise. 

From, Skipping the Bag: The Intended and Unintended Consequences of Disposable Bag Regulations:

We 􏰂find that plastic bag bans lead retailers to circumvent the regulation by providing free thicker plastic bags which are not covered by the ban. A regulation change that replaced the ban with a tax on all disposable bags generated large decreases in disposable bag use. Our results suggest that plastic bag bans—stricter, but more narrowly defi􏰂ned regulations—􏰁are less effective than market-based incentives on a more comprehensive set of products

There are a cornucopia of incentives to use to change behavior. Sometimes money works well (bag tax). Sometimes social norms work (the authors note that this may be present in their study). The best thing to try might be small bets

Thaler, Massey, and Losers

Well before he won The Sveriges Riksbank Prize in Economic Sciences in Memory of Alfred Nobel, Richard Thaler wrote The Loser’s Curse. Along with co-author Cade Massey (well before he hosted the Wharton Moneyball podcast), they found that NFL teams tend to trade too many assets to choose higher in the draft.

With the idea that it’s not what you buy, it’s what you pay we’ll revisit the paper. (Here’s a related 2018 work, also a gated 2020 analysis)

Also, Thaler spoke at SSAC20 about the paper in a panel with Bill James.

Thaler and Massey noticed that in-practice, NFL draft picks declined steeply in value. Top picks were worth a lot. Low picks were worth much less. The chart looked like a roller coaster’s first hill.

What they found was what’s known in the league as ‘The Chart’. The rule-of-thumb systems born in Dallas in 1991, and spread through the league. Thaler and Massey wrote that the chart standardized trades and created the norm to ‘gain a round waiting a year.'” It was the way things had always been done and few teams at the time considered why we’ve always done it that way. 

“What our analysis shows is that while this chart is widely used, it has the ‘wrong’ prices.”

The chart suggested that the first pick of the first round in the draft delivered less value than the last pick of the first round.

Thaler and Massey found the ‘right’ prices by comparing draft slot against games started and pro bowls awarded. Lower drafted players scored higher.

Thaler and Massey wondered if there was a ‘star premium’. “Over their first five years, first-round draft picks have more seasons with zero starts than with selections to the Pro Bowl.” Busting was as likely as breaking out.

Thaler and Massey wondered how often one pick was ‘better than the next guy’ at the same position. “Across all rounds, all positions, all years, the chance that a player proves to be better than the next best alternative is only slightly better than a coin-flip.”

In the paper Thaler and Massey lay out a variety of reasons why ‘The Chart’ was so different from the results. Let’s add four more.

Possible explanation 1: Measurement error. Quarterbacks are tremendously important and the stats underrate their impact. On the Wharton Moneyball podcast Massey brings this up and hints at it on Twitter. In the paper Thaler and Massey do boost performance scores by 50% without seeing value returns shift.

However maybe there was a trend they didn’t or couldn’t quite measure. More and more quarterbacks throw for more than 4,000 yards.

Possible explanation 2: Ownership incentives. The NFL—or any sport—isn’t just about winning. Though Massey and Thaler write that people don’t tune in to see their team lose, they don’t address whether people view interesting as different from winning. We think there’s only one honest sport.

If owners see values rise, share revenues, watch mediocre play, and they themselves face little (social) cost, how strong is the incentive to ‘just win baby’?

Possible explanation 3: Luck matters a lot . Michael Mauboussin writes about guidelines for situations that are more dependent on skill and ones more dependent on luck. For situations with more randomness (and luck) people should trust the base rates more and give more weight to environmental rather than personal factors. By thinking they can find a diamond in the rough, teams are operating like drafting players is more skill than luck based.

However, there’s always a chance to develop better talent evaluation, incorporate new technologies, or coach players better. Those are all skills that could improve the 52% ‘better than the next guy’ success rate.

Possible explanation 4: Culture is king. The effect that Thaler and Massey find could be partially driven by bad organizations picking at the top of the draft. Imagine these were not football franchises but restaurant franchises.

The best chefs keep their jobs and the talent pool for the open positions is a mix of unproven leaders, bad situations, or people who have already failed ‘but learned the right lessons’. The ownership of these franchises are people who have already proved they themselves are bad talent evaluators, or else they wouldn’t be looking for a new coach/chef.

What’s great is that while the data is old the ideas from Thaler and Massey are still present. They’ve taken new forms and changed in many ways but good decision making is still something worth thinking about.

 

Thank you for reading and supporting.

The incentives of salted roads

On NPR’s, The Indicator, listener Tara Harvey asks host Cardiff Garcia why we salt the roads so much. “Research shows oversalting increases the possibility of slipping, costs more for materials and labor, and causes property damage.”

The Indicator is an economics podcast and Garcia offers economic reasons. Salt is cheap and effective. Salted roads reduce accidents by 87%. Salted roads allow people to get to work, “and that is an economic activity that needs to be counted against the cost.”

Everything Harvey asks and everything Garcia says are logical statements. But while numbers convey authority, they aren’t always right.

An alternative theory focuses on incentives. Consumption and payment aren’t connected

Salt is purchased by governments and governments take heat for not being prepared. Their incentive is to salt more, not less. The consumer’s immediate incentive is excess too. Better safe than sorry.

Until however it comes time to pay. Garcia knows this, “It all costs money. Not just the direct use of the salt itself, which has to be found and paid for, but also the environmental and ecological damage that it leaves behind.”

However, consumers never connect consumption with cost. Sure taxes are too high, but does anyone know how much of their taxes go to salting the roads? Ditto for the environmental effects that follow overselling. Consumers pay, but later and indirectly.

This salty situation is similar to the psychology behind too much debt. Act now, pay later. If later is long enough away and opaque (how much does salt cost a taxpayer?) then the cost seems especially low.

Photo by Simon Matzinger from Pexels