Mismeasurements

Prices are set by the amount supplied and the amount demanded. When supply is mostly fixed, like top home-run-hitters, prices rise. This is the market mechanism.

One way to shimmy around this feature is to find things nearly as valuable, but less demand. This is Moneyball. It’s also investors who “fish in smaller ponds”. It’s also art. Collectors pine for Picasso but many fewer for real estate. Discretionary income + housing budget is a lot of money. Find a different attribute to compete on can be good advice.

Sometimes. We can overcorrect. Kristen Berman noted that one experiment which shifted the incentives from monthly to daily saw sales reps “focus on selling large numbers of cheaper items rather than more expensive items that have higher margins. A focus on short term returns can undermine pursuits of higher impact goals.” It was a case where 100 monthly sales did not equate to 4 daily sales.

This is Goodhart’s Law, when a measure becomes a target it ceases to be a good measure. A textbook example is higher education ranking hackings. Some schools counted “a postcard expressing interest” as an applicant. More applicants meant more rejections and a selectivity shine that was only a veneer.

But wait. Goodhart’s Law is a human quirk and quirks can be hacked. Airbnb grew because like eBay or Amazon, stars replaced brands. But while a four star hotel is mostly the same four star homes were not. So the company added subcategories.“We picked the subcategories based on what guests want,” said Jiaona Zhang, “but we also picked subcategories based on what we wanted our hosts to do.” 

Airbnb used Goodhart’s Law to direct their host’s attention. Once a category was counted hosts worked toward it.

Measures are a tool. They can be like Moneyball and show cheap things. They can be like Goodhart observed and show unintended consequences, but also tweaked for tidy Airbnb hosts. Measures only seem static but really reveal a lot.

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s

This site uses Akismet to reduce spam. Learn how your comment data is processed.