“Let’s say you want to learn about trade in the ancient world. If you’re using archeology I have good news. We know a lot about trade in olive oil and wine in the Roman world. Olive oil and wine both moved in clay (Amphora) vesicles. They were disposable. When a giant pot of wine reached its endpoint, you poured it into bottles and chucked the big transport pot. Pottery survives, you can shatter it but the pieces are still there.” – Bret Devereaux, EconTalk, August 2021
Grain meanwhile was transported in sacks. “Imagine what parts of your life,” Devereaux asks, “are archeologically visible.”
This example highlights a bigger question: how do I know something is true? Big right? We can’t know everything, so we use samples. And Amphora highlights two caveats.
First, is that easier-to-find data may overstate the case. Covid19 in 2020 had a lot of early data that was easy to collect but not necessarily a good sample. During the first six months we talked a lot about the number of cases, maybe we still do depending on when you read this. Cases are okay, but the best predictors in the early days looked at hospitalizations and deaths. Not to stop there, Covid19 also affected ages quite differently, a fifty year old was fifty times more likely to die than a fifteen year old.
Second, easy data is usually ‘expensive’ because many people use it. If they use it in the same way then, like an auction for a Beanie Baby, prices rise. If the information turns out to be wrong, then prices fall. The heart of moneyball was to find data that was also good, but ‘cheap’ because fewer people used it.
Archeological visibility is a neat analogy to use for thinking about sampling. Though an ancient effect, we can use it still today. The next time we see data, we can ask, is this data more like a clay pot or a cotton sack?
The fifty/fifty/fifteen ratio was an estimate based on CDC data. Also, “Today’s persuaders don’t want you to stop and think,” writes Tim Harford in The Data Detective, “They want you to hurry up and feel. Don’t be rushed.”