Final Jeopardy

Supported by Greenhaven Road Capital, finding value off the beaten path.

Final Jeopardy by Stephen Baker is the story of IBM Watson, the Jeopardy champion who defeated Brad Rutter and Ken Jennings.

In December we looked at AI with posts from Andrew Ng, Paul Daugherty, Hal Varian, and Avi Goldfarb and Ajay Agrawal. The book Final Jeopardy is a polaroid for AI. It’s a precursor to those blog posts. A sort of this is how things were.

In the late 1970’s IBM crested the technology wave and then big blue began a descent as Microsoft rode the new wave of PCs. IBM’s mainframes were a vertically integrated business model and the era of PCs, horizontal integration ran roughshod over a prostrate Big Blue.

By 1992 the company was saddled with a 5B$ loss and their shift to software and consulting began. In 1997 IBM’s Deep Blue defeated Kasparov in chess and in Harry Friedman was promoted to Jeopardy producer and began to include more popular culture and current events in the show.

He thought he was going to figure out the computer. That BBC video demonstrates something Rory Sutherland rails on; that creativity means solving problems in new ways. Kasparov resigned after 19 moves becuase the computer solved the problem in new ways.

Aside from Trebek shaving his mustache (2001) and SNL Celebrity Jeopardy, Jeopardy, America’s favorite quiz show stood stolidly with Wheel of Fortune as American staples.

In 2003, the five-show limit was lifted and Ken Jennings took the game by storm, drawing 15 million viewers – according to some – igniting a question at IBM. Could we beat him?

Watson was part curiosity, part marketing, part interestingness. Baker wrote, “IBM’s biggest division, after all, was Global Services, which included one of the world’s largest consultancies. It sold technical and strategic advice to corporations all over the world. Could the consultants bundle this technology into their offerings?”

Internally there were some high hopes. They just had to make it and that proved damn difficult.

“The biggest obstacle (early on in AI) was language.” In addition to that, there was no consensus on how to approach trivia questions. Some researchers want general AI, others craved specific. Reading a book written in 2011 about events in 2009 in 2018 gave one perspective but it helps to remember that whatever is being done at the time doesn’t seem easy to the people doing it. We all Google all day, but Baker does a good job articulating all the issues the team faced.

In 2005 IBM starts considering the challenge in earnest and one engineer is tasked with an off-the-shelf approximation. We see this advocated in companies from IBM to IDEO, there’s huge value in prototyping. Before jumping in, assemble a simple solution. IBM discovered that their first internal version was much better than off-the-shelf software. They moved on.

The IBM team used stuff they had worked on but also invented new ways to parse data. They also created new ways to work together. One key to Watson was parallel inquiries. Once the system figured out what a question asked; the sausage celebrated every year since 1953 in Sheboygan Wisconsin, it would try to come up with a collection of answers.

Ideally, some of the answers would be the same and quantity lent credence to quality. It was a sort of wisdom of crowds, what does the collective algorithm suggest? And the team needed to work this way too. “Ferrucci decided to take the same approach with his team. He would cluster them. He found an empty lab at Hawthorne and invited his people to work there.” They had to be together.

This might be true beyond teams like the Jeopardy engineers. What if schools work so well not because of economies of scale but network effects? I give you a good idea and you give her a good idea and so on. What if we should prioritize the gestalt rather than the CBA.

Vicinity also kept values aligned. Small groups don’t need mission statements so much as communication. The Watson team didn’t necessarily want Watson to Win. They wanted Watson to demonstrate IBM’s prowess. That included not looking bad, the team worried that “a humanoid Watson might frighten people.” Watson needed to represent IBM well, and not embarrass them. It could be funny, but not profane. Early answers examples like “This is the type of diet grasshoppers eat – Kosher” brought laughs. But in the category of Just Say No, ‘This is a four-letter German word.’ Watson’s gave the answer ‘Fuck’.

Oops. IBM staffers chuckled and began work on a profanity filter.

Straight facts were the easiest questions. “There were factoids, each one wrapped in the most helpful data for Watson: hard facts unencumbered by humor, slang, or the cultural references that could tie a cognitive engine into knots.”

But Jeopardy is full of puns and ambiguity. “At low levels of confidence, I think we’ll just have it say it doesn’t know, sometimes that sounds smarter,” said Chu-Carroll. Watson needed to look like a helpful took, not an unhelpful HAL.

…

What’s clear from the book is that Watson took a lot of work to work well in a limited domain. What’s also clear since then is that technology has gotten quite good at working well in many different domains. From grammar to adaptive cruise control, it seems like technology mostly works and mostly makes life a lot better. One step along that way was Watson.

 

Thanks for reading.