Thoughts on Jeopardy! and Watson

We here at TPS are big fans of Jeopardy!, and I'm sure you've heard of the Watson vs. Ken Jennings and Brad Rutter event that took place last night and will continue today and tomorrow. A few thoughts (and just for ease of writing, I recognize that I've switched "answers" and "questions" in the framework of a Jeopardy! game):

- It's clear that Watson will do better with fact-based questions as opposed to interpretation-based questions; I'd attribute a large amount of its success last night to questions of the former type. I would guess-- unless it's been directly programmed to deal with it-- Watson would be helpless on before-and-after-type questions.

- There was a video going around a few weeks ago of a test round between the three contestants, with much the same result as last night-- Watson does well early, the remaining contestants perform better as the round progresses. One striking aspect of the practice round, however, was that when Watson felt it was confident enough to answer the question, it always rang in first. Every. Time. That's the big advantage of the machine, because Jeopardy! isn't just about knowing the answers-- it's being able to time the buzzer correctly so you actually get a chance to answer. A machine, naturally, will have a faster reaction time than a human. You've effectively eliminated the competitive portion of the game. So while it is impressive that you can design a machine to parse through mountains of data to come up with an answer in short period of time (more on that in a second), but don't look past the fact that you've basically given one player a massive advantage over the other two.

But last night, I believe only once, Ken Jennings beat Watson with the buzzer. (When Watson has a "green" answer, he's trying to buzz in.) In my eyes, that was the most surprising moment of the night. It's a risky move to try and anticipate the proper time to ring in (constestants can ring in once a light goes on; if they ring in too early, they are locked out for a few seconds), but up against a machine that's going to react faster than you, contestants don't have much of a choice.

- It's natural to think that you tell a computer to do something, and it does it in the blink of an eye. But what the computer has to do here is titanic-- seconds matter, milliseconds matter. To that end, the more time Watson has to think, the better of a chance it has. I noticed last night that Watson did distinctly worse on short questions-- I remember one particularly short question where it struggled to formulate anything of a guess. Most of the questions last night were of the longer, factual variety-- right in Watson's wheelhouse. As long as both of those margins remain, expect Watson to perform pretty well.