PDA

View Full Version : Watson on Jeopardy



Taeolas
2011-Feb-15, 03:49 PM
Frankly I'm a little surprised there hasn't been more discussion about this in here; though there are probably many more appropriate forums out there for it I guess.

Still, he did his first showing of 3 last night, and it was interesting to see, to say the least.

I still call "Hax!" on him picking the Daily Double on his very first pick. :)

His board choosing algorithm was also very obvious, but it's also one we've seen contestants use too. (Randomly pick high value clues until you hit a daily double(s), then chew through the lowest values left on the board)

Alex as a host dealt with Watson smoothly, treating him as just another contestant, complete with the chidding for repeating a wrong answer.

I was also impressed with how well he was doing, though he clearly had trouble with some of the trickier questions. Hopefully we'll see some trickier categories coming up with more compound questions. (like the "Missing a leg" one Watson missed). Watson had the gist of the needed question, but missed the specific details.

Judging just from 1 round viewing, he's well above "SNL Celebrity Jeopardy" skill, and even above most of the Jeopardy special Tournament skill levels (Celebrity, Teen, College), but probably not quite at Tournament of Champions level.

All in all, a very impressive showing so far. I'm looking forward to tonight and tomorrow.

The Backroad Astronomer
2011-Feb-15, 04:35 PM
Watson seemed to have a problem with the decade category, and few othe clues but yes he is doing well. Last week they had a NOVA episode on Watson. You may be able to view it directly on the PBS* website or thru youtube.
*sometimes I can view it directly thru their own website but other times I have to go thru youtube or google video. I think my internet gets routed thru an american router at times so I appear I am in the the states.

Fazor
2011-Feb-15, 05:50 PM
Yeah; I didn't realize it was going to be on Jeopardy this week, but I did watch the Nova last week (I'm actually taking part in a study and have to watch Nova. Since I always watch them anyway, though, I'm not really making any sacrifices in the name of science.)

Taeolas
2011-Feb-16, 01:32 AM
Well he really dominated. I hope they slow his reaction speed a few milliseconds to give the humans a better chance.

All in all he did pretty good, but his daily double betting strategy was weird. No weirder than other contestants have done though.

But why the hell did he think Toronto was a US City? That should have dropped it out quickly. It seems like he doesn't draw enough connections between the Clue, the Category, and Alex's Meta data. (as in the extra category information Alex gives). Considering Toronto's two airports are "Pearson" (named after a Prime Minister who didn't do anything major in any world war) and "Island" which doesn't really fit any World War battle, he missed on just about all of the criteria for that final Jeopardy question.

Extravoice
2011-Feb-16, 01:51 AM
His performance this evening was very impressive. As for speed, it looked like Ken Jennings was getting frustrated because Watson rang in so quickly. As for Alex's meta data, Watson is deaf and is provided the clues via text message. He probably knows what quotes in the category title mean, but can't discern any info from Alex's tone of voice.

I was pretty surprised he picked Toronto for the final question, too. I can understand his having difficulty parsing the clue, but the category seemed pretty clear. To be fair, Watson added many question marks, indicating that he was guessing.

Somebody has to say it: "I, for one, welcome our new computer overlords." :)

Taeolas
2011-Feb-16, 02:06 AM
I assume that to make things 'fair' Alex's category script is also passed in so Watson would have as much info on the category as the other contestants. Still doesn't handle the nuance problem but it clears up the classic clarification comments like "All answers must begin with the letter 'P' or 'Q'" and similar details (not a good example but an example).

I would have loved to have heard what they talked about during the end credits. I'm sure the programmers have heard it too with open ears.

Chuck
2011-Feb-16, 02:56 AM
What game is next? Diplomacy (http://en.wikipedia.org/wiki/Diplomacy_(game)) would be nice.

Extravoice
2011-Feb-16, 01:27 PM
What game is next? Diplomacy (http://en.wikipedia.org/wiki/Diplomacy_(game)) would be nice.

Maybe, or maybe not (http://www.imdb.com/title/tt0064177/).

"Colossus' speech makes clear that it will establish its own "absolute authority" over Earth, completely controlling mankind and eliminating freedom, achieving the same goals the President intended, but by a totally different means". :eek:

Fazor
2011-Feb-16, 02:14 PM
Bah. I missed it again last night as I didn't get settled in and the tv fired up until almost 8, but it runs at 7 here. I have scheduled tonight's to record, so I can catch it regardless of what time I get to finally settle onto the couch.

. . . though, until two weeks ago, if you ever told me I'd actually schedule an episode of Jeopardy! to record on my DVR I would have called you crazy. Not that I hate the show or anything, but it's not something I ever watch.

Ronald Brak
2011-Feb-16, 02:38 PM
So Watson gets her messages by text? I read some stuff that made it sound like Watson was listening to the host, which would be very impressive and difficult to do accurately, but not astounding given the amount of recordings of Alex's voice they'd have available to prepare Watson with.

Buttercup
2011-Feb-16, 02:48 PM
Puh-leeze. :rolleyes: Keep it HUMAN.

It's likely a total "given" a computer will win. Too many odds in its favor ruins the suspense of otherwise total human involvement.

Chuck
2011-Feb-16, 02:52 PM
Watson should also have a camera with which to watch the Jeopardy board so it would get the clues by sight and sounds just as the human players do. If it gets its clues as text messages directly into its brain while humans have to use eyes and ears then Watson can start trying to find an answer sooner.

Chuck
2011-Feb-16, 02:55 PM
Maybe, or maybe not (http://www.imdb.com/title/tt0064177/).

"Colossus' speech makes clear that it will establish its own "absolute authority" over Earth, completely controlling mankind and eliminating freedom, achieving the same goals the President intended, but by a totally different means". :eek:
This might have already happened but the computers see no need to upset us by telling us about it.

Trebuchet
2011-Feb-16, 03:37 PM
I watched about 10 minutes last night, but gave it up. I watch Jeopardy regularly and thoroughly enjoy it but this was boring. The human competitors could most likely have answered most of the questions (as could I) but they had no chance of ringing in. I like Chuck's suggestion that the computer should have to read the answer on the board. I also noticed a complete lack of clues based on audio, video, or images. The computer should have to parse those as well, just as human contestants regularly do.

DonM435
2011-Feb-16, 04:25 PM
I watched about 10 minutes last night, but gave it up. I watch Jeopardy regularly and thoroughly enjoy it but this was boring. The human competitors could most likely have answered most of the questions (as could I) but they had no chance of ringing in. I like Chuck's suggestion that the computer should have to read the answer on the board. I also noticed a complete lack of clues based on audio, video, or images. The computer should have to parse those as well, just as human contestants regularly do.

Of course. There should be no spercial accomodations. If a situation arises that only a human can handle, the computer has to accept that, move on and compensate elsewhere where it has obvious advantages. It has to play all of the game, I'd say.

Chuck
2011-Feb-16, 04:26 PM
This demonstration might lead a lot of people to believe that artificial intelligence is more advanced than it actually is. Watson's ability to make sense of English is impressive but isn't it still just looking things up in a database to come up with its responses?

SeanF
2011-Feb-16, 04:36 PM
Of course. There should be no spercial accomodations. If a situation arises that only a human can handle, the computer has to accept that, move on and compensate elsewhere where it has obvious advantages. It has to play all of the game, I'd say.
I'm going to disagree with this. I'll concede that what they're playing would properly be described as a subset of "Jeopardy!" and not actually "Jeopardy!" itself, but still - the computer and the people playing against the computer are playing the same game, it's just a slightly different game than what is normally played by only people. It's still a level playing field.

Chess has rules regarding the physical touching of the chess pieces which chess-playing computers do not have to deal with...

Extravoice
2011-Feb-16, 05:36 PM
Watson's ability to make sense of English is impressive but isn't it still just looking things up in a database to come up with its responses?

How is this different from what the human contestants are doing?

I got last night's Final Jeopardy correct because I remembered that Midway Airport serves Chicago, Midway was a major WW II battle, Chicago has another airport named after a person (I had no idea who O'Hare was), and Chicago is a US City. In my mind, this was ranked higher than any other cities I was considering. If my understanding (from the NOVA episode) is correct, this is pretty much the same thing Watson does.

Chuck
2011-Feb-16, 06:34 PM
There's a big difference between looking something up, such as what problem some community is having, and being able to discuss what can be done about it.

Fazor
2011-Feb-16, 07:16 PM
Right. That's what's so impressive. It's not like it has a catalog of all the questions and just has to reference them. It has to determine what's being asked, reference all the information it has on the subjects, and then decide what solution best fits what Watson thinks the question is.

. . . I didn't see last night, but from what's been said here, it's managed to do all that faster than the human contestants can manage to buzz in. That's pretty impressive.

Now I would argue that this, in and of itself, shouldn't really be considered a true AI. But it would be an important part of a true AI system.

SeanF
2011-Feb-16, 07:19 PM
. . . I didn't see last night, but from what's been said here, it's managed to do all that faster than the human contestants can manage to buzz in.
Not necessarily. It may be programmed to buzz in before it has deduced the answer, and then continue processing until it has to actually produce an answer.

I guess that's what some human contestants do, too. :)

Buttercup
2011-Feb-16, 07:36 PM
If I were one of the contestants to go up against Watson, I'd be miffed. It'd be like going up against Mr. Spock. C'mon -- Watson's going to win. There go my chances at making $20,000 in 22 minutes for the only time in my life!

Click Ticker
2011-Feb-16, 08:07 PM
I anyone saw the Nova episode - it did take a lot of work to get Watson to where it is now. When I saw Nova, I did not come away with the impression that the outcome was a forgone conclusion.

About the only thing that is really unfair is the buzzing in previously mentioned. Humans simply can't buzz in as fast as a computer. If they can't modify the reaction time to be on par with the average reaction time of the human Jeopardy champions - the computer will aways have the advantage.

To be truly fair, just take turns answering trivia questions delivered in a round robin format to the three contestants and see who has the most right at the end. Eliminate the buzzer and see who is better at just trivia and understanding the clues.

Fazor
2011-Feb-16, 08:12 PM
Not necessarily. It may be programmed to buzz in before it has deduced the answer, and then continue processing until it has to actually produce an answer.

I guess that's what some human contestants do, too. :)

I'm not a big follower of Jeopardy, but don't you have to answer pretty much right away? I mean, there's some room for 'buzz and keep thinking', but I thought it was a fairly small window. Either way, if it's not programmed to do so, I'd think it only fair to have Watson only buzz in once it's calculated an answer.

Jim
2011-Feb-16, 08:15 PM
Not necessarily. It may be programmed to buzz in before it has deduced the answer, and then continue processing until it has to actually produce an answer.

I guess that's what some human contestants do, too. :)

If I recall the explanation correctly, Watson only buzzes in when it has an answer that exceeds a confidence threshold. They show it's top three answers on a chart at the bottom of the screen, along with its confidence in those answers. Several times I noticed Watson had the correct answer, but wasn't sure enough of it to buzz in.

Human contestants, otoh, will buzz in and then try to come up with the right answer.

SeanF
2011-Feb-16, 08:22 PM
IIRC, Fazor, there's a certain duration after buzzing in in which the contestant has to give the answer. So you don't have to begin speaking the answer right away, but there's only a few seconds.

Which brings up another question - is Watson giving its answers audibly or just on a display screen? It should have to do the former in order for the time issue to be "fair."

Fazor
2011-Feb-16, 08:31 PM
On the Nova, at least, it was giving them audibly. It just made it cooler, because it was in that horrible synthesized speech voice. :)

Van Rijn
2011-Feb-16, 10:57 PM
Minor limitations aside, Watson was doing astonishingly well, given what I would expect from a language parser and expert system. There are occasions where I feel like I've stepped into a science fiction story. This is one of those times.

Van Rijn
2011-Feb-17, 12:47 AM
If I recall the explanation correctly, Watson only buzzes in when it has an answer that exceeds a confidence threshold. They show it's top three answers on a chart at the bottom of the screen, along with its confidence in those answers. Several times I noticed Watson had the correct answer, but wasn't sure enough of it to buzz in.

Human contestants, otoh, will buzz in and then try to come up with the right answer.

Yes, there was a New York Times article someone linked to in an earlier thread on Watson that discussed that. It's actually one of the human advantages. The researchers were amazed at the reaction time the good players had, how quickly they would buzz in. Human players often buzz in before they've worked out the answer, if they think they've got a good shot at it. The top players can usually pull that off (they rarely give a wrong answer after buzzing in). Watson only buzzes in after it fully evaluates the clue and calculates a high confidence in the answer - no shortcuts there. Apparently, it was very hard to get Watson to the point where it could do all that faster than the good human contestants could buzz in.

Taeolas
2011-Feb-17, 01:10 AM
They may have slowed him down a hair tonight, or the questions were more challenging for it, but he definitly had more of a challenge tonight. Ken especially seemed to have gotten back into his grove for playing and was a serious contender for first place. Loved his final answer. :)

Watson uses a generated computer voice that was surprisingly well done, though he had some problems with some words. He handled the made up words like "Mouse-tery" nicely too.

As a developer who knows enough programming to know how hard what Watson did was, I was VERY impressed by what was pulled off. Not quite Turing-test passable, but very close. And real time language parsing like Jeopardy is one of the keys (I think) of AI, since that is basically language.

Extravoice
2011-Feb-17, 01:22 AM
Ken especially seemed to have gotten back into his grove for playing and was a serious contender for first place. Loved his final answer. :).


:whistle:

Van Rijn
2011-Feb-17, 01:44 AM
This demonstration might lead a lot of people to believe that artificial intelligence is more advanced than it actually is. Watson's ability to make sense of English is impressive but isn't it still just looking things up in a database to come up with its responses?

It's more than that. In the case of Watson, information from a database is used to build the answers, but it isn't just parroting answers, and finding the correct answer on a regular basis is extremely difficult. Properly parsing and responding to natural langauge is a huge issue in AI. I'm used to seeing expert systems where the knowledge is on a very focused subject, so parsing is relatively easy. In the typical expert system, things outside the limited scope of the subject simply don't need to be considered. For instance, an expert system dealing with some aspect of avian biology wouldn't need to consider questions about fictional animals, like "Big Bird," but Watson does. Also, the rules for building an answer in an expert system are usually far more constrained as well. I'm sure some will overestimate Watson, but still, this is extremely impressive.

DonM435
2011-Feb-17, 01:25 PM
It was a very impressive showing, despite the rough spots noted here.

When it came up with "Toronto" as a U.S. City, I wondered if, somewhere early in its search tree, it translated "U.S." to "American," and then tripped itself by reverting to the broader definition. But it did manage to link up those defunct Transylvanian provinces with Bram Stoker.

In the future, they might consider letting three computer programs compete against one another, to test different design philosophies.

Chuck
2011-Feb-17, 02:25 PM
There's also a Toronto in California. Maybe that's the one it meant.