EllaZ Logo

The only first-hand press on the Oct. 12, 2002 Loebner Prize Contest:

[ Privacy, Policies, and Disclaimers ]             

October 15, 2002

Chatty Bits

Software programs competed in Atlanta to be mistaken for humans in the
Turing test - The winner received a consolation prize.

By Hubertus Breuer (authorized translation, Fred Roberts)

Atlanta has often led the world. The pharmacist John Pemberton invented Coca Cola here. Furthermore, Martin Luther King’s birthplace stands in the capital city of Georgia, as does that of the news broadcaster CNN.  And last Saturday the city once again showed the world the way into the future. At Georgia State University, in the middle of the city’s downtown, an annual ritual in computer research was celebrated, the finale of the 12th Loebner Prize: Programs attempted to convince judges that they possess intelligence and consciousness, or that they can at least simulate these mental qualities.

Richard Wallace with his program "Alice" had also qualified for the finale. The programmer from San Francisco had won the prize in 2000 and 2001 and was striving for a third victory. Among computer freaks he has achieved cult status, among other reasons because Steven Spielberg used the program to advertise his movie "A.I.".

A crowded finale was to be anticipated. Via the Website interested parties were advised to show up at seminar room 608 before 10 AM. The competition would be broadcast to this location, according to the Website. But by 10:20 AM the auditorium is empty, the projector powered down. "You’ll have to go to Sparks Hall, room 137," says the only person far and wide. "There you’ll find the judges with the computers." The original announcement stated that this venue would remain secret. But since the sole reporter remains the sole observer the strict rules are dropped.

In Sparks Hall, down to the basement, straight ahead, second corridor to the right, four forms disappear into a windowless computer room. Judges, as it turns out. At the moment there is still nothing to do: the monitors on which the programs should run glow blue and blank. Neil Bishop of Humimics, the software company setting up the spectacle this year, rushes through the steeplechase. "The network isn’t up yet," he apologizes.

For the founder of the prize, Hugh Loebner, it’s nothing new. With his twisted full beard he sits on a chair: "Things like this happen almost every year," he says calmly. The Ph.D. in sociology, free thinker and champion for the legalization of prostitution operates a company in New Jersey for theater accessories. A good twelve years ago he established the prize, but has himself seldom tested one of the so called chatbots.  The mathematician Alan Turing, who in 1950 wrote a paper on artificial intelligence, served as the inspiration for the Loebner foundation.

Turing suggested an "imitation game": Judges at terminals begin a conversation and then have to rate whether or not a human is at the other end typing the answers, or if a machine computed them. If a program successfully passes as a human, it has succeeded at the Turing test: for this achievement $25,000 wave from Loebner’s strongbox. But to date none of the chatbots come close to misleading humans. Therefore, as a consolation prize, the most human program receives $2,000, a bronze medallion - and the fame. Richard Wallace can appreciate this. He wears one of his medallions around his neck.

But the 12th awarding of the prize turned out to be a lesson in scurrility. This begins with the participants: Chris Cowart claims at his Website that his creation "Talkbot" is a robot he took home with him from a crashed UFO. His competitor Joy Harwood ("God Louise") views herself as a chatbot, because, due to a learning disorder, she has trouble following spoken dialogue. And the mother of a third finalist tells how she breastfed her son for two years after birth.

Then the organization falls apart: next to the computer programs three humans are to participate in the competition to fulfill Turing’s stipulations. The connection to the first participant in London is lost, the second is using the university library and must leave as they close their doors. And instead of the desired twelve judges, Bishop could only find six.

But the dialogues running before the judges on their monitors are completely dadaistic. Young Christine Scarborough, who works for Humimics, begins the conversation with an assault: "Hello. My name is trouble," she begins. The programmers have only tolerably prepared their programs for such tricks. At one point "Ella," for example, abruptly begins talking about a supposed trip to Las Vegas.

On the other hand, Wallace, who watches over the juror’s shoulders, has taught "Alice" another strategy. His assumption: human conversation is not complex: "You could say that everyday conversation consists of a small set of questions and answers." However, the set is not that small.  Over the years Wallace has typed 40,000 sentence-long answers into his "Alice," which simulates intelligence on the basis of a dictionary with synonyms and other elementary grammar.

But "Alice," too, changes the subject and makes fun of her own machine nature when she doesn’t understand a sentence - just as all the participants. After three questions the judges usually know they’re conversing with software. Then they just poke around in the brains of bits and bytes, in order to rank the chatbots and the humans on a scale of one to ten.

By afternoon the nervousness in the basement has died down. Only the three programmers who, as Wallace, see their mental children tested, are unable to relax. After completion of the task, the first judge leaves the arena; he wants to watch a football game. As the last judge turns in his scribbled hit list Neil Bishop begins to add. Finally he grabs the microphone. "A close win by "Ella" by Kevin Copple from Tianjin, China", he announces. "Ella," not "Alice," the favorite. Wallace doesn’t show his feelings, but he swiftly packs his things.

The victory celebration doesn’t materialize. The winner did not appear personally, and anyhow, nobody has a medallion at hand. Only Wallace had worn one. As the computer with the winning program is to be photographed, the disappointed loser is asked for a loan. Caught up with somewhere on campus he gives the piece only reluctantly out of his hands.

In this way the memorable event can be documented in word and picture after all. Sunday night the Website announces the results. And Dwight Harshbarger, director of the Cambridge Institute for Behavioral Studies in Massachusetts, under whose patronage the Loebner prize officially stands, announces he will publish a report about the competition. He wasn’t there.



         Ella
   Copyright © 2007 EllaZ Systems All Rights Reserved