Serious Play: Are We Humans the Biggest Computer of All?

Would you be interested in a future where the pursuit of fun and enjoyment was one our our major roles in life because it leads to solving extraordinarily large and complex world problems? For example, what if all of us helped to digitize all the content contained in all the books in the world? Almost none of this content is currently available on the Web or in any digital form and as such, is largely inaccessible to most. And would you be interested in a world where the relationship between computers and humans is a very positive and symbiotic one? I know I would. 

I'd like to bring your attention to some of the ways in which this is already happening. Let's see if this excites you so much that you'd like to not only participate in some of these processes, but also start to use some of these models to help solve some of the problems you and your profession or areas of interest face.

In my previous posting "Moving aLOM", I mentioned some of the exciting, yet daunting, challenges of the future of metadata, such as how to create, in staggering volume, some of the more "subjective" metadata—things like the infinite characteristics that describe people, places, and things—where we humans are still the only source. This effort would include things like creating metadata for all the images and videos out there—still a largely unsolved problem—the absence of which not only makes them very hard to find, but also makes the Web and computers very inaccessible to the visually impaired, which, with age, might include a lot of us!

The Exciting Work of Luis von Ahn

Perhaps most notable in this area is a relatively young new professor at Carnegie Mellon University, Luis von Ahn, standing at right in this photo with his PhD advisor Manuel Blum. Luis has already completed some amazing work on what he refers to as "human computation" and how to put "wasted" human cycles to use in solving problems a computer cannot solve at this point in time, but humans can solve easily. Luis also picks up on a theme we have discussed here on Off Course - On Target in many other contexts—the power of the "network effect" achieved by connecting everything and every one together. Human computation is obviously focused on the latter, and Luis wants us to consider having all of our brains connected together as an extremely advanced large-scale distributive processing unit. Not to worry, no wires or direct connections to your head are required!

Before I go any further, and especially if you are more of a visual and auditory learner, let me recommend that you immediately watch this talk called "Human Computation" that Luis gave on July 26, 2006, about the power of human cycles. This 51-minute talk is part of the Google Video Text Talk series (also highly recommended), and while it is long by some current standards, I feel very comfortable recommending this to you, since I'm convinced you'll agree it was a VERY good use of your time (actually Luis' talk only runs 40 minutes, and is followed by about 10 minutes of a good Q&A session).

Another excellent reference for you, which contains more fascinating details and examples of von Ahn's work, can be found in Clive Thompson's article "For Certain Tasks, the Cortex Still Beats the CPU"  in the June 2007 edition of Wired magazine.

Games with a Purpose

But for those who don't have the time right now to look at these things more, here is a quick synthesis of what I find so exciting and interesting about the innovative use of our human "compute cycles", and the use of "fun and games" for very significant and "serious" results. What von Ahn likes to call "games with a purpose".

One of the most common and effective examples of this type of human computation is one of Luis' first applications, which is known as "Captcha". The name may be new to you, but I'm sure you're already a veteran Captcha expert! Captchas are those slightly difficult to make out words that you are asked to identify and type into a box when you are signing up for web sites. Captchas are used for responses online and in other situations where we want to prevent automated "bots" from generating unending amounts of "spam" or other undesirable exploitations of such online experiences. The problem is how to differentiate between a human response and a computer response, and Captchas are a simple solution to this problem, as well as a simple example of a problem that computers can't solve by themselves.

In itself, this doesn't sound like that interesting of a problem, although certainly it is an annoying one! However, part of what I would see as Luis' brilliance is in the more primary problems he is solving with this process.

In the case of Captcha, the real problem being solved pertains to my initial reference about the challenge of digitizing all the content of the world's printed matter, such as books. For more background on this digitizing and scanning challenge, you may want to refer to my previous posting from Jan 2007 "Books—the NEW old medium". Specifically, the problem is with all the words found in printed matter that scanning and conversion technology cannot make out, because the medium has a crease running through it, or it is partly missing, or other factors which make it impossible for the technology to recognize the words correctly.  Yet, show these words to almost any one of us and we can easily recognize the word.

So all those "fuzzy" words in Captchas are NOT just some random words that are blurred to fool a computer. Instead, these are the images of words which scanning technology has failed to recognize correctly! Luis refers to this specific application as ReCaptcha and you'll find much more information there, as well as instructions and free plugins for you to embed within your own sites, blogs, etc.

And that's just one side of why Luis von Ahn was awarded one of the MacArthur "Genius" awards and a Microsoft Research grant, for he has also managed to put these types of solutions into a game format that starts to look at solving these kinds of problems at a scale that is truly breathtaking! 

Solving World Problems or Playing Solitaire?

In his talks, Luis likes to use a very compelling metric of human-hours, and he often compares statistics on the the amount of human-hours that are "wasted", in his opinion, doing something like playing Solitaire on a computer. I too have always been amazed at the number of people I observe when walking down the aisles of an airplane, for example, who are hard at "work" playing Solitaire, but I had no idea just how much time is spent on this. According to the statistics that Luis uses, over 9 billion human-hours were expended playing Solitaire in 2003 alone! Better yet, he puts this into perspective by comparing this activity to such things as:

• The building of the New York City Empire State building, which consumed about 7 million human-hours, and thus equates to just 6.8 hours of collective Solitaire playing.
• Building the entire Panama Canal, which took 20 million human-hours and amounts to less than a day of collective solitaire playing!

Metadata for All Images?

Now imagine if we were able to put this kind of "human computation" to more effective use AND still do so within the format of games that people can enjoy doing themselves! One example is another one of Luis' creations, and one that has been running with staggering results for over three years called the ESP Game. As we've discussed many times, experiential "learning by doing" is often one of the best ways to learn about something new, so I'd encourage you to not only read about the ESP Game on that site, but to play it for awhile. (Caution: can be very addictive and time consuming!) When you do, you'll see how it puts two or more players (there is also a single player version) into a friendly competition by typing in descriptive words for a given photo (that metadata thing again), and they get points whenever they both type in the same word.

So what? While progress is being made in image recognition technology, this is still largely a problem that computers cannot do. And ask yourself, do YOU take the time to "tag" or create all the metadata for the photos and videos that YOU post, such as who and what is in the photo? Didn't think so. Yet by using this type of game format, the ESP Game has been running for over three years with no drop off in popularity and as of mid-2006, it was very fast, very cheap, and very accurate. If this were done as a popular online game site, it would be possible to label all the images on Google Image Search in just a few weeks! No surprise then that the ESP Game has already been licensed by Google in the form of the Google Image Labeler, and is used to improve the accuracy of the Google Image Search. We humans are relatively competitive animals and we like to do what we enjoy, so this approach appears to have a lot of promise.

Yes, but WHERE is that object in the photo?

Another problem that is even more challenging than identifying WHAT objects are in the image is identifying WHERE they are in the image. To do this, Luis has created another game called "PeekaBoom". The first player sees an image along with a word that describes an object within the image, and then clicks on the image where the named object is located. The second player sees only the object that the first player clicked on and types the word associated to that object. Once the second player guesses the correct word, the two players move on to the next image and switch roles. More details are explained in the video (you really should take the time to watch it!).

Human Computer Relations: Parasitic or Symbiotic?

Luis also notes how this transforms the current relationship between humans and computers from what he calls a parasitic relationship to a symbiotic one where:

"...humans solve some problems, computers solve others, and together we work to create a better world."

Sound far fetched? Well, in the less than two years that his limited experiment of the ESP Game has run, over 75,000 players have come up with over 15 million "agreements" (matched words). This rate would indicate that 5,000 players playing simultaneously could label all images on Google Images in about two months. Think about that...5,000 is NOT a very big number when you consider the numbers on many gaming sites. Therefore, it should be possible to label all the images on the Web in a few months. Again, I strongly recommend that you check out the video to get not only more details, but to see just how accurate, pragmatic, and promising this approach is. 

For example, it turns out that the results of a game such as PeekaBoom can in turn be used to help train computers to recognize objects and their location. Turns out that one of the reasons that computers are not yet very good at this type of object recognition and automated metadata generation is that there is very little data and examples to use to "train" the computers on how to do it. By capturing the results of all the human play in location identification of objects within images, this data can then be used to train computers to do the same thing"—allowing us move on to new challenges... and more fun.

Super Side Effects

I think you'll agree that this approach not only shows great promise in terms of solving some very large scale problems, but has some surprising and equally amazing "side effects", like how some people have used this to help them learn a language. This approach has spawned its own game called Babble, where two English-speaking players are shown a sentence in a foreign language that neither of them speak, and are presented with a list of possible meanings (in English) below each word. Players try to agree upon a set of English words that forms the most coherent sentence. The result is that this activity is surprisingly effective in translating foreign text into English without requiring anyone fluent in both languages. Think of the possibilities of this running at a larger scale!

Another "side effect" of this approach is how many players have noted that they end up finding other people who think very much like them, and thus they have a great sense of "intimacy" and closeness with their counterparts who play these games. Therefore, many  ask if they can find out who their anonymous competitors are to continue the conversation. At this point in time, all the game players are anonymous and no identities are revealed, but one could imagine this being used as a way to help discover other people "like you"—ones you'd want to meet and get to know better.

Common Sense Isn't that Common; yet!

And lastly from von Ahn's work, check out his new game Verbosity, which helps to generate what he calls "common sense facts" (again just more metadata really). One player is given a word and the other tries to guess what it is by completing fill-in-the-blank-type templates, such as "It is a type of ____" or "It contains ___".  The player who entered the original word can answer "true" or "false", but can't use the word itself. All this is very much like some party games that many of you have probably played, but the important difference here goes back to the original point of the summative network effect and how this can all be put to greater use. In the example that Luis shows in his presentation, the word "milk" would have some common sense facts such as;

• It is white
• It is a liquid
• It is often used to eat cereal
• It has lactose

Again, computers can not currently solve this kind of problem, and it is another an example of the need for massive amounts of metadata. Imagine if we started generating massive volumes of these "common sense facts" and they were readily available to all. 

More Competition = Less Carbon?

Lest you should think this is just a "one man show" from Luis von Ahn, I want to point out that there are many others who have been developing, adopting, and adapting similar models. For example, "Carbonrally: Carbon Challenge", which you can learn more about from the Nov. 20th, 2007 Webware post "Carbonrally: My carbon footprint's smaller than yours" by Martin LaMonica, is an application that is showing some great promise for improving the reduction in greenhouse gas emissions or the "carbon footprint" of individuals and organizations alike. Carbonrally adds the dimension of some fun and healthy competition to do better than others. As Martin describes it, Carbonrally is "tapping into people's tribal competitive spirit".

Whew!  That's quite an introduction to what I believe is both a powerful and profound pattern emerging—where the natural pursuit of fun, healthy competition, and challenges are combined into a game-based model that has already shown some of the ways we can solve large-scale present and future problems. It also creates a whole new relationship between us and technology. This model is not only interesting and fun, but it is a fascinating example of "user generated metadata", which I mentioned in my previous "Moving aLOM" posting.

Your Turn to Play!

Besides raising your awareness about "human computation" and the power of this approach, I also want to encourage all of us to put more time and energy into figuring out how we can inject more fun into work and other problem-solving situations. As you do so, I think you'll see an important job or task or problem  can be more fun if it's solved with some kind of game play, and where the solution remains very much a human one.

We can not, for now at least, expect computers to come up with such fun and game-based solutions by themselves! 

For starters, if you have websites or other applications where you have problems preventing spam or other misuses, consider taking advantage of some of the freely available plugins and nullities, such as those from the ReCaptcha site. Longer term though, please put some thought into which problems you could address with this model, and the ways you could do so by injecting the fun and challenge of a game-based approach into the more serious problems you need to solve...then share them with us here at Off Course - On Target.

I'm reminded of the great quote from Brian Suton-Smith who said:

"The opposite of play is not work; its depression!"

And I look forward to hearing all the innovative and creative ways you will come up with to solve problems—large and small—and replace depression with play. Have fun!

Comments

[selif]

Interesting that you call the text in captchas "slightly difficult to make out".

The big problem with captchas is that people with vision impairments have a lot of difficulty in solving those cursed things.

It's all fine for those with 20/20 vision, but there's some of us that just can't make out most of the bloody things. How about a captcha that at least lets you read the thing? Better yet, ask me a simple math question.

[selif]

just a quick followup to say that the captcha you have for comments took me several tries to get.... no idea if I'll be able to do it again or not for this one.

[Wayne]

Thanks for the feedback and perspectives. Agreed that there needs to be some choice when it comes to the use of these sign on methods to provide means that will work well for all. Keep in mind the the captchas "feature" is that they are bits of text that a scanning computer recognition program can't recognize and so we humans are helping to resolve these problems and doing some of the recognition for the scanning of the world's books, which is a good thing I think. However the flip side of this is that these captchas are going to be difficult to decipher, even for us supposedly smarter humans! As per my article I very much like this model of dual purpose and benefit where a very strong sign on encryption is provided and in the process of using it we get to help scan the world's books well. However as you so rightly point out, this is NOT a good solution for many and therefore there needs to be some sign on or protection alternatives offered that are specifically targeted at spanning a more complete range of what will work well for different individuals.

As for the captcha's used here on TypePad these are built into the comment system that TypePad uses and not something I choose or get to change so my regrets that it provides such a barrier to your contributions and I sincerely thank you for your tenacity and endurance in order to submit these comments. Most appreciated.

w
a
yne
=====

[Wayne]

BTW, if it makes you feel any better you might be amused to know that I too have to use these captchas when I go to post comments like these!

[mulberry bags uk factory]

So cute! I already like you on FB and also get your posts on Google Reader. :)

[online home based business]

I agree with most of the people that left comments, the article is well written and informative.

[how to take green coffee extract pills]

Thanks for any other informative web site. The place else may just I am getting that kind of information written in such a perfect approach? I've a mission that I'm simply now working on, and I've been at the look out for such information.