Lurching Towards More Natural Interfaces

In “Coming to our Senses?”  I introduced the topic of finding ways to make better use of our human senses to interact and interface with tools, technology, and machines, and even between ourselves. The LiveScribe Pulse digital pen and paper provides a good example of more “natural” and sense-based ways of interacting, communicating, and connecting.

For more of an overview of this new product, read my article  “LiveScribe Pulse Digital Pen + Paper”. This relatively new addition to digital pen and paper offers exciting capabilities, such as the ability to create sketches and drawings and embed audio explanations or sounds within them, or the ability to write in one language and have the words spoken through the speakers in another language.  Its combination of sight, sound, sketching, and writing reminds me that we are making progress, however slow and lurching, towards more natural interfaces and interactions and in particular, those which are and sensory or sense-based.  In addition, the Pulse pen lets you create some fascinating new "mashups" by mixing and matching written words with sketches, and drawings with speech and sounds.

Deafening Silence Coming to an End?

When you stop and think about it, isn’t it rather odd that we have been working this long with computers and other forms of capturing our thoughts and ideas without much effective use of sounds and speech?  We have the ability to capture audio so clearly and so easily, and yet we are only just now seeing an expansion of its application through podcasts, text to speech, speech to text, and voice recognition.  I suspect we will look back in a few years and view this time as akin to the days when movies were silent or before before audio recording and playback were possible.  Our children will surely ask how we could have ever put up with “silent” notes, sketches and drawings and to have managed to live without the ability to just create whatever interfaces and interactions we wanted on the fly.

In light of this, I’m hoping this article and others to follow, will make you aware of and encourage you to try for yourself, some of the additional new capabilities this digital pen puts into our hands.

Draw Your Own Interface

One of the best new features recently introduced by digital pens such as the Pulse and the Fly Pen is the ability to “draw your own interface”.

I’m fascinated by this overall capability for combining writing, drawing and programming. The Pulse pen lets you create your own interfaces anywhere you want (on digital paper) by just drawing them. For example, you can draw a cross symbol, which the Pulse pen understands for navigation of the pens various features.  You can then use this simple graphic interface to scroll through the programs on the pen, playback, and recording of audio, etc.

Programs can also be developed for the Pulse pen that enable the creation of any kind of interaction between the “ink” and sounds.  For example, a demo has you create a piano by drawing an eight note keyboard, which you can then “play” by touching the pen on the virtual key on the paper and hearing whichever note you touch through the speaker in the pen or via headphones that you can plug into the end of the pen. 

Another powerful capability lets you to write in one language and convert it to spoken words in another.

These examples of "draw your own interface" are limited and minor at this point to be sure, but just start to imagine what this opens up as we begin to draw our own interfaces.  And not just for literal human-computer interfaces but the ability to make anything we draw come to life, draw and play our own instruments, draw and listen to our own stories, etc.  Try out some of these new capabilities yourself.

My Very Own Purple Crayon at Last?!

These capabilities are about the closest thing yet to making one of my all-time favorite children’s stories, Harold and the Purple Crayon become a reality!  If you don’t know this 50-year-old classic by Crockett Johnson, then you should.  It is an absolute "must read "in my book.

In the story, whatever Harold draws with his purple crayon becomes real.  He wants to go for a walk so he draws himself a sidewalk.  He’s hungry, so he draws an apple tree and then adds a ferocious dragon to protect it for him. So scary is the dragon that Harold’s hands start shaking, causing the crayon to create water and waves, so Harold needs to draw a sailboat to climb into...and you get the idea. 

I encourage you to stop right now, and imagine you have this ability to have whatever you draw come to life and interact with you. What do you come up with for some other practical uses?  What does your wild imagination conjure up?   I suspect that you will soon have a list of amazing and truly exciting possibilities, and I hope you will take a few minutes to post them to your own site or share them here.

Natural Interfaces at Last?

These recent advances with digital pen, paper, and sound provide yet another significant marker on our journey towards truly “natural” and sense- based interfaces. I believe that there are some characteristics that are essential and relatively timeless when it comes to anything we humans are going to use effectively to communicate, live, learn, and work together.  Sketching and drawing will continue to be some of the primary means we use to communicate and will be highly effective ways of capturing, sharing, and explaining our thoughts and ideas. What could be more “natural” to us as humans than drawing, writing, and talking?  So too will the spoken and written word, acting, gesturing, storytelling, feeling and forming things with our hands,  and so on. 

Anything and everything that can be done to capitalize and leverage our very human senses and natural ways of interacting, communicating and connecting will be a move in the right direction. Stay tuned for many more postings I’ll be doing on this overall topic and trend towards more natural sense-based interfaces.

Comments

[LiquidMercury]

Great article! I share much of the same passion for the possibilities of this type of technology. To me, innovations like this are a wonderful sign that our species is beginning to think abstractly to solve problems, rather than the historic liner method of continually improving upon the last Greatest Thing.

The Pulse smartpen harmoniously utilizes not only the greatest features of present technologies, but those of the past. This synergy is what attracted me into a job as a Campus Rep for Livescribe; I want to be on the groundfloor of this companies products.

If you don't have a Pulse yet, you can get one from livescribe.com with the coupon code SCRIBE5A18 to recieve a 5% discount. I know its not much, but hey, every little bit counts in today's economy. :-)

[Wayne]

Glad you enjoyed the article and the pen. More so appreciated your observation of the deeper changes and opportunities this represents for more abstract thinking and problem solving.

Thanks for reading and contributing to OCOT.

w
a
yne
=====