Off Course-On Target

Great Thoughts

Here's another posting derived from my ever growing list of items to blog about.  This one springs from an article I read in the IEEE Spectrum back in Jan 2007.   

Did I mention I’m behind in my postings and the list is growing? However on the plus side I like this delay because it acts as a check on just how relevant and useful things are.  If it stands the test of time, it gets posted, and this one passed the test with flying colors.

The title of this posting “Great Thoughts” is the same as the article by Robert W. Lucky (IEEE Fellow), now retired. In it, he wrote: 

“There is a theory of creativity that holds that creativity is most often the product of the unexpected intersection of two previously unconnected thoughts. If you are thinking very hard about one such thought, perhaps you are suppressing the other thoughts that could connect with it. On the other hand, if your mind is a perfect blank…”

This is an interesting conundrum: not focusing too much on one thing yet not shutting out all thought, something that my ADD riddled brain can't even comprehend!

But for me, the takeaway here has more to do with the question of what constitutes “new” these days?  What makes something original?  What constitutes plagiarism or copying?  What is at the heart of innovation and creativity in the world today?

In this era where more and more things are mashups, a new assembly of previously existing things, is there any such thing as an original idea, song, drawing, text, etc.?  The answer is more "yes" than "no", I think, but it is worth noting that most of the time what is new and what makes something unique is not the bits and pieces that it is made from, but the selection of certain items and the way they are assembled into a new whole. And most often, the value of this new assembly is much greater than the sum of the parts.

This concept is nothing new really. Think of some fundamental examples such as text, where we have almost all existing words available in the dictionary (although the list is growing and evolving all the time), yet from this finite set of letters and words we can create an infinite number of written documents (e.g. articles, books, poems, and scripts). Or music where we have another finite set of notes and chords that enable us to create an infinite number of new songs, melodies, and even ring tones!

This combining of existing things to create new things, the heart of the Snowflake Effect is what makes it a scalable model. Not that we won’t also be creating new bits and pieces along the way, such as the new words which are added to dictionaries every year, but for the most part I think most of what is new and valuable, most of the innovation and creativity we are and will be seeing, comes from the serendipitous intersection of two or more previously existing things, be they thoughts, ideas, notes of music, letters and words, lines, arcs or circles.

So here’s to celebrating and fostering the creativity in all of us by focusing on being the unique snowflakes we all are and designing more and more new original ideas and “stuff” from what we already have around us.

Finding More by Looking Less

For the past 2 weeks, I’ve noticed that I’ve been recalling a favorite metaphor many times in a great variety of conversations and communications with different people so I take this as a reminder to post a bit more about it. 

Many of you know my fondness for serendipity, which I describe simply as the unexpected discovery of something better while searching for something else. I’ll leave the topic of serendipity to another longer exploration and explanation, but for me serendipity is the way I live my life, both professionally and personally, and I believe it is the most powerful way to success.

Today I wanted to just mention one technique I have found to be very useful in the pursuit of more serendipitous discoveries and that is to purposefully NOT focus directly on whatever it is that you are looking for at the time. I think I first hit upon this  technique when I learned about an interesting fact about the way our eyes work and, specifically, why we have a “blind spot” in our vision.  As I understand it, because of the way the optic nerve attaches to the back of the eye, we have a blind spot where it attaches. The best example of how this comes into play is what happens when you are looking at the stars. The next time you get a chance to look up on a starry night, try to find either a faint star, galaxy, or a cluster of stars.  What you will find is that you can only really see these when you do NOT look directly at them but instead focus your attention off to one side.

I thought this effect was quite fascinating and as usual, I’ve turned this into a general concept which seems to hold true in other instances, not just with sight. Ever notice that when you get stuck trying to remember something or are solving a tough problem, the harder you try to focus on it or solve it, the less you succeed?  And then when your mind has wandered off onto something else or you do something completely different, the answer or solution comes to you in a flash?

On an even more general scale and conceptual level, I have observed that this same phenomenon is true. For example, as I’ve studied people and organizations who are exceptionally successful, I’ve noticed that they are usually the ones who focus the least directly on what other might see as their main goal. In business, for example, the most successful and profitable businesses seem to focus the least on profitability itself and more on such things as figuring out a fundamental value proposition for their customers and finding as many innovative ways as possible to deliver on it. 

Or in other cases, an organization starts out assuming it is going after one thing and discovers that the real value is in something else. Netflix, the “DVD in the mail” subscription company comes to mind as a good example. You’d think that their focus would be on the movies, having the most available, getting the ones you want to you the fastest, etc.  And these are indeed important elements of their success. However, their greatest value is helping people find “just the right movie” to watch. This capability comes in the form of their recommender system known as Cinemax. This technology takes the input you provide on which movies you like and don’t like and uses this information to create recommendations that become almost frighteningly accurate. If you’re interested in more on this, I wrote up a longer explanation recently on the Long Slow Chat called “$1 Million Dollar Snowflake Prize”, which details a fascinating competition Netflix is running for the first person who can increase the effectiveness of the Cinemax system by 10%. 

Most recently, my friend Marcia Conner sent me a note to her amusement on serendipitously discovering some recent research by Paul Whitmore Sas that seems to essentially support this same model of indirection. Paul titled his work descriptively as “How to spot your life’s guiding star” and you can download the full dissertation if you want more.  He talks about the Delmore Effect (named after the apparently failed poet Delmore Schwartz), which is the situation where:

“most people tend to set much more explicit goals for low priority domains than for their most important ambitions.”

To prevent this from happening, Paul’s research was based on the same star gazing example that I cited earlier and which Paul referenced as follows:

"Astronomers know to look slightly away from the point at which they expect to locate a star. Analogously, when a person aims to most clearly articulate her own guiding goals, she would be more successful by calling to mind the values which are peripherally related and supportive of her complete self.

Instead of directly confronting the value of greatest import, a person can become more articulate about their central life goals by taking a slightly less direct approach."

Interesting reading.  I recommend you give this method of indirection at try and see if you don’t find more by looking less.

The Fourth Screen and Counting

I don’t get to see much TV in this new grand adventure I’m on, not that I ever seemed to have time to watch much before either, so many of you may have seen this before but it was new to me. 

Here is great example of why I’ve often suggested that new perspectives are so powerful and valuable and I try to find and create as many as I can.  Oh, and yes this is an advertisement; get over it!  Great content often comes from the most unlikely sources.  This one comes from Nokia.

As you’ll see when you watch the video, they present a fascinating and I think very useful way of seeing the evolution of “screens” and media as moving from:

1. Movie theatre screens
2. Television screens
3. Computer screens
4. mobile/smart phone screens

It’s been rather amazing for me to watch how the screen has quietly embedded itself into our understanding of what a “phone” is. Not so many years ago the word phone would have been would have been thought to mean a device for talking only.  The notion of “watching a phone” would have sounded ridiculous and still sounds rather funny when you say it like that.

By Any Name; Handy is the Game

They now go by many other names such as mobile, cell, smartphone, PDA, and yes even the iPhone of late.  There is even this long list of “Mobile phone terms across the world”.  But I think I like the one common in many European countries the best where they are often called a “handy”

Handy (plural form Handys), pronounced /hɛndi/, a pseudo-anglicism that is used in Austria and Germany for a mobile phone (rare alternative spelling: Händi). In German, the word "Handy" has the meaning of "Hand-Telefon" or "handgehaltenes Mobiltelefon" (translated to English: "handheld mobile telephone"). The term possibly derived from the 1940s product name Handie-Talkie for a handheld military radio. (The backpack version was introduced as Walkie-Talkie.)               - entry from Wikipedia

Whatever the origin or spelling “handy” is certainly an accurate description of these amazing devices we used to call phones and only use to talk on.  And it is this dramatic increase in utility that I think is the really important part of the transformation of these “screens” have brought to our world.

Mashing, Melting and Morphing

More than just the addition of a screen these devices are the mashup or melting pot of a lot of different devices and capabilities from many different directions and original purposes including digital cameras, MP3 or music players, TV and video players, and PCs.  This is what I thought was most clever and useful about the way the Nokia Fourth Screen video presented the evolutionary stages, as you can see how each successive screen swallows the one before it.

* Please also note that none of these stages has eliminated the one before it and  ALL remain valuable and in great use!  As I’ve often pointed out our history is one where technology never really dies and while disruptive, the next new stage does not eliminate the one before it.

While the addition of a screen that is capable of displaying most any kind of content is important, so much more so than the screen it has been the freedom that comes from mobility that has been the game changer.  In fact the screen, while necessary is most often quite inferior to the ones that came before.  A reverse trend if you notice going through the four screens is that each one has been smaller and arguably inferior in visual quality to the one that came before!   However in each case of the steps along the way we gained more freedom, choice and opportunity of both what we could have on the screen, where we could have the screen and our control over the interactions with with other people and their screens.

These new phones have also brought us the closest thing to wireless, untethered and unfettered we’ve ever been.  WiFi can be useful when used with things like desktop PCs and laptops for other reasons, but it only really delivers on its true value IMHO when it enables you to be completely free to have what you want whenever and wherever you are.  Even with the shrinking size of laptops and the new “netbooks” they are still large enough and cumbersome enough that you often don’t have them in your hands at your fingertips when you need them.  The “4th screen”, with it’s ability to slip into your pocket or clip onto your belt or just stay in your hand, is the most likely to be there whenever you need it and brings us closest yet of truly having the world at our fingertips anytime, anywhere.

Evolving the Screen to Death

And will there be fifth screen?  I think so and more after that.  What I’d see coming next is the further shrinking and transparency of the overall physical devices and the elimination of the fixed nature of the screens and input devices.  In the case of the screens what I mean is when there is no limit to the size of the screen and you can view, and you can have whatever size makes sense for the situation you are in.  Sometimes all I need is a screen big enough to tell me the time, temperature, a phone number or a factual answer to a question.  Some times black and white is fine too.  But other times I want screens the size of whole walls or the size of buildings, with very high definition sound and visual fidelity.  And I want them to be there when I need them.  So for example in a previous post on Coming to our Senses: Touch we saw some advances heading toward having screens that can be stretched to any size you’d like, so you could have one rolled up in your pocket or purse and whenever you needed a screen you could simply stretch it out to whatever size you’d like from postage stamp to a wall.  Or perhaps we’ll see mainstream versions of “virtual screens” that might fit on the rim of your glasses or slip over your ear and let you see any size screen in front of you that you’d like. 

After that, the Sixth Screen perhaps?  By this stage I think there will be no screen at all!  Again in my posting Coming to our Senses: Touch in the section on “Digital Clay” I laid out the coming of fully 3D images that might be along the lines of holograms for example and which would also include haptic bi-directional feedback so I can both feel what I see and shape or change it with my hands. 

WYSIWYG superceded by WYNIWYG

The trick however to having all this amazing advances in “screens” is not so much in the screens themselves but in the freedom to see who and what I need whenever and wherever I want to.  This is all part of the trend I’ve been calling the Snowflake Effect or getting it “just right” and what I believe Gartner first started calling WYNIWYG: What You Need Is What You Get.

“Context driven networks will connect us in new and more meaningful ways than the bandwidth-focused networks of today. The idea is no longer WYSIWYG (What You See is What You Get), but now we look for WYNIWYG (What You NEED Is What You Get)! Networks will be able to anticipate your needs and make information available to you when you need it.”   
 - Gartner Emerging Trends 2008

For me, the destination of this evolution of the screen is when it disappears completely leaving me free to see (hear, touch and even smell) just what, who, when and where I want.  I don’t know or care if this will be the 5th, 6th or 21st “screen” but it will be just right 4me and it will be here much sooner than most people suspect.

Coming to Our Senses: Touch

As I sit here typing on my keyboard and clicking my mouse, I find myself desperately hoping we may finally be moving beyond our narrow view of what form interfaces between humans and technology can take to one where we can be more creative and innovative...and effective!

Seems to me that today, almost every interface is limited to no more than a variation on more button pushing!  Computers, for example, come with keyboards and mice, and we even carry this metaphor of button-pushing onto the screen with “virtual buttons” that we have to "push".  And  even smaller devices, such as mobile phones, are designed with button-driven interfaces.

Recently we’ve transferred button-pushing over to “touch” screens on phones and we think that this is just "oh so cool' and innovative.  Give me a break!  Most examples don’t even work as well as a real button!  Ever tried to type more than a word or two on a touch screen keyboard?

Enough with the buttons already!!

There is a glimmer of hope, however, in the world of interfaces!  Some of the new phones from Samsung (currently only available in China) use Immersion’s VibeTonz technology, a tactile feedback that simulates the feeling of pushing a mechanical button even though the surface is completely flat and does not move. It is the latest example of a new breed of haptic technologies that do for the sense of touch what high definition displays and surround sound do for eyes and ears. (see links at end for more on haptic technologies)

But I believe that the lack of feedback in touch screen devices will severely limit their success and adoption, and that someday we will shake our heads when we remember how we lived “out of touch” for so long without the use of touch in interfaces.

Hope on the Horizon?

But surely we can be more creative and innovative with our interactions and interfaces than limiting every solution to simply a variation on more button pushing!  Currently we are seeing innovative uses of touch screens that provide a better interface and interaction with our digital world. Applications on the iPhone and other smartphones, for example, let you flick or flip things around, such as a set of digital photos or stacks of music CD artwork. This use of “gestures” as a way of interacting has great promise and recent examples, such as the Wii controller and some of the multi-person, multi-touch interfaces on digital surfaces are showing great promise. 

Since touch works in both directions—as feedback to us and feedback from us—I'm delighted to see the use of haptic interfaces with the Wii controller, which generates physical feedback to your hands as you strike a tennis ball or golf ball, for example, and simultaneously uses your physical input to provide velocity and angle as you swing a golf club and the resultant force applied to the ball. Airplane and car simulation game controllers (joy sticks, steering wheel, brake & gas pedals) also provide similar haptic feedback. Flight simulators for training pilots and amusement park “virtual reality rides” show us how truly real, natural, and effective touch interfaces can be.

Let’s Stop Flapping and Start Flying!

But these incremental improvements, like touch screen buttons and limited haptic feedback, also serve to continue our "flapping" behavior of mimicking physical button pushing.  Our tendency to focus on copying what worked well in the past, such as designing planes that mimic flapping wings on birds or pushing buttons for machines, is in fact holding us back from truly "flying"!  If you are not familiar with my use of this metaphor of "flapping vs. flying", take a few minutes to read or listen to my story on “Confusing Flapping with Flying”. 

To start truly "flying", that is developing truly innovative solutions, I think we need to focus on the use of touch and tactile feedback as some of the essential elements of successful interactions and interfaces.  For example, the ability for us to feel and touch any object even though it is not “really there” and similarly for our technology to acquire the sense of feel and touch.

Less Virtual, More Real

Early applications of virtual reality used to seem pretty cool but now they seem, as my daughter liked to say to me, “Oh so yesterday, Dad!”.  These early efforts failed because they were too virtual and too far from reality. Today, we can see clearly how much of a difference it makes when, for example, motion, sound ,and color is added to movies. More recently ,the addition of some 3D, high definition, and high fidelity has narrowed this "reality gap" to our eyes and ears. And virtual reality applications required all that extraneous equipment, such as headsets, goggles, helmets, gloves and even full body suits, to provide the feedback to or from us, which severely separate or distanced us from the reality of the situation.

So, what if we moved towards being more real and more natural?  What if we had digital models with haptic interfaces that let us feel, handle, turn, assemble and disassemble them? How about applying full haptic feedback to a holographic images? Imagine if holographs provided not just stunning visual 3D reality, but also came with haptic feedback to your hands, so that you could feel their edges, curves, and shapes?

Digital Clay:  Digital Prototyping gets Real

Moving along the reality scale, we already have very functional and bidirectional digital prototyping via 3D scanners and 3D printers.  Together these can both create digital models from existing objects (input) as well create new fully-functional physical objects (output). Such digital prototyping can all be done BEFORE we start using large amounts of materials and energy to produce the “real things” ** and before we use the energy and resources to create these physical entities. This capability may well turn out to be one of the single greatest ways of conserving resources and energy and bringing some much needed practical reality to the “green” movement.

Leaping (and lurching) further forward, wouldn’t it be great have true virtual digital prototyping that gave us an experience equal to that of having the real thing in our hands?  How about if we could move beyond digital clay to “real clay” where haptic sensors and generators were built-in at the nano level or could be more like the holographic + haptic models I mentioned previously.  With this digital clay in your hands, you could now create any shapes and objects you like "by hand" or you could add some "clay" to your 3D printed models or you could use 3D printing to create a whole model out of digital clay and use this as the starting point of your design work or testing.

AND, now how about if we also hook up some of that digital clay “backwards”, so that rather than limiting the feedback to one direction—from our hands to the clay— it could also move from the clay to our hands?!

Say what, Wayne?

Well, imagine you are on a team whose members reside in many different locations. Your team is designing some physical object, such as a new smart phone. Rather than “just” having a 3D printer create the latest design iteration of the phone on your desk as a “solid”, what if every team member had one made out of digital clay and whenever you changed the phone's shape or moved some buttons, for example, the other phones at the other locations changed accordingly. You each would have a working version.

THAT sounds a lot more real and a lot less virtual to me, and it's all possible by adding the use of our sense of touch to our interactions with technology.

** BTW, this example starts to show how increasingly strained the concept of “real” is becoming and what I see as another trend towards the closing and ultimate elimination of the “reality gap”.  But I’ll expand more upon this in a future posting.

eSkin

The integration of sensory feedback and control within technology provides another look at touch working in reverse (so to speak), and it is equally important and fascinating.  One recent example is the progress being made with “e-Skin” for robots. This new material  is the first that combines two key characteristics:

• The ability to stretch like rubber
• The ability to conduct electricity like metal into a single substance

They’ve accomplished this with carbon nanotubes and used it to allow robots or machines to “feel” heat, pressure, and texture. I was equally intrigued with e-skin's potential to to stretch into larger sizes and onto curved non-flat shapes. I can imagine how this could lead to having displays in your pocket that you can pull out, unroll, and stretch into a large scale screen for viewing, or applications as "skin" grafts for burn victims.

Hiding Everything but the Benefits

Some have suggested that Jules Verne completely missed the computer and all its technology in his futuristic books such as 2000 Leagues Under the Sea, and Around the World in Eighty Days. But I think Verne got it completely right by NOT focusing on or imagining computers and technology. He simply assumed that the compute power was just “there” behind the scenes. I loved the way he kept the focus on the fantastic capabilities and the grand adventures we would have if such capabilities existed.  Who cares how or what enables them?

So it is my belief and my hope that we are finally seeing technology dissolve into the background and into ourselves so that the focus can be on people, challenges, solutions, and great adventures in life. I know I'm signed up for the pursuit of joy and as such I welcome and champion any and all movement towards more natural interfaces and the use of all of our human senses to communicate and work more effectively with each other and with our supporting cast of machines and technology.

BTW, here are links to some good animation on haptic interfaces/technology from Grabriel Robes-De-la-Torre and a full paper on “Haptic interfaces and Devices”.

Coming to our Senses

If you have been kind enough to listen to me or read some of my entries on OCOT or my other writings, you may have correctly concluded that I suffer from a severe case of Attention Deficit Disorder (ADD). Contrary to popular opinion (and even some “experts”), ADD is not about a LACK of “attention” but rather, an excess of it! 

And Now a Word from My Symptom

Those of us with ADD are unable to pay attention to just ONE thing at a time and are instead forced to pay attention to every sensory input all of the time. We try to make sense of things when EVERY thing is demanding our focus. All of our sensory inputs are dealt with equally and simultaneously. 

What we can’t do, for example, is turn down the volume on every sound except the one we want to focus on. We can't tune out the movements we see around us or choose not feel every input through our skin. In other words, we can only pay attention to EVERYTHING, all at the same time and all at equal levels.

I’m one of the fortunate few with ADD who, long before I knew what it even was, began to develop  coping skills and techniques to survive with these traits. They didn’t seem like anything special to me.  For the longest time, I just assumed everyone else was having the same experience.  And somehow I managed to start turning these “bugs” into features that have enabled me to be as successful as I am and to do what I do. If everything demands your attention and you find everything interesting, then life is very full!

If you are interested in learning more about ADD, look at some of these  alternative descriptions.  And on the topic of attention itself, I know of no better source than that of my dear friend and fellow ADDer Marcia Conner and her Ageless Learner site’s section on Attention and on How Adults Learn.

But now, back to the topic at hand (see what I mean?!)

It All ADDs Up

So why are my ADD tendencies relevant here?  Because its a way to introduce a meta theme that I'd like to address on OCOT. Just as my own natural sensory interface presents challenges, so too are the challenges of developing human/machine sense-based interfaces. Over a series of postings, I'd like to address:

• The use of our human senses as ways through which we communicate and interact with machines, computers, and technology, and ourselves. 
• Other interrelated topics both individually and as a collective whole. 

I’ll do my best to stay "on target" as I wander around "off course" and divide my pursuit of this trend toward more natural and sense-based interfaces into several separate posts. I see it playing a critical role in the future I am both designing and imagining.  

The Missing Link?

Interestingly, this whole topic of natural or sense-based interfaces does not seem to have any common singular term or definition. Or perhaps I just have not found it and some of you can point me to it?  I’m aware of categories such as:

• HCI  Human Computer Interaction/interface
• HMI Human Machine Interface/interaction
• User Interface

But these all seem to be differentiated based on interactions with specific types of technology, computers only, or segmented into individual human senses, such as touch with the use of haptic interfaces and technology with tactile feedback . And I can find definitions of HCI, HMI such as this one from Wikipedia that describe them as:

“...the 'layer' that separates a human that is operating a machine from the machine itself.”

However, my interest is in the larger meta theme of using all human senses and eliminating the “separation layer” between machines and ourselves so that they are invisible and we can directly connect and communicate through our senses. I’m want to ensure that we are taking the more holistic and inclusive view of the use of ALL of our senses, not only because we have so many that are being underused, but also because it is often the combination of multiple senses that make up our best experiences and interactions.

Making Sense of and Better Use of Our Senses

So here’s what I’m going to do. I’ll address the meta theme of sense-based and more natural interfaces through a variety of posts, podcasts, and perhaps some interviews over the next few months. I’ll alternate between the big picture view of the use of senses in general and the use of them collectively, then look at the use of individual senses such as touch and feel, audio, visual, and so on. I will also cover the use of combinations of senses for better, more innovative designs and solutions for interacting with technology...and each other. Add to this the coverage of tools, technology, and examples of real world applications of sensory-based and natural interfaces and I think we’ll have a lot of fun and learn a lot in the process. These posts will be intermixed with other topics on a regular basis and spread over many months for as long as my and your interest continues to be high.

Oh!  Look at that cool blue butterfly that just flew by!  Hmm, I wonder what kind of plane that is flying overhead?  Now what was I saying?  Oh yeah, senses, interfaces, human interaction... 

Hang on, fasten your seatbelts low and tight, we’re about to take off on what should prove to be a fun and very "off course", very "on target" sensory journey!