Haptics--Coming To A Cellphone Near You

Pretty soon we're not going to be able to trust any of our own senses. We are rapidly approaching the day when the line between illusion and reality won't be easily determined by casual examination. Science fiction is creeping further and further into the real world, and some of the time we don't even notice it. The phrase, "I'll believe it when I see it." has lost a good deal of it's meaning in recent decades. We see perfectly believable things all the time which are nothing but computer generated graphics, or clever use of Photoshop. We hear things all the time which are studio fabrications--like the beautiful diva's aria from the movie The Fifth Element. We taste and smell things, probably daily, that are chemical facsimiles of nature's offerings, and never give it a second thought. The only relatively trustworthy sense left in our arsenal is that of touch, and if the truth be told, that one's going on the reliability chopping-block too.

If you've read the Meow for any length of time, more specifically since last August, then you've probably seen my posts about haptics. In the first, I spent a little time discussing what the science of haptics is, and in the second, I shared some links sent to me by Dr. Gabriel Robles-De-La-Torre, founder of the International Society for Haptics, and a prominent researcher in the field. For those of you who don't have the background, I'll describe haptics the same way I did before:

Haptics uses computer technology to make you feel things that aren't really there, or feel things differently than they actually are. It's a virtual reality type of fooling the senses, but haptics involves the sense of touch.

It's not really an adequate definition, but it gets you started on the idea, and you can follow the links if you want to know more. There are explanations for how force can imply shape, and there are graphic demonstrations to help those of us who aren't neuroscientists and computer engineers like Dr. Robles-De-La-Torre. It's interesting stuff, and the Meow will still be here when you get back, so go have a look... Go on... Shoo!

Back now? Okay, then let's move on. The thing which got me "hooked on haptics" is reading the explanations of how Dr. R's work can be applied in the real world. I find it absolutely fascinating that your body could touch something, and feel something else. The good doctor has developed mechanisms where a person running their finger over something smooth will feel something sharp, or pointy, which is impressive enough in itself, but what is even more amazing is the growing list of ways the science can be applied to things which you and I use, and will probably grow to take for granted. I wrote before that there are hopes that haptics will advance to the point where surgeons could practice operations using a haptic interface, perfecting their technique before ever touching a real live patient, but feeling everything they would if a living breathing bleeding person lay on the table before them. Space technicians, too, might be able to get the "feel" of dangerous maneuvers, like docking a spaceship to an orbiting station, aiding the chances that when they perform the real thing in space, they won't damage either people or expensive space toys. There are versions of these technologies in use today, although there's a long way to go. More and more advanced forms of haptics technology could be saving lives in the very near future.

These applications are wonderful, of course, even noble, but you might be thinking, "Okay, but that stuff is all pretty distant from me. Although I might have surgery one day, I'm not going to be performing one any time soon, and space shuttle docking is even further out of my reach, so there's not much chance I'm going to even experience haptic technology, let alone take it for granted." Clearly, I have something further to offer for your haptics edification, or I wouldn't bother writing this post, so read on.

I got an email yesterday from our friend Dr. Robles-De-La-Torre, and he sent me to an article from The Economist (via TCMnet) that just came out this month, about some more applications of the kind of work that he's doing. The Economist explains how the science is moving beyond using only force to fool you into believing you're touching something you're not (as amazing as that is), and adding a new dimension--skin stretch:

Most of today's haptic devices rely on motors that either prod or vibrate the skin, but a new technology is emerging that is an even more flexible and effective means of stimulating the sense of touch: skin stretch. By laterally stretching the surface of the skin (without pushing or poking into it) it is possible to mimic the feeling of complex shapes and sensations. This is because the sense of touch seems to depend far more on the way in which the skin is deformed and stretched than it does on the degree of pressure applied. So it should be possible to recreate sensations purely by stretching skin, says Vincent Hayward, a researcher who first developed such a device at the Centre for Intelligent Machines at McGill University in Montreal, Canada.

"It's analogous to watching a TV screen," he says. The human eye can be tricked into seeing a range of colours on a video display, even though it really only consists of tiny red, green and blue dots. In much the same way the sense of touch can be fooled into feeling shapes and textures that are not there, says Dr Hayward.

Okay, cool. Skin stretch is another tool in the haptic bag of tricks, and the body gets fooled like it does with television, but this still doesn't answer how any of this becomes something you and I are likely to encounter. How does the science of haptics affect me!! (It is all about me, you know--or you, as the case may be.) Patience. The Economist points out some of the ways the rest of us may be experiencing haptics--it also includes a much better definition than mine of exactly what haptics is:

Haptics is the science of simulating pressure, texture, vibration and other sensations related to touch. The term is derived from a Greek word meaning "able to lay hold of". Devices that exploit haptics have been around for decades: many modern aeroplanes, for example, have haptic control columns that shake or vibrate to warn the pilot of an approaching stall. The technology has also found its way into video-game consoles, where it adds an extra layer of realism. Players can feel when they are veering off course in a driving game, or when they have been hit in a shooting game. Force-feedback technology, another offshoot of haptics, is used in robotic telesurgery and in surgical simulators to enable surgeons to feel resistance as they move their surgical instruments around, just as they would in conventional surgery. Even the "vibrate" mode on a mobile phone, which discreetly alerts the user to an incoming call or text message, is an example of haptics.

Okay, this is getting a little more down-to-Earth. We've all played video games, and used a cellphone, right? This haptics thing might apply to us after all! It might even be worth reading the rest of this post to see what's around the corner.

It's the cellphone that is taking haptics to a new level, in terms of real-world application. Some of the new mobile phones are coming sans keyboard, with nothing but a touchscreen interface, but one phone in particular, from Samsung, is taking it a step further, and adding a new haptic twist:

The SCH-W559 handset, which is so far available only in China, fools the user's sense of touch and mimics the feeling of pressing a mechanical button, even though the surface is actually completely flat. It is the latest example of a new breed of "haptic" technologies that do for the sense of touch what lifelike colour displays and hi-fi sound do for eyes and ears.

Wow. They're making you feel like you've pressed a button--that satisfying little click you feel--when all you've done is touch a screen. Unbelievable, huh? That whole skin stretch thing must be making its mark. I get why you would want the feel of real buttons, even if it's only a screen you're touching. You'd have to pay much closer visual attention if you couldn't tell by feel when a button had indeed been pushed. I'm mentally working on what the advantage is to a simulated button over a mechanical one, and the best I'm coming up with so far is that if you spill something it can't seep between the buttons and gum up the works. Dust would have a hard time finding its way into tech devices that are dust-sensitive, too, if the surface of the keyboard was actually a solid screen. Maybe mechanical buttons would wear out more quickly than simulated ones. I'm sure there are many other advantages, and Smart People somewhere have thought of most of them. They will become clearer to the rest of us as the technology gets more mature. What I suspect is part of the motivator now, though, for playing with haptics is that they can. It's just plain cool and fun to be able to do this stuff, whatever it's ultimate uses.

Of course, this whole world of haptics technology is one of those things that one day we'll all take for granted, and future generations might even understand. In a few years our kids will probably be able to build their own haptics interfaces for school projects, and it will have worked its way into children's toys as well as the grownups' gadgets. Mattel will sell some version or other of Barbie's Haptic Dream House , where you can pet little haptic Barbie pets with Barbie's hand, and feel it with yours, or feel the new Barbie Dream Carpet through Haptic Barbie's feet. Wouldn't that blur the line between reality and illusion--if you could feel what Barbie "feels"? Hey, you're scoffing now, but can you imagine what it would be like to go to sleep a hundred years ago and wake up today? Microwaves and cellphones alone would send you loopy. Haptic Barbie is not that much of a stretch. I wonder if her cellphone will come complete with haptic buttons?

Another Tour Around The Holodeck

First things first. Here's a link to a post I wrote a few weeks ago on advances in haptics technology, and how we're edging ever closer to the Holodeck from Star Trek: The Next Generation. I'll sum haptics up this way (keep in mind, this is not the official definition): haptics uses computer technology to make you feel things that aren't really there, or feel things differently than they actually are. It's a virtual reality type of fooling the senses, but haptics involves the sense of touch. I know this is a little challenging, but read the original post to get yourself up to speed. Wait, let me clarify. When I say up to speed, I mean up to my speed, which is about 2.5 miles per hour. I do not pretend to have any real insight into this whole field. In fact, until a few weeks ago, I didn't even know there was such a field, at least one that wasn't imaginary. Did you? Well, there is, and it's starting to grow some pretty interesting crops.

Last time, I wrote briefly about an article by Duncan Graham-Rowe, examining the work of Dr. Gabriel Robles-De-La-Torre, founder of the International Society for Haptics, who has developed ways to create physical illusions, like the feeling that you are running your hand over something sharp, or pointy, when you really just have your finger inside a thimble, attached to a computerized mechanical arm, which applies pressure to fool your mind into "feeling" different shapes. Confused? Well, here's the nifty part that led me to address this topic again so soon. I got an email today from Dr. Robles-De-La-Torre, with all sorts of links to help us understand his work better. I admit, I never expected such a smart person to be so helpful, and to voluntarily go out of his way to provide us with more information. "Really smart people" are supposed to be aloof and far above us all. I'm glad to be wrong, and also really glad to get all those links to follow up. It really has helped me to understand some more of what the doctor, and others working in the same field, are accomplishing.

I'll just list the links and where they lead. I've followed them all this morning, and done a fair bit of reading now, and I can assure you that the information is really helpful for understanding how sensory illusion can be created, and, more basically, how important the sense of touch is to our interaction with the world. It's not just dry reading, either. I really found everything I saw and read fascinating:

1) A demo of computer haptics. This is a short visual demonstration, but will help you get your head around the ideas.

2) A demo to help us understand what some illusory objects feel like. Be sure and read the text.

3) An article that discusses some related, earlier work with more illusory cases. The doctor explained to me that the technology goes beyond what I originally wrote about creating the illusion: "The cases mentioned in the article are even more surprising. It is possible to actually touch a combined real and virtual object and have our perception of the real object changed by the virtual one."

4) A broader article on why touch is important (but frequently misunderstood and underrated.) This article is amazing. Dr. Robles-De-La-Torre tells the story of Mr. Waterman, a man who permanently lost his sense of touch from the collarline down, and how he had to learn to control his body completely through visual reference. It's really an incredible tale. The doctor explains all the ways that we rely on touch information to function in the world, and how the lessons learned from Mr. Waterman relate to virtual environments. Read this one, if you can. For me it was a totally new look at the sense of touch, and gave me a new appreciation for its importance. God knew what he was doing when he gave us all five senses.

5) The good doctor also invited me (and thus, you) to keep up with the latest on haptics. Guess how? He's got a blog!! He calls it a newslog, which is much more legit sounding, but I'm a little more plebeian than that, so, a blog it is.

I hope you enjoy the opportunity to explore this subject some more. Anyone who reads the Meow much knows that I just eat this stuff up, and then ask for dessert. The world is an amazing place, and the accomplishments of people like Dr. Robles-De-La-Torre make it even more so. I'm looking forward to seeing where this research into haptic technology will lead us. There are exciting applications on the horizon, especially medical ones. Wouldn't it be great if surgeons could practice a procedure over and over with a haptic interface, before they ever took a scalpel to a real live human patient? What other precise and dangerous work will we one day be able to practice in realistic virtual environments? Will I ever be able to take a virtual vacation to the Moon? Will I ever be able to take a real vacation to the Moon? All of these questions have answers that are just around the corner. I'll be checking in on the doctor's world from time to time to see how close we're getting to the holodeck.

Next Step: The Holodeck

The world is getting more complex all the time, and an ever-increasing list of technologies can fool us into thinking something is a tangible, physical object, that doesn't actually exist, or make us believe something occurred that never happened, or cause us to experience something that seems so real, but turns out to be total illusion. 3D holographic images like the ones in the Haunted Mansion at Disneyland, can make us think something is right in front of our faces that is nothing more than clever video projection. Other amusement park rides can have us screaming in terror as we fly down the mouth of a giant dinosaur, or find ourselves in the middle of a battle with a Death Star; and, of course, we all know from recent journalistic experience that photographs can be altered to convince us that something occurred which actually didn't. However, the thing that all these illusions have in common is that they are primarily visual in nature. Some amusement park rides combine visual illusion with motion to make us believe we are flying and swooping at great speeds over treacherous terrain, but our brains wouldn't really buy it, if our eyes weren't seeing it. Seeing is believing, as far as our bodies are concerned.

So, would you believe that it's possible for the genius computer nerds to come up with ways to make your body and brain feel sensations that have you completely convinced you are physically touching shapes and textures that are nothing but computer generated fakes?--not something you see that convinces you it's real, but something you feel with your skin, and would even if your eyes were closed--like making your hand experience the sensation that it is petting a porcupine, or stroking sandpaper, when really it's stroking a smooth surface? Okay, from what I read today, they are doing this kind of thing. Maybe not at such a level that you'd believe you were petting a porcupine, but enough to make you feel like you were touching something sharp or pointy, when it was really all in your mind.

I don't get this at all. I think it's totally cool, but I really don't understand how it works. What I've been reading about are new developments in haptics, which, according to Encarta online dictionary is: "the science of applying tactile sensation to computer applications in order to enable users to receive feedback in the form of felt sensations. Haptics technology is used to train hand-eye coordination in tasks such as keyhole surgery and spaceship maneuvers." MIT's Technology Review has an article by Duncan Graham-Rowe, in which he looks at advances in the field of making computer's fool your brain into believing that your hand is feeling something other than it is. Graham-Rowe looks at research where they're using mechanical devices to simulate different surfaces, not by changing the shape of the device, but by varying the force the device applies to the hand that's touching it:

To create this illusion of sharpness, Robles-De-La-Torre, working with Carlo Alberto Avizzano and colleagues at the Scuola Superiore Sant'Anna in Pisa, Italy, used a haptic interface called GRAB, which consists of a thimble connected to the end of a motorized, extendable arm. A user is able to move the thimble freely when placing their forefinger in it. Then carefully controlled motors provide force feedback, so the thimble's movement is impeded in ways that create "virtual" surfaces.

By setting up the system so subjects can move only their finger along one axis, from left to right, the researchers were able make people feel like they were running a finger over a range of different sharp and pointed edges, just by applying lateral resistance to their movement. The sensation was so convincing that the subjects were even able to match the shapes of the edges to images of them, such as a saw tooth or a hump with a pointed peak.

I'm really not entirely comprehending this, but it sounds like we're one step closer to the holodeck from Star Trek: The Next Generation. If they can get our minds to believe illusions we feel as well as illusions we see, how far off can we be from the ability to completely immerse ourselves in totally illusory worlds? I'm not sure how I feel about all this. I mean, a vacation toy is one thing, but what are the more nefarious uses to which such technology could be put? Virtual torture chambers? Virtual coersion tool, say convincing a head of state that his family is being killed before his eyes, in order to make him agree to your terms? Of course, there are much more benificent uses of such technology. If the technology gets advanced enough, the physically impaired could know what it's like to experience things the rest of us take for granted. How about a walk on the beach for a paraplegic, or maybe skydiving? Of course, right now we're still looking at feeling sharp and pointy edges, not complete environmental transformation. I do wonder what's next, though. Don't you?

Hat tip: Futurismic