Touching Sounds, Missing Limbs, And Other Weird Haptic Surfaces

These tactile feedback research projects give us insight into where touch interfaces, pioneered by smartphone makers, are headed next.

Covering our entire bodies and existing as our largest perceptive organ, the potential power of skin has somehow been woefully underutilized by technologists. But with touch and gesture technologies having become increasingly ubiquitous over the past few years, the world may finally be ready for the next generation of haptic technology applications.

Based on the Greek word haptikos--meaning “the sense of touch”--haptic technology (otherwise known as tactile feedback technology) has been around in some form for the best part of the last 50 years. Up until now, however, limited computing power and a range of other technical challenges have meant that this technology never worked as well as it could.

That may now be changing, though, thanks to the work of three groundbreaking applications which promise to use haptics to alter everything from the way we enjoy entertainment, to how we develop and build prosthetic limbs.

Has tactile technology finally come of age?

Disney Makes Touching Entertainment

As one of the world’s largest media and entertainment companies, Disney is constantly on the lookout for new and groundbreaking technologies to take advantage of. For the past several years, Disney Research has been hard at work on developing a haptic feedback algorithm capable of simulating tactile 3D geometric features, such as bumps, on a flat touchscreen surface. The technology works by modulating friction forces between the user’s finger and the touch screen by modifying the voltage of the display. The result is a feedback sensation that not only increase the effectiveness of touch interactions, but also provides users with a more realistic and immersive experience than traditional touchscreens.

“We proposed that the percept of a 3D bump is created when local gradients of the rendered virtual surface are mapped to lateral friction forces,” says Ivan Poupyrev, Principal Research Scientist at Disney Research, who Fast Company named one of its most creative people in business in 2013. At least in the short term, the haptic feedback algorithm is likely to find a home in Disney’s amusement parks, which Poupyrev hopes to one day make completely interactive.

“Because I work for an entertainment company, that is where I focus on thinking of applications for,” he says. “In this way most of my work has been thinking about how haptic feedback could be used to help people interact with graphics media. I’ll give you an example: you could have a large table-sized touchscreen designed as a sandpit, over which people must run their hands looking for a key, which has a very different feeling from the surrounding sandy areas. Another application might be the user dragging a character across a map, and feeling the stones, wooden planks and other surfaces beneath their feet as they do this. It’s all about making interactions with virtual objects more realistic.”

At the same time, as an acclaimed interface designer, Poupyrev acknowledges that the technology could have almost unlimited application. “For instance it could be used very well as part of a user interface, where you would be able to more easily feel the depth of having windows on top of windows,” he says.

Currently the algorithm can only recreate the feeling of bumpy surfaces, although finer textures are next on the list: meaning that one day users may be able to have the tactile experience of scan through a material swatch book using their future-generation iPad. “At the moment the algorithm we’ve designed wouldn’t work so well for textures, since our perception of bumps is different to our perception of textures,” Poupyrev says. “It’s definitely something we’re interested in pursuing, but it remains to be seen how far we can go in that direction. Currently it’s an open research question--although it’s something that should certainly be approached.”

A Wikipedia For Feelings?

Imagine being able to send a real, three-dimensional tactile feeling to another person using your smartphone. More than that, imagine being able to send it as an MP3 music file. As farfetched as this might sound, it is the research topic of Japan’s Tachi Lab, a world leading laboratory in the field of virtual reality, augmented reality and telexistence.

One of the key researchers at Tachi Lab is Masashi Nakatani, co-creator of the TECHTILE toolkit, a prototype tool for the design and creation of haptic media. “The technology I have developed can convey the sense of touch information in terms of pressure or force feedback using sound waves,” Nakatani says. His work is based on the observation that while the air vibration an object makes is usually perceived as sound, if touched directly it can instead be perceived as a tactile sensation. As a result--using the correct microphones, speakers and vibrators--an auditory sensation can be recorded as a sequence of sound waves and then transformed back into a hyper-realistic haptic sensation at another location.

For example, a person wanting to deliver the haptic sensation of a plastic cup full of ball bearings could attach a haptic recorder (a cheap microphone in Nakatani’s studies) to the bottom of a cup using scotch tape, and then record the sound the ball bearings make as they drop into the cup as it fills up. Using this sound information, the haptic sense of collision and rotation of balls can then be copied to another cup in real-time.

“As we move forward, it should also become possible to design tactile feelings in the same way that we can with sounds or images,” Nakatani continues. Ultimately, he says, his plan is to create a library for “feelings” (a bit like a tactile Wikipedia) not entirely dissimilar to the kind of sound libraries used for putting together movies. “My goal is to gather a lot of tactile impression-data from the general public, which they can upload using YouTube, Flickr, or other video and sound sharing websites. This would create a large tactile archive through which you could access to download different feelings and sensations.”

Putting Feeling Back in Phantom Limbs

You might be familiar with the idea of phantom limbs: the sensations experienced by a person who has had a limb amputated that the limb in question is still there. A similar idea is at the heart of several medical research projects which may, for their users, represent the most important use of haptic technology available.

Although brain-controlled prosthetics have taken enormous strides over the past 15 years, the concept of incorporating tactile feedback (i.e. allowing a person to feel through their prosthetics) has been largely absent. And while simple advances have been made in this capacity--for instance, offering audio feedback to illustrate resistance when a prosthetic pushes an external object--these developments pale in comparison to the possibilities offered by haptic technology.

One of the most advanced haptic feedback devices is a prosthetic arm developed by researchers at the Johns Hopkins University Applied Physics Laboratory in Laurel, Maryland. Constructed as part of a US Department of Defense research program which has spent $144 million since 2006 to improve prosthetics for injured soldiers returning from war zones, the arm contains more than 100 sensors to detect sensations ranging from temperature to pressure. Scientists at both the University of Pittsburgh and the California Institute of Technology in Pasadena are now seeking regulatory approval to use brain stimulation as a way of delivering sensory feedback to patients from their prosthetic limbs.

“Applications like this shows how useful haptic technology could be in medicine and in rehabilitation,” says Masashi Nakatani, expressing his admiration for this use of his field of expertise; just one more way in which haptic feedback can change the world.