The Seeing Tongue
By: Peter Weiss
From: Science News, Vol. 160, No. 9, Sept. 1, 2001, p. 140.

Blind since birth, Marie-Laure Martin had always thought that candle flames
were big balls of fire. The 39-year-old woman couldn't see the flames
themselves, but she could sense the candle's aura of heat. 

Last October, she saw a candle flame for the first time. She was stunned by
how small it actually was and how it danced. There's a second marvel here:
She saw it all with her tongue. 

The tongue, an organ of taste and touch, may seem like an unlikely substitute
for the eyes. After all, it's usually hidden inside the mouth, insensitive to
light, and not connected to optic nerves. However, a growing body of research
indicates that the tongue may in fact be the second-best place on the body
for receiving visual information from the world and transmitting it to the
brain. 

Researchers at the University of Wisconsin at Madison are developing this
tongue-stimulating system, which translates images detected by a camera into
a pattern of electric pulses that trigger touch receptors. The scientists say
that volunteers testing the prototype soon lose awareness of on-the-tongue
sensations. They then perceive the stimulation as shapes and features in
space. Their tongue becomes a surrogate eye.  

Earlier research had used the skin as a route for images to reach the nervous
system. That people can decode nerve pulses as visual information when they
come from sources other than the eyes shows how adaptable, or plastic, the
brain is, says Wisconsin neuroscientist and physician Paul Bach-y-Rita, one
of the device's inventors. 

"You don't see with the eyes. You see with the brain," he contends. An image,
once it reaches an eye's retina, "becomes nerve pulses no different from
those from the big toe," he says. To see, people rely on the brain's ability
to interpret those signals correctly. 

With that in mind, he and his colleagues propose that restoring sight is only
one of the many trajectories for their research. Restoring stability to those
with balance disorders is another. So is bestowing people with brand new
senses, such as the capability to use heat to see in the dark. 

Restoring lost vision 

First things first, however, and for the Wisconsin scientists that means
restoring lost vision. Swapping the sense of touch for sight is not a new
idea. In the 1960s, Bach-y-Rita, his colleagues, and other scientists began
developing and testing devices that enable the skin of blind people to pick
up visual information. 

For Bach-y-Rita, the experiments also provided insight into the brain's
plasticity. His more general goal has been to find out how well one sense can
take the place of another. 

Until the 1980s, "one of the axioms of neuroscience was that there was no
plasticity in the adult central nervous system," says Edward Taub of the
University of Alabama in Birmingham. Today, the field has turned around in
response to many studies, including Bach-y-Rita's. Now, scientists view the
brain as almost as malleable in old age as in youth, he adds. 

The idea of tongue as eye evolved from the earlier skin-as-eye studies.
Bach-y-Rita and his coworkers had been placing touch-stimulating arrays on
areas of people's skin, such as the back and the abdomen. The scientists used
either electrodes or little buzzers to excite nerve endings of the skin in a
pattern that corresponded to visual images. 

They found that after receiving training, blind people using these systems
could recognize shapes and track motion. Some subjects could perceive the
motion of a ball rolling down an inclined plane and bat it as it rolled off
the plane's edge. Others could carry out an assembly-line task at an
electronics plant. It required them to recognize glass tubes lacking solder
and then to deposit some solder into those tubes. 

These results impressed Bach-y-Rita and his colleagues enough to begin trying
to apply their basic research toward designing aids for the blind, he says. 

The researchers' early systems had the look and feel of what they were
-experiments. The buzzers were noisy, heavy, and power hungry. Although
electrodes could stimulate nerves quietly and efficiently, high voltages and
currents were necessary to drive signals through the skin. That sometimes led
to uncomfortable shocks. 

Because of these drawbacks, Bach-y-Rita began thinking about the tongue. "We
brushed him off," recalls coworker Kurt A. Kaczmarek, an electrical engineer
and perception researcher, also at the University of Wisconsin. "He tends to
be a bit ahead of his day."  

In time, however, Kaczmarek was convinced. "One day, I said 'Okay, Paul.
Let's go up to the lab and try it.' It turns out, it worked quite well," he
says. 

Tongue stimulation, however, isn't the only way to circumvent blindness. One
competing approach, for example, is to implant microchips in the eyes or
brain (SN: 4/12/97, p. 221). Another scheme, devised by a Dutch scientist,
converts images to what he calls soundscapes, which are piped to a blind
person's ears. 

Tongue stimulation 

To Bach-y-Rita, his team's switch from skin to tongue stimulation was
crucial. "We now, for the first time, have the possibility of a really
practical [touch-based] human-machine interface," he declares. He and his
coworkers founded the Madison-based company Wicab, to exploit the potential.
Kaczmarek points out the fledgling company may be in for some competition,
since a German inventor already has been granted a U.S. patent for a
tongue-vision system. 

"Using the tongue for seeing is a whole new approach. . . . I think it has
great promise," says Michael D. Oberdorfer, program director for visual
neuroscience at the National Eye Institute in Bethesda, Md. His office has
been funding some of the Wisconsin group's work. 

The tongue is a better sensor than skin for several reasons, says
Bach-y-Rita. For one, it's coated in saliva-an electrically conductive fluid.
So, stimulation can be applied with much lower voltage and current than is
required for the skin. 

Also, the tongue is more densely populated with touch-sensitive nerves than
most other parts of the body. That opens up the possibility that the tongue
can convey higher-resolution data than the skin can. 

What's more, the tongue is ordinarily out of sight and out of the way. "With
visual aids to the blind, there are cosmetic issues," says Oberdorfer. "And
you'd want something easy to wear that doesn't interfere with everyday
activities."  

Currently, the Wisconsin researchers' tongue-display system begins with a
camera about the size of a deck of cards. Cables connect it with a
toaster-size control box. Extending from the box is another cable made of
flat, flexible plastic laced with copper wires. It narrows at the end to form
the flat, 12-by-12, gold-plated electrode array the size of a dessert fork.
The person lays it like a lollipop on his or her tongue. Stimulation from
electrodes produces sensations that subjects describe as tingling or bubbling. 

The Wisconsin researchers say that the whole apparatus could shrink
dramatically, becoming both hidden and easily portable. The camera would
vanish into an eyeglass frame. From there, it would wirelessly transmit
visual data to a dental retainer in the mouth that would house the
signal-translating electronics. The retainer would also hold the electrode
against the tongue. 

The tongue display still has a long way to go in terms of performance, the
researchers admit. In the July 13 Brain Research, Bach-y-Rita and his
colleagues Eliana Sampaio and Stphane Maris, both of the University =Louis
Pasteur in Strasbourg, France, report results from the first clinical study
of the tongue display. 

After an initial, brief training period, 12 first-time users - sighted but
blindfolded and 6 congenitally blind, including Marie-Laure Martin -tried to
determine the orientation of the E's of a standard Snellen eye chart. On
average, they scored 20/860 in visual acuity. The cutoff for legal blindness
is 20/200 with corrected vision.  

"It's not normal sight," comments Taub. "It's like very dim shadows. But it's
remarkable. It's a beginning."  

One obstacle to better vision with the device is the low resolution of its
144-electrode display. Engineers on the team say they expect to quadruple the
array density in the next few years. 

A more serious problem is the range of contrast that can be replicated on the
tongue, Kaczmarek notes. In a typical image, the eye may simultaneously see
lighted regions that are 1,000 times brighter than the dimmest ones. But the
ratio of strongest to weakest tongue stimulation can only be about 3 to 1.
"That's one of the things we're struggling with," Kaczmarek says. 

Visual sensations 

Exactly how the tongue supplies the brain with images remains a focus of the
Wisconsin team's research. In his 1993 book, The Man Who Tasted Shapes
(Putnam), Washington, D.C.f.based neurologist Richard E. Cytowic made much of
how flavors stimulating the tongue of a friend and, later, an experimental
subject, would elicit visual sensations. However, that type of involuntary
and poorly understood sensory blending, which is known as synesthesia,
probably goes beyond what's needed to explain the operation of the tongue
display, Bach-y-Rita says. 

Instead, there's plenty of evidence, he says, that even those brain regions
devoted almost exclusively to a certain sense actually receive a variety of
sensory signals. "We showed many years ago that even in the specialized eye
region, auditory and tactile signals also arrive," he notes. 

Also, many studies over the past 40 years indicate that the brain is capable
of massively reorganizing itself in response to loss or injury. When it comes
to seeing via the sense of touch, reorganization may involve switching
portions of the visual cortex to the processing of touch sensations,
Bach-y-Rita says. 

In that vein, the first clinical study of the tongue device showed that users
got better with practice. Of the dozen subjects in the initial evaluation,
two went on to receive an additional 9 hours each of training. When retested,
they had doubled their visual acuity, scoring an average of 20/430. 

The brain's apparent ability to shunt data for one sense through the
customary pathways of another may enable the Wisconsin researchers to apply
their device beyond vision replacement. "It's not just about vision," says
Mitchell E. Tyler, a biomedical engineer with the group. "That's the obvious
one, but it's by no means the only game in town."  

The team began tests this summer of a modified system that's intended to
assist people who have lost their sense of balance because of injury,
disease, or reactions to antibiotics. The unit gathers signals from
accelerometers mounted on a person that indicate when he or she is tilting
and in what direction. By stimulating the tongue with patterns representing
the degree and direction of tilt, such a device may act as an artificial
vestibular system. Then, the person might be able to correct bodily position
and avoid falling, Tyler explains. 

Although the main emphasis of the Wisconsin research has been rehabilitation,
the group also foresees using its technology to aid people who don't have
sensory deficits. 

Interest in enhancement of the senses has come primarily from the military.
While Bach-y-Rita and his colleagues were using external skin as a receiver
of light-derived images, the Defense Advanced Research Projects Agency in
Arlington, Va., funded them to develop a sonar-based system to help Navy
commandos orient themselves in pitch darkness. The prototype worked,
Bach-y-Rita says. 

Tyler proposes that ground soldiers could also receive data by means of
infrared cameras or other sensors that would alert them, through the tongue,
to the presence and positions of enemy troops or tanks. Civilian workers,
such as firefighters, might also benefit from such interfaces. 

That's pure speculation right now. Martin's bouts of vision; however, are
much more than that. In a new film that aired on Canadian television in June,
a smile spreads across Martin's face as she gets her first glimpse of a
candle flame. 

The film, Touch: The Forgotten Sense, highlights some of the Wisconsin work.
Its message is this: Touch works in a thousand ways, often without people
even being aware of its roles. 

By taking this sense into new arenas, such as the tongue display, Bach-y-Rita
and his coworkers intend to extend touch's repertoire even more. 


References

Sampaio, E., S. Maris., and P. Bach-y-Rita. 2001. Brain plasticity 'Visual'
acuity of blind persons via the tongue. Brain Research 908(July 13):204. 

Further Readings 

2001. Tongue seen as portal to the brain. University of Wisconsin-Madison
press release. March 26. Available at
http://www.engr.wisc.edu/news/headlines/2001/Mar26.html.

Bower, B. 1995. Brain changes linked to phantom-limb pain. Science News
147(June 10):357. 

_______. 1999. Ear implants resound in deaf cats' brains. Science News
156(Sept. 11):167. 

Seppa, N. 1998. Do blind people track sounds better? Science News 154(Sept.
19):180. 

_______. 2001. Gene therapy cures blindness in dogs. Science News 159(May
12):296. 

Travis, J. 2000. Snap, crackle, and feel good? Science News 158(Sept.
23):204. Available at http://sciencenews.org/20000923/bob2.asp. 

_______. 2000. Perfect pitch common among the blind. Science News 158(Nov.
25):344. 

Wu, C. 1997. Solar cells may sub for retinal receptors. Science News
151(April 12):221. 

For online information about the University of Wisconsin's tongue display,
see http://kaz.med.wisc.edu/.

To learn about transforming light images into "soundscapes," see
http://ourworld.compuserve.com/homepages/Peter_Meijer/. Be sure to check
out the Java demo.

For more information about the film Touch: The Forgotten Sense, directed by
Kun Chang, contact Max Films 518, Rue Sherbrooke Est. Montreal, QC H2L 1K1
Canada 

Sources:

Paul Bach-y-Rita
Department of Rehabilitation Medicine
University of Wisconsin-Madison E3/348
600 Highland Ave.
Madison, WI  53792-3256 

Richard E. Cytowic
4720 Blagden Terrace, NW
Washington, DC  20011-3720 

Kurt A. Kaczmarek
University of Wisconsin-Madison
Department of Rehabilitation Medicine
Department of Biomedical Engineering
1300 University Ave.
Madison, WI  53706 

Stephane Maris
Laboratorie d'Etudes des Systemes Perceptifs et Emotionnels
University Louis Pasteur
12 rue Goethe 67000
Strasbourg, France

Michael D. Oberdorfer
National Eye Institute
Executive Plaza South, Suite 350
6120 Executive Blvd. MSC 7164
Bethesda, MD  20892-7164

Eliano Sampaio
Laboratorie d'Etudes des Systemes Perceptifs et Emotionnels
University Louis Pasteur
12 rue Goethe 67000
Strasbourg, France

Edward Taub
Department of Psychology
University of Alabama at Birmingham CPM 712
1530 3rd Avenue S.
Birmingham, AL  35294-0018

Mitchell E. Tyler
Department of Biomedical Engineering
University of Wisconsin-Madison
1410 Engineering Dr., Room 268
Madison, WI  53706-1608

