From the archive, originally posted by: [ spectre ]
WE SEE WITH OUR BRAINS, NOT WITH OUR EYES
The Seeing Tongue
In-the-mouth electrodes give blind people a feel for vision
BY Peter Weiss / Sept. 1, 2001
Blind since birth, Marie-Laure Martin had always thought that candle
flames were big balls of fire. The 39-year-old woman couldn’t see the
flames themselves, but she could sense the candle’s aura of heat.
Last October, she saw a candle flame for the first time. She was
stunned by how small it actually was and how it danced. There’s a
second marvel here: She saw it all with her tongue.
The tongue, an organ of taste and touch, may seem like an unlikely
substitute for the eyes. After all, it’s usually hidden inside the
mouth, insensitive to light, and not connected to optic nerves.
However, a growing body of research indicates that the tongue may in
fact be the second-best place on the body for receiving visual
information from the world and transmitting it to the brain.
Researchers at the University of Wisconsin-Madison are developing this
tongue-stimulating system, which translates images detected by a
camera into a pattern of electric pulses that trigger touch receptors.
The scientists say that volunteers testing the prototype soon lose
awareness of on-the-tongue sensations. They then perceive the
stimulation as shapes and features in space. Their tongue becomes a
Earlier research had used the skin as a route for images to reach the
nervous system. That people can decode nerve pulses as visual
information when they come from sources other than the eyes shows how
adaptable, or plastic, the brain is, says Wisconsin neuroscientist and
physician Paul Bach-y-Rita, one of the device’s inventors.
“You don’t see with the eyes. You see with the brain,” he contends. An
image, once it reaches an eye’s retina, “becomes nerve pulses no
different from those from the big toe,” he says. To see, people rely
on the brain’s ability to interpret those signals correctly.
With that in mind, he and his colleagues propose that restoring sight
is only one of the many trajectories for their research. Restoring
stability to those with balance disorders is another. So is bestowing
people with brand new senses, such as the capability to use heat to
see in the dark.
Restoring lost vision
First things first, however, and for the Wisconsin scientists that
means restoring lost vision. Swapping the sense of touch for sight is
not a new idea. In the 1960s, Bach-y-Rita, his colleagues, and other
scientists began developing and testing devices that enable the skin
of blind people to pick up visual information.
For Bach-y-Rita, the experiments also provided insight into the
brain’s plasticity. His more general goal has been to find out how
well one sense can take the place of another.
Until the 1980s, “one of the axioms of neuroscience was that there was
no plasticity in the adult central nervous system,” says Edward Taub
of the University of Alabama in Birmingham. Today, the field has
turned around in response to many studies, including Bach-y-Rita’s.
Now, scientists view the brain as almost as malleable in old age as in
youth, he adds.
The idea of tongue as eye evolved from the earlier skin-as-eye
studies. Bach-y-Rita and his coworkers had been placing touch-
stimulating arrays on areas of people’s skin, such as the back and the
abdomen. The scientists used either electrodes or little buzzers to
excite nerve endings of the skin in a pattern that corresponded to
They found that after receiving training, blind people using these
systems could recognize shapes and track motion. Some subjects could
perceive the motion of a ball rolling down an inclined plane and bat
it as it rolled off the plane’s edge. Others could carry out an
assembly-line task at an electronics plant. It required them to
recognize glass tubes lacking solder and then to deposit some solder
into those tubes.
These results impressed Bach-y-Rita and his colleagues enough to begin
trying to apply their basic research toward designing aids for the
blind, he says.
The researchers’ early systems had the look and feel of what they were–
experiments. The buzzers were noisy, heavy, and power hungry. Although
electrodes could stimulate nerves quietly and efficiently, high
voltages and currents were necessary to drive signals through the
skin. That sometimes led to uncomfortable shocks.
Because of these drawbacks, Bach-y-Rita began thinking about the
tongue. “We brushed him off,” recalls coworker Kurt A. Kaczmarek, an
electrical engineer and perception researcher, also at the University
of Wisconsin. “He tends to be a bit ahead of his day.”
In time, however, Kaczmarek was convinced. “One day, I said ‘Okay,
Paul. Let’s go up to the lab and try it.’ It turns out, it worked
quite well,” he says.
Tongue stimulation, however, isn’t the only way to circumvent
blindness. One competing approach, for example, is to implant
microchips in the eyes or brain (SN: 4/12/97, p. 221). Another scheme,
devised by a Dutch scientist, converts images to what he calls
soundscapes, which are piped to a blind person’s ears.
To Bach-y-Rita, his team’s switch from skin to tongue stimulation was
crucial. “We now, for the first time, have the possibility of a really
practical [touch-based] human-machine interface,” he declares. He and
his coworkers founded the Madison-based company Wicab, to exploit the
potential. Kaczmarek points out the fledgling company may be in for
some competition, since a German inventor already has been granted a
U.S. patent for a tongue-vision system.
“Using the tongue for seeing is a whole new approach. . . . I think it
has great promise,” says Michael D. Oberdorfer, program director for
visual neuroscience at the National Eye Institute in Bethesda, Md. His
office has been funding some of the Wisconsin group’s work.
The tongue is a better sensor than skin for several reasons, says Bach-
y-Rita. For one, it’s coated in saliva–an electrically conductive
fluid. So, stimulation can be applied with much lower voltage and
current than is required for the skin.
Also, the tongue is more densely populated with touch-sensitive nerves
than most other parts of the body. That opens up the possibility that
the tongue can convey higher-resolution data than the skin can.
What’s more, the tongue is ordinarily out of sight and out of the way.
“With visual aids to the blind, there are cosmetic issues,” says
Oberdorfer. “And you’d want something easy to wear that doesn’t
interfere with everyday activities.”
Currently, the Wisconsin researchers’ tongue-display system begins
with a camera about the size of a deck of cards. Cables connect it
with a toaster-size control box. Extending from the box is another
cable made of flat, flexible plastic laced with copper wires. It
narrows at the end to form the flat, 12-by-12, gold-plated electrode
array the size of a dessert fork. The person lays it like a lollipop
on his or her tongue. Stimulation from electrodes produces sensations
that subjects describe as tingling or bubbling.
The Wisconsin researchers say that the whole apparatus could shrink
dramatically, becoming both hidden and easily portable. The camera
would vanish into an eyeglass frame. From there, it would wirelessly
transmit visual data to a dental retainer in the mouth that would
house the signal-translating electronics. The retainer would also hold
the electrode against the tongue.
The tongue display still has a long way to go in terms of performance,
the researchers admit. In the July 13 Brain Research, Bach-y-Rita and
his colleagues Eliana Sampaio and Stéphane Maris, both of the
Université Louis Pasteur in Strasbourg, France, report results from
the first clinical study of the tongue display.
After an initial, brief training period, 12 first-time users–6 sighted
but blindfolded and 6 congenitally blind, including Marie-Laure Martin–
tried to determine the orientation of the E’s of a standard Snellen
eye chart. On average, they scored 20/860 in visual acuity. The cutoff
for legal blindness is 20/200 with corrected vision.
“It’s not normal sight,” comments Taub. “It’s like very dim shadows.
But it’s remarkable. It’s a beginning.”
One obstacle to better vision with the device is the low resolution of
its 144-electrode display. Engineers on the team say they expect to
quadruple the array density in the next few years.
A more serious problem is the range of contrast that can be replicated
on the tongue, Kaczmarek notes. In a typical image, the eye may
simultaneously see lighted regions that are 1,000 times brighter than
the dimmest ones. But the ratio of strongest to weakest tongue
stimulation can only be about 3 to 1. “That’s one of the things we’re
struggling with,” Kaczmarek says.
Exactly how the tongue supplies the brain with images remains a focus
of the Wisconsin team’s research. In his 1993 book, The Man Who Tasted
Shapes (Putnam), Washington, D.C.-based neurologist Richard E. Cytowic
made much of how flavors stimulating the tongue of a friend and,
later, an experimental subject, would elicit visual sensations.
However, that type of involuntary and poorly understood sensory
blending, which is known as synesthesia, probably goes beyond what’s
needed to explain the operation of the tongue display, Bach-y-Rita
Instead, there’s plenty of evidence, he says, that even those brain
regions devoted almost exclusively to a certain sense actually receive
a variety of sensory signals. “We showed many years ago that even in
the specialized eye region, auditory and tactile signals also arrive,”
Also, many studies over the past 40 years indicate that the brain is
capable of massively reorganizing itself in response to loss or
injury. When it comes to seeing via the sense of touch, reorganization
may involve switching portions of the visual cortex to the processing
of touch sensations, Bach-y-Rita says.
In that vein, the first clinical study of the tongue device showed
that users got better with practice. Of the dozen subjects in the
initial evaluation, two went on to receive an additional 9 hours each
of training. When retested, they had doubled their visual acuity,
scoring an average of 20/430.
The brain’s apparent ability to shunt data for one sense through the
customary pathways of another may enable the Wisconsin researchers to
apply their device beyond vision replacement. “It’s not just about
vision,” says Mitchell E. Tyler, a biomedical engineer with the group.
“That’s the obvious one, but it’s by no means the only game in town.”
The team began tests this summer of a modified system that’s intended
to assist people who have lost their sense of balance because of
injury, disease, or reactions to antibiotics. The unit gathers signals
from accelerometers mounted on a person that indicate when he or she
is tilting and in what direction. By stimulating the tongue with
patterns representing the degree and direction of tilt, such a device
may act as an artificial vestibular system. Then, the person might be
able to correct bodily position and avoid falling, Tyler explains.
Although the main emphasis of the Wisconsin research has been
rehabilitation, the group also foresees using its technology to aid
people who don’t have sensory deficits.
Interest in enhancement of the senses has come primarily from the
military. While Bach-y-Rita and his colleagues were using external
skin as a receiver of light-derived images, the Defense Advanced
Research Projects Agency in Arlington, Va., funded them to develop a
sonar-based system to help Navy commandos orient themselves in pitch
darkness. The prototype worked, Bach-y-Rita says.
Tyler proposes that ground soldiers could also receive data by means
of infrared cameras or other sensors that would alert them, through
the tongue, to the presence and positions of enemy troops or tanks.
Civilian workers, such as firefighters, might also benefit from such
That’s pure speculation right now. Martin’s bouts of vision; however,
are much more than that. In a new film that aired on Canadian
television in June, a smile spreads across Martin’s face as she gets
her first glimpse of a candle flame.
The film, Touch: The Forgotten Sense, highlights some of the Wisconsin
work. Its message is this: Touch works in a thousand ways, often
without people even being aware of its roles.
By taking this sense into new arenas, such as the tongue display, Bach-
y-Rita and his coworkers intend to extend touch’s repertoire even
Dr. Paul Bach-y-Rita, 72, passed away peacefully at home in his sleep
on Monday, November 20, 2006. Paul was first introduced to science
when he studied at the Bronx High School of Science, from which he
graduated at age 15. He completed his college degree at Mexico City
College (now the University of the Americas in Puebla) and began
Medical School at the UNAM (Universidad Autonoma de Mexico) at age 17.
Paul’s adventurous spirit always played a large role in his life and
led him to the next step. Not only did Paul hitchhike to and from
Mexico City and New York while a student, he even decided to attend
medical school on a dare. He eventually dropped out of medical school
and embarked on a series of experiences that would shape his career in
the future. During his year away from medical school, Paul worked as a
salmon and shrimp fisherman, on a Boeing assembly line alongside many
disabled colleagues, and was trained as a masseur for Floridas tourist
industry. During training, he was then hired to teach anatomy and
physiology to blind veterans in the massage program. The impact of
these experiences convinced him to go back to medical school. He
learned so much about blind people while working with them that he
worked with them from that point on. After medical school, he was the
first and only doctor in Tilzaptla, Morelos, Mexico, a village that
had no roads or electricity, before taking a job at the Smith-
Kettlewell Institute of Visual Sciences in San Francisco. A full
professor at age 37, his career took another pivotal change in
direction after his father suffered a major stroke. He was inspired by
his fathers miraculous recovery after his brother George defied
conventional wisdom that would have condemned his father to a nursing
home, by creating a rehab program that led to his full recovery. His
recovery was so immense that he was able to return to his
professorship and died at 73 from a heart attack, while hiking at
9,000 feet in Columbia. This led Paul to trade his established and
promising career in the field of eye movements, and complete a
residency in rehabilitation medicine at the Stanford Santa Clara
Medical School to study people like his dad. He was then recruited to
UW-Madison as Chair of the Rehab program in 1983. Because of Paul’s
realization that his love for research surpassed his love for
administration, he decided it was time to step down and focus mainly
on his research. He was able to do this as a professor at UW Medical
school, Department of Orthopedics and Rehabilitation Medicine and the
UW-Madison Engineering School, Department of Biomedical Engineering,
while taking on many projects and collaborations around the world;
including at the Karolinska Institute in Stockholm, Universit Sorbonne
VI in Paris, and UAEM in Cuernavaca, Mexico. He realized that the
findings he had been working on since 1962 were being validated by
other researchers: the existence of non-synaptic diffusion
neurotransmission (NDN) as a complementary mechanism of information
transmission, that may play multiple roles in the brain, including in
normal and abnormal activity, brain plasticity and drug actions was
becoming accepted. One of his latest accomplishments, Paul founded
Wicab Corporation in 1998, named in honor of his beloved wife,
Esther’s family. The CEO of Wicab, Robert Beckman, wanted to share,
“One of Paul’s favorite expressions was ‘we see with our brain, not
with our eyes.’ And he was able to harness that concept so that
totally blind subjects have recently been seen navigating hallways and
even shooting baskets-seeing with their brain, with an assist from the
BrainPortAll Wicab employees salute Paul today for his visionary
leadership. We recognize Paul as the father of sensory substitution
and brain plasticity, now commonly accepted concepts, but novel ideas
when Paul first conceived of them in the 1960’s. Truly, Paul was one
of those rare individuals that see things as they might be and ask,
Can You See With Your Tongue?
The brain is so adaptable, researchers now think, any of the five
senses can be rewired
BY Michael Abrams / 06.01.2003
I’m sitting at a table draped in black, surrounded by black curtains.
Candles, spheres, and unfamiliar symbols have been placed before me.
My right hand, arms, and head are strapped with wires, and my mouth is
filled with electrodes. I’m blindfolded.
Although this may sound like a scene for a Black Mass, it’s even
stranger than that: I’m trying to see with my tongue.
The gear I’m wearing was invented by Paul Bach-y-Rita, a
neuroscientist at the University of Wisconsin at Madison. Bach-y-Rita
has devoted much of his career to a single, revolutionary concept:
that our senses are interchangeable. The brain, Bach-y-Rita and many
other neuroscientists believe, is an organ of astonishing plasticity:
If one part of it is damaged, another part can serve the same
function. To prove the point, his collaborator Kathi Kamm, a professor
of occupational therapy at the university’s Milwaukee campus, has
strapped a small video camera to my forehead and connected it to a
long plastic strip hanging from my mouth. A laptop computer reduces
the camera’s image to 144 pixels. Those pixels are converted to an
electric current that is sent to the business end of the plastic strip–
a 12-by-12 grid of electrodes that rests on my tongue.
Kamm sits down in front of me. She says she’s holding a ball, but I
can’t hear a sound as she rolls it back and forth over the cloth-
covered table. She says the ball will soon be rolling toward me–to my
left, my right, or straight at me–but my eyes and ears have no way to
tell where it’s going.
That leaves my tongue. It has more tactile nerve endings than any part
of the body other than the lips. What the camera sees is zapped onto
my tongue’s wet, conductive surface. As Kamm rolls the ball, my
blindfolded eyes see nothing, but a tingling passes over my tongue.
When she sends the ball my way, my hand leaps out to the left.
I’ve caught it.
“We don’t see with our eyes,” Bach-y-Rita is fond of saying. “we see
with our brains.” The ears, eyes, nose, tongue, and skin are just
inputs that provide information. When the brain processes this data,
we experience the five senses, but where the data come from may not be
so important. “Clearly, there are connections to certain parts of the
brain, but you can modify that,” Bach-y-Rita says. “You can do so much
more with a sensory organ than what Mother Nature does with it.”
Bach-y-Rita, who is 69, looks like a cross between Albert Einstein and
Harpo Marx. His hair springs from his head in a wild gray Afro, and
his face often bears a comic, knowing smirk. He owes his iconoclastic
spirit to his late father, he says. A professor of Spanish at the City
University of New York, with a passion for 16th-century Catalonian
poetry, Pedro Bach-y-Rita nearly destroyed his career in 1947 by
organizing the country’s first civil-rights strike at a university. He
encouraged his children to be equally rebellious. Rather than raise
Paul as a Catholic like himself or Jewish like his wife, for instance,
he urged him to choose his own religion. Paul chose to become a
Swedish Lutheran–he liked the pastor at Bernadotte Lutheran Church in
the Bronx. But when he later won a scholarship to a Lutheran college,
he turned it down. He didn’t feel right accepting the money, he
explained, since he was not a believer.
In 1958, at the age of 65, Pedro Bach-y-Rita suffered a stroke that
left him confined to a wheelchair, hardly able to move or speak.
Paul’s brother, George, was a medical student at the University of
Mexico at the time. Rather than let his father vegetate in a nursing
home, George brought him to his house and put him to work. “It was
tough love,” Paul says. “He’d throw something on the floor and say
‘Dad, go get it.'” The neighbors would watch in dismay as the old man
struggled to sweep the porch. “But for him, it was so rewarding,” Paul
says. “This useless man was doing something.”
Neurologists in those years believed that brain damage was impossible
to reverse. If a stroke caused memory loss, paralysis, or dementia for
more than a few weeks, the condition was permanent. Nevertheless,
after three years Paul’s father recovered completely. He went back to
teaching and worked for another five years. When he died in 1969 at
the age of 73, it was from a heart attack while hiking at an altitude
of 9,000 feet in the mountains of Colombia.
The neuropathologist who autopsied Pedro’s brain later published a
paper on the case in the American Journal of Physical Medicine,
complete with pictures of Pedro’s devastated brain. “It was shocking,”
Paul says. “My father had recovered so much that we’d figured he
didn’t have much brain damage.” Why did he recover, Paul remembers
thinking, when everyone else said he couldn’t?
Reteaching the Brain to Balance
The sense of balance may be the simplest of the senses and
therefore the easiest to redirect in the brain. It stems from tiny
hair cells in the inner ear that are surrounded by a layer of gel.
When you move your head, the gel is pushed against the hair cells,
which relay the information to the brain. The whole system is called
the vestibular sense.
Over the past 40 years or so, several thousand people in the
United States have lost this sense, due to an antibiotic called
gentamicin. One of the drug’s side effects is ototoxicity: It can kill
the hair cells in the inner ear. Cheryl Schiltz, seen in the
photograph on the opposite page, lives in Windsor, Wisconsin. In
November 1997, after taking gentamicin for 17 days, she woke up and
couldn’t stand.”I had to crawl,” she says. “It was like being
extremely intoxicated. I was scared to death.”
Schiltz also suffers from tinnitus, short-term memory loss, and
vision problems. “It’s a living hell,” she says. She eventually found
solace among other victims of gentamicin, who call themselves The
Wobblers, but real relief came only after her physician referred her
to Paul Bach-y-Rita.
Schiltz was dubious at first. “He explained it to me, and I’m
going, ‘the tongue?’ I thought he was kidding.” Nevertheless, she let
Bach-y-Rita outfit her with a hard hat and a strip of electrodes for
her tongue. The hat contained an accelerometer that registered
Schiltz’s movements and relayed the information to a circle on the
grid in her mouth. If she leaned forward, the circle moved forward
too. All Schiltz had to do, to stay balanced, was keep the circle
centered on her tongue.
The results were almost instantaneous. “All of a sudden, I started
crying,” Schiltz says. “I had forgotten what it was like to see
clearly, what it was like not to stagger. It was like the hand of God
coming down and touching me.” Within half an hour she was standing
without assistance. “I was shocked,” Bach-y-Rita says. “She learned it
almost immediately. I think the reason is that she already had
partially trained herself to understand tactile cues. She’s been using
the contact of her feet on the ground.”
Schiltz later took the experiment even further. After 20 minutes
spent centering the circle, she took off the hat, pulled out the
electrodes, and kept her balance for a full hour without any
apparatus. “I ran through the building in my socks,” she says. “I
danced with Paul and climbed up and down chairs and tables. I felt
cured, literally cured.”
Paul’s career changed course after Pedro’s death. He quit the job he
had taken after medical school, at the Smith-Kettlewell Institute of
Visual Sciences in San Francisco, and took a residency at Stanford’s
Santa Clara Valley Medical Center. “It was quite stupid or brave or
something to drop out and go into residency,” Bach-y-Rita says. But he
wanted to study people like his father–to re-create the miracle he had
After settling down as professor of rehabilitation medicine at the
University of Wisconsin, Bach-y-Rita turned his attention back to the
senses. He knew that victims of leprosy, for instance, can lose the
sense of touch in their limbs, so he developed a glove with
transducers on each fingertip that were connected to five points on
the forehead. When his test subjects touched something with the
gloves, they felt an equivalent pressure on their heads. Within
minutes they were able to sense the difference between rough and
smooth surfaces–and they quickly forgot that their foreheads were
doing the feeling.
If sight and touch can swap paths to consciousness, Bach-y-Rita
reasoned, so can sound. In the 1980s, his team plugged a microphone
into a vibrating belt. Low frequencies picked up by the mike tickled
the left side of the waist; high frequencies tickled the right. Deaf
people who donned the belt claimed it helped them read lips.
Impressive as they were, Bach-y-Rita’s experiments did not impress
mainstream neuroscientists. As early as 1969, he published a paper in
Nature on one of his devices, but his mentor, the Nobel Prize-winning
neurophysiologist Ragnar Granit, couldn’t understand what he was up
to. “He called me into his parlor and said ‘Paul, you know how I
appreciate your work on eye muscles. But why are you wasting your time
on this adult toy?'”
The skepticism was understandable. In those early years, and to a
lesser degree today, many neuroscientists believed that the brain is
compartmentalized–that visual information, for instance, goes straight
from the eye to the visual cortex through a fixed network of nerves.
If any part of the system is damaged, sight is impossible. Only the
eyes can see.
This notion dates back to 1861, when the pioneering French neurologist
Paul Broca found lesions in the frontal lobe of a speechless man.
Broca concluded that certain parts of the brain are responsible for
certain tasks, and a deluge of later research seemed to prove him
right. Most recently, functional MRI and PET scans have shown that
different areas of the brain light up depending on whether a person is
identifying colors, recognizing faces, registering emotions, or
learning a language.
Bach-y-Rita says that’s only part of the story: “In any given field
there’s a conceptual substance–I love that phrase–a general
understanding that’s not easily changed.” In trying to understand the
brain, for instance, neuroscientists have focused on synapses–the
junctions between nerve-cell endings–as the essential transmitters of
thought and feeling. Children both grow and prune back synaptic
connections at a furious rate as they develop, but the process all but
stops in adulthood. Many researchers still believe, therefore, that a
damaged brain causes permanent deficits.
“The synapse is a concept in evolution; it’s what’s observable under a
microscope,” Bach-y-Rita says. “There are other things going on
between cells.” Only 10 percent of the cells in the brain are neurons,
he says. They make up the brain’s hard wiring and send messages with
electrical pulses. The rest are glial cells whose precise function is
not well understood. Neurons release neurotransmitters that are taken
up by specific receptors, but many glial cells receive and emit
neurotransmitters that float through the brain as free agents. Some
glial cells congregate near lesions, for instance, and in areas of the
brain where learning is going on. “It’s so much less cumbersome to
have changes in this system than it is in the whole wiring system,”
Bach-y-Rita says. Much of the human intellect, he believes, may come
from these nonelectrical, free-floating signals. How else can our
brains achieve so much mind power without using any more energy, pound
for pound, than the brains of other animals?
Whether or not Bach-y-Rita is right about glial cells, more and more
evidence suggests that the senses can be redirected. At Harvard
University in the late 1990s, for instance, neurologist Alvaro Pascual-
Leone performed brain scans of blind subjects. When he asked them to
read braille with their reading fingers, their visual cortex lit up.
When sighted people performed the same task, their visual cortex
stayed dormant. More recently, neuroscientist Mriganka Sur at MIT took
young ferrets and connected fibers coming from their retinas to their
auditory pathway. They grew up with perfect vision.
Thanks to such studies, the term “plasticity,” once taboo in
neuroscience papers, has become fashionable. “At any meeting you see
loads and loads of papers on plasticity,” Bach-y-Rita says. Still,
even some of his allies think he claims too much. Sight is a rich and
complicated phenomenon, they say, and the eye such an astonishing
organ, that it can never be replaced. Michael Merzenich, a
neuroscientist at the University of California at San Francisco, has
been a leading proponent of brain plasticity for two decades. Bach-y-
Rita’s tongue device demonstrates “a powerful substitution,” he says,
but he doubts that it could provide anything like actual sight. “If
it’s not stimulating the retina, it’s unlikely, to my mind, that it’s
“I totally disagree,” Bach-y-Rita says. “There’s nothing special about
the optic nerve. The brain doesn’t care where the information comes
from. Do you need visual input to see? Hell, no. If you respond to
light and you perceive, then it’s sight.”
Bach-y-Rita sounds convincing, but in the lab I’m still left wondering
what exactly I’m experiencing. The images have a sour, battery taste
and feel like the pelting of a hot summer cloudburst. They certainly
convey some sense of where things are around me, but is that the same
In practical terms, the answer may be irrelevant. When Kamm places a
small white cube somewhere on the table, I can reach out and grab it
nine times out of 10, even though I’m blindfolded. I can even
recognize large letters, as long as I can bob my head around to get a
better sense of their outlines. Given a few more hours with the
device, I might eventually learn to forget the tingling in my mouth
and just see. Is that sight?
The question might best be asked of one of Kamm’s subjects, a 16-year-
old named Beth with a gift for music. Beth is the top singer in her
high school choir and hopes to study music in college and become a
composer. She has also been blind since birth. Until she met Bach-y-
Rita, she never knew how a conductor gestures to keep time, but by
wearing the electrodes, she learned the gestures in half an hour. If
she eventually learns to “see” these movements across a room, and to
understand their meaning, is it useful to call this anything other
Perhaps it is, in which case Bach-y-Rita’s research is teaching us
something even more interesting–that sight is not just a detailed
understanding of the light and space around us; it’s a particular,
even arbitrary, feeling.
To Bach-y-Rita and his clients, though, the difference isn’t all that
important. The Navy SEALs are working with him on a system that will
allow them to see infrared through their tongues and to find their way
through murky waters, leaving their eyes free for other tasks. NASA
has worked with him to develop sensors to enable astronauts to feel
things on the outside of their space suits. And the Institute for
Human and Machine Cognition in Pensacola, Florida, is using his ideas
to build vests that will tickle pilots to alert them to other planes
or incoming missiles.
Last October Bach-y-Rita received the Coulter Award from the American
Congress of Rehabilitation Medicine in recognition of his
contributions to the field of neurorehabilitation. After decades of
struggling at the fringes of his discipline, he now has the financial
backing to bring his work into the mainstream. Within the next couple
of years, he hopes to create a miniature version of his tongue-vision
system that will fit into a wireless retainer. A tiny camera in a pair
of glasses will send the image via radio waves into the mouth. If the
device works, not only will the blind see better, but the rest of us
may have access to senses we’ve never even dreamed of. “Anything that
can be measured can be transported to the brain,” Bach-y-Rita says.
“We can get it to the brain, and the brain can learn how to use it.”
New Tools to Help Patients Reclaim Damaged Senses
BY Sandra Blakeslee / November 23, 2004
Cheryl Schiltz vividly recalls the morning she became a wobbler. Seven
years ago, recovering from an infection after surgery with the aid of
a common antibiotic, she climbed out of bed feeling pretty good.
“Then I literally fell to the floor,” she said recently. “The whole
world started wobbling. When I turned my head, the room tilted. My
vision blurred. Even the air felt heavy.”
The antibiotic, Ms. Schiltz learned, had damaged her vestibular
system, the part of the brain that provides visual and gravitational
stability. She was forced to quit her job and stay home, clinging to
the walls to keep from toppling over.
But three years ago, Ms. Schiltz volunteered for an experimental
treatment – a fat strip of tape, placed on her tongue, with an array
of 144 microelectrodes about the size of a postage stamp. The strip
was wired to a kind of carpenter’s level, which was mounted on a hard
hat that she placed on her head. The level determined her spatial
coordinates and sent the information as tiny pulses to her tongue.
The apparatus, called a BrainPort, worked beautifully. By “buzzing”
her tongue once a day for 20 minutes, keeping the pulses centered, she
regained normal vestibular function and was able to balance.
Ms. Schiltz and other patients like her are the beneficiaries of an
astonishing new technology that allows one set of sensory information
to substitute for another in the brain.
Using novel electronic aids, vision can be represented on the skin,
tongue or through the ears. If the sense of touch is gone from one
part of the body, it can be routed to an area where touch sensations
are intact. Pilots confused by foggy conditions, in which the horizon
disappears, can right their aircraft by monitoring sensations on the
tongue or trunk. Surgeons can feel on their tongues the tip of a probe
inside a patient’s body, enabling precise movements.
Sensory substitution is not new. Touch substitutes for vision when
people read Braille. By tapping a cane, a blind person perceives a
step, a curb or a puddle of water but is not aware of any sensation in
the hand; feeling is experienced at the tip of the cane.
But the technology for swapping sensory information is largely the
effort of Dr. Paul Bach-y-Rita, a neuroscientist in the University of
Wisconsin Medical School’s orthopedics and rehabilitation department.
More than 30 years ago, Dr. Bach-y-Rita developed the first sensory
substitution device, routing visual images, via a head-mounted camera,
to electrodes taped to the skin on people’s backs. The subjects, he
found, could “see” large objects and flickering candles with their
backs. The tongue, sensitive and easy to reach, turned out to be an
even better place to deliver substitute senses, Dr. Bach-y-Rita said.
Until recently sensory substitution was confined to the laboratory.
But electronic miniaturization and more powerful computer algorithms
are making the technology less cumbersome. Next month, the first fully
portable device will be tested in Dr. Bach-y-Rita’s lab.
The BrainPort is nearing commercialization. Two years ago, the
University of Wisconsin patented the concept and exclusively licensed
it to Wicab Inc., a company formed by Dr. Bach-y-Rita to develop and
market BrainPort devices. Robert Beckman, the company president, said
units should be available a year from now.
Meanwhile, a handful of clinicians around the world who are using the
BrainPort on an experimental basis are effusive about its promise.
“I have never seen any other device do what this one does,” said Dr.
F. Owen Black, an expert on vestibular disorders at the Legacy
Clinical Research and Technology Center in Portland, Ore. “Our
patients are begging us to continue using the device.”
Dr. Maurice Ptito, a neuroscientist at University of Montreal School
of Optometry, is conducting brain imaging experiments to explore how
Dr. Eliana Sampaio, a neuroscientist at the National Conservatory of
Arts and Métiers in Paris, is using the BrainPort to study brain
plasticity. Sensory substitution is based on the idea that all sensory
information entering the brain consists of patterns carried by nerve
In vision, images of the world pass through the retina and are
converted into impulses that travel up the optic nerve into the brain.
In hearing, sounds pass through the ear and are converted into
patterns carried by the auditory nerve into the brain. In touch, nerve
endings on skin translate touch sensations into patterns carried into
These patterns travel to special sensory regions where they are
interpreted, with the help of memory, into seeing, hearing and touch.
Patterns are also seamlessly combined so that one can see, hear and
feel things simultaneously.
“We see with the brain, not with the eyes,” Dr. Bach-y-Rita said. “You
can lose your retina but you do not lose the ability to see as long as
your brain is intact.”
Most important, the brain does not seem to care if patterns come from
the eye, ear or skin. Given the proper context, it will interpret and
understand them. “For me, it happened automatically, within a few
minutes,” said Erik Weihenmayer, who has been blind since he was 13.
Mr. Weihenmayer, a 35-year-old adventurer who climbed to the summit of
Mount Everest two years ago, recently tried another version of the
BrainPort, a hard hat carrying a small video camera. Visual
information from the camera was translated into pulses that reached
He found doorways, caught balls rolling toward him and with his small
daughter played a game of rock, paper and scissors for the first time
in more than 20 years. Mr. Weihenmayer said that, with practice, the
substituted sense gets better, “as if the brain were rewiring itself.”
Ms. Schiltz, too, whose vestibular system was damaged by gentamicin,
an inexpensive generic antibiotic used for Gram-negative infections,
said that the first few times she used the BrainPort she felt tiny
impulses on her tongue but still could not maintain her balance. But
one day, after a full 20-minute session with the BrainPort, Ms.
Schiltz opened her eyes and felt that something was different. She
tilted her head back. The room did not move. “I went running out the
door,” she recalled. “I danced in the parking lot. I was completely
normal. For a whole hour.” Then, she said, the problem returned.
She tried more sessions. Soon her balance was restored for three
hours, then half a day. Now working with the BrainPort team at the
University of Wisconsin, Ms. Schiltz wears the tongue unit each
morning. Her balance problems are gone as long as she keeps to the
How the device produces a lasting effect is being investigated. The
vestibular system instructs the brain about changes in head movement
with respect to the pull of gravity. Dr. Bach-y-Rita speculated that
in some patients, a tiny amount of vestibular tissue might survive and
be reactivated by the BrainPort.
Dr. Black said he had seen the same residual effect in his own pilot
study. “It decays in hours to days,” he said, “but is very
Blind people who have used the device do not report lasting effects.
But they are amazed by what they can see. Mr. Weihenmayer said the
device at first felt like candy pop rocks on his tongue. But that
sensation quickly gave way to perceptions of size, movement and
Mr. Weihenmayer said that on several occasions he was able to find his
wife, who was standing still in an outdoor park, but he admitted that
he also once confused her with a tree. Another time, he walked down a
sidewalk and almost went off a bridge.
Nevertheless, he is enthusiastic about the future of the device. Mr.
Weihenmayer likes to paraglide, and he sees the BrainPort as a way to
deliver sonar information to his tongue about how far he is from the
Dr. Ptito is scanning the brains of congenitally blind people who,
wearing the BrainPort, have learned to make out the shapes, learned
from Braille, of capital letters like T, B or E. The first few times
they wore the device, he said, their visual areas remained dark and
inactive – not surprising since they had been blind since birth. But
after training, he said, their visual areas lighted up when they used
the tongue device. The study has been accepted for publication in the
Dr. Ptito says he would like to see if he could teach his subjects how
to read drifting letters like those in advertising displays. Not
seeing motion is a big problem for the blind, he said.
In another approach, Dr. Peter Meijer, a Dutch scientist working
independently, has developed a system for blind people to see with
their ears. A small device converts signals from a video camera into
sound patterns delivered by stereo headset to the ears. Changes in
frequency connote up or down. Changes in pixel brightness are sensed
as louder or softer sounds.
Dr. Yuri Danilov, a neuroscientist and engineer who works with Dr.
Bach-y-Rita, said the research team had thought of dozens of
applications for the BrainPort, which he called a “USB port to the
In one experiment, a leprosy patient who had lost the ability to
experience touch with his fingers was outfitted with a glove
containing contact sensors. These were coupled to skin on his
forehead. Soon he experienced the data coming from the glove on his
forehead, as if the feelings originated in his fingertips. He said he
cried when he could touch and feel his wife’s face.
The federal government has also shown interest in sensory substitution
technology. The Navy is exploring the use of a tongue device to help
divers find their way in dark waters at night, said Dr. Anil Raj, a
research scientist at the Institute for Human and Machine Cognition at
the University of West Florida in Pensacola.
The sensors detect water surges, informing Navy Seals if they are
following the correct course. The Army is thinking about sending
infrared signals from night goggles directly to the tongue, Dr. Raj
In another application, student pilots have been fitted with body
stimulators attached to aircraft instruments. When the airplane starts
to pitch or change altitude, they can feel the movements on their
Sensory substitution technology may eventually help millions of people
overcome their sensory disabilities. But the devices may also have
more frivolous uses: in video games, for example.
Dr. Raj said the tongue unit had already been tried out in a game that
involved shooting villains. “In two minutes you stop feeling the buzz
on your tongue and get a visual representation of the bad guy,” he
said. “You feel like you have X-ray vision. Unfortunately it makes the
Wicab, Inc. has developed the BrainPort(tm) technology to transmit
external sensory information to the brain through a substitute sensory
channel: electrotactile stimulation of the tongue. The use of the
tongue as a sensory substitution channel has been previously
established (see technical papers section of web site). For the brain
to correctly interpret information from a sensory substitution device,
it is not necessary for the information to be presented in the same
form as the natural sensory system. For example, we do not see with
the eyes; the optical image does not go beyond the retina where it is
turned into spatio-temporal patterns of action potentials along the
optic nerve fibers. The brain then recreates the images from analysis
of the impulse patterns. Thus, for a sensory substitution event to
occur, one need only to accurately entrain action potentials in an
alternate information channel, which do not differ significantly for
the individual senses. With training, the brain learns to
appropriately interpret that information and utilize it to function as
it would with data from the intact natural sense.
In the proposed balance applications, the BrainPort device substitutes
for the lack of vestibular input by transmitting information about
head position, as sensed by an accelerometer, to an electrode array
placed on the tongue. Prior research with a prototype device has shown
that the BrainPort device permits people with severe or profound
Bilateral Vestibular Loss (BVL) to maintain near-normal static posture
control under a variety of conditions, including quiet sitting or
standing and during activities of daily living, such as walking in a
crowd or on soft uneven surfaces.*
The BrainPort balance device is essentially a mechanism to supply
sensory information to the subject through electrical impulses
directed to the tongue. The tongue is a uniquely qualified target for
electrical impulses because of 1) the density and sensitivity of nerve
fibers at this site and 2) the suitability of the tongue to receive
and maintain electrical contacts in terms of chemical environment,
which minimizes the electrical energy requirement and point of contact
Toll Free: 888.449.4222
email: info [at] wicab [dot] com
For commercial collaboration inquiries please send an email to:
ceo [at] wicab [dot] com or phone Mr. Beckman at 608.829.4502.
Wicab Balance Inquiries
For balance related academic and clinical research inquiries please
send an email to: clinicaltrials [at] wicab [dot] com
Wicab Vision Inquiries
For vision related academic and clinical research inquiries please
send an email to: vision [at] wicab [dot] com