Integrating Movement Into Tech For Visual Impairment

On a hill overlooking a small airport whose runway stretches into the Ligurian Sea sits the Unit for Visually Impaired People, part of the Italian Institute for Technology (IIT) in Genoa. Here, a team of psychologists and engineers works to create assistive devices for people with visual impairments.

They use a bevy of tools to do this, from relatively simple buzzers and sensors to a more complex moving platform whose seat was taken from a race car. This is a (surprisingly quick) rotating chair on a track that is used to assess the body’s vestibular system (which is responsible for the sense of balance and spatial orientation).

A particularly striking tool is a metal case that unsnaps to reveal a variety of white rubber hands. Research volunteers who see it compare it to the TV series Wednesday, laughs psychologist Carolina Tammurello, referring to the latest Addams Family adaptation, which features the sentient hand known as Thing.

Tammurello explains that this collection of 3D-printed hands allows her to run a version of the rubber hand illusion. This is a common (though not perfect) psychology experiment where a participant’s hand is stroked in sync with a rubber hand. A participant watching the fake hand will often start thinking of it as their own. Tammurello is testing whether visually impaired children and adults also experience this illusion.

The rubber hand illusion tests the interaction of several senses: touch, sight, and proprioception (the body’s sense of movement and position). Indeed, the interaction of different senses is one of the key concerns of the Unit for Visually Impaired People.

At a cognitive level, the benefits of this multisensory integration add up to more than the sum of its parts, says Monica Gori, who leads IIT’s Unit for Visually Impaired People. When a young child touches a new object, they’re likely using their other senses to comprehend it. And people with and without sight integrate touch, hearing, and vision in different ways. Gori explains, “if you don’t have vision, you don’t develop certain multisensory skills” – especially those linked to spatial and body representations.

Gori’s research group can see that perceptions of symmetry depend on children’s visual abilities, and that blind infants reach for, and interact less with, objects outside their own bodies.

Timing is critical here. “This is really set in the first period of life,” explains Gori. Even in the first few months, sighted children have a sense of where their bodies are in space, unlike congenitally blind children. And the first three years are a critical “developmental window” before changes in the brain’s visual cortex start to set in. According to Gori, interventions have to happen before the age of six, when some of these cognitive changes become more embedded.

The differences in multisensory integration affect when a child starts to grasp objects, crawl, walk, and run. Blind children reach these milestones much later than sighted children, which has social and psychological repercussions (like sighted children of the same age not wanting to play with them). The domino effects of the early delays can be enormous. “Senses are connected to everything, also to our psychological state,” Gori notes.

Yet Gori says that there’s relatively little research on this essential first part of a visually impaired person’s life. Her group is working to understand exactly when the behavioral changes are occurring.

The ultimate aim is to develop technologies that might help to address weak links among the senses, following consultation with visually impaired people. Gori believes that her lab is unique as the only one in the world able to investigate the brains of very young visually impaired children, with the aim of generating assistive tech devices for early multisensory intervention. What makes this possible, she thinks, is the diverse expertise of her team, the supportive institutional structure where she works, and the network of partner organizations (including hospitals) that is built on her many years of working in this field.

This network is needed because, for one thing, there’s a limited group of visually impaired children that the researchers can work with to understand their needs and test any resulting devices. They might have to travel hundreds of kilometers, spanning multiple provinces, to test a single infant.

And all that travel might not result in any data. Tammurello estimates that about half the time, children refuse to keep on the electrode cap that is recording their brain activity. Especially from the age of about 1½ onward, Tammurello says, some kids distrust fussing around their heads. And of course the researchers can’t force the children to participate.

This line of research has resulted in some prototypes. The Gori lab previously developed an Audio Bracelet for Blind Interactions (ABBI), which makes a variety of sounds – including an elephant’s roar – as the wearer moves around in space.

One of the people who recalls testing ABBI years ago at the Chiossone Institute, a nonprofit organization in Genoa that supports visually impaired people, is Michela, who’s now 20. She explains of ABBI, “the one that wears it moves the arm and can be reached by the person who is searching for it. I thought it was very useful for not shouting the other person but being able to reach [them] thanks to sound, it was like seeing [them].”

“It’s a very simple idea, but it works,” Gori comments. Her team and low-vision and blind children have developed dozens of games to make ABBI fun to use, which will make kids more likely to play with it. Hide and seek has proven popular among the child testers.

“ABBI seemed to me a very useful invention,” says Lara, now 19. “I discovered it late but for very young children I think it can be developed to teach, for example, how to stand up without falling, to walk without seeing. At that age I wore a helmet, so I guess a function that teaches the child balance would be nice.”

More broadly, Lara feels, “For blind or visually impaired children it would be important to have games that make more noise. Even for example tennis [balls] or hockey [pucks].” She remembers wanting to play tennis with friends as a child. Similarly, Gori has seen the benefits of devices and activities that encourage visually impaired children to move, as in dance or basketball, rather than only stationary activities like piano lessons.

According to Gori, using ABBI for one hour per day, for one year, would restore the visual cortex of a blind child to the level of a sighted child. And for greater accessibility, including in low-income countries, she and her colleagues are seeking investment to adapt ABBI as a free-to-use smartphone app.

The researchers also run trials with adults, whose previous experiences might override their receptivity to familiar interventions. For a new impact with adults, new solutions are needed, according to Gori.

For Lara, in general, more work needs to be done. “The aids that are there now are fine, but I think everything needs to be strengthened and improved. In fact, more and more progress is needed so that everything can be accessible.” She’s especially interested in more accessible cars, which she points out already have onboard computers that ideally could work with various types of inputs. “Maybe one day we’ll even drive,” Lara muses.

Michela would like that as well. Even further, “I would like that all the technology could be accessible, because I feel excluded, and I suffer it so much. There are many apps not accessible, such as seeing images that are not read or partially described at least.”

Having experienced barriers like this has made Michela interested in creating her own assistive technologies. This will involve more resources devoted to making education inclusive. As Michela says, “I want to learn programming by myself, because I want to make things accessible. But I am not very patient.”

This story was reported during a journalism-in-residence fellowship at the Italian Institute for Technology (IIT), funded by the European Research Council (ERC).