Pop’Balloons: The First Serious Mixed-Reality Game for Autistic Children
Summary: Researchers have developed a new mixed reality game that can help children on the autism spectrum develop better motor skills.
Source: University of Montreal
The demand for the assessment and intervention of mental health is increasing; available services are stretched to the limit, centralized and difficult for many to access. This is especially true for children with autism spectrum disorder (ASD).
“Serious games” that can help meet the urgent need are now starting to appear. One of the first is Pop’Balloons, a game designed to help autistic children develop their motor skills and based on years of fundamental research.
The project was led by Guillaume Dumas, a professor of computational psychiatry in Université de Montréal’s Faculty of Medicine and director of the Precision Psychiatry and Social Physiology Laboratory at the CHU Sainte-Justine Research Center. Dumas also holds the IVADO Chair in AI and Mental Health and is an Associate Academic Member of Mila, the Quebec Artificial Intelligence Institute.
It is possible to record a person’s brain activity by placing them in a scanner or attaching electrodes to their scalp and performing an encephalogram, but these technologies are designed to examine only one person at a time , and therefore had disadvantages.
Typically, to measure brain activity during social interaction, researchers will expose people to recorded or simulated social stimuli. So, paradoxically, neuroscientists who studied social interactions did so by observing the brains of isolated individuals in laboratories.
Fortunately, the eventual development of “hyperscanning” made it possible to record the activity of multiple brains simultaneously and thus study real social interactions.
People’s brains synchronize
In 2010, while using hyperscanning to study spontaneous imitation in his native France, Dumas discovered that people’s brains synchronize with each other during social interaction – proving that people can literally be on the same wavelength.
Four years later, Dumas left France to do a post-doctorate in the US, where he continued his research. “We developed a system to record a human interacting with a virtual avatar,” Dumas recalled. “The avatar’s behavior was not strictly scripted; it reacted to the human in real time, in milliseconds.”
This virtual partner passed the Turing test, meaning it was indistinguishable from a human. Dumas called the cognitive-neuroscience-based human-machine interaction system he created the Human Dynamic Clamp. The avatar was able to adapt to human behavior and also to prompt new ones through its interactions with users. Initially, it was tested on neurotypical people.
Studies have suggested that sensory-motor differences in individuals diagnosed with ASD may be the cause of their social interaction problems.
The human-machine interface developed by Dumas supported the extraction of motor coordination and interpersonal synchronization indices, so he exploited these capabilities to create a new machine-assisted method of neuropsychological evaluation of ASD, which has since been clinically tested on more than 155 subjects and peer-validated.
As the machine demonstrated its ability to learn new behaviors, Dumas’ team also worked on a version that could help people with autism develop their motor skills.
“Your video games suck!”
Dumas tested this version on autistic children at the Hôpital universitaire Robert-Debré, in Paris. “Kids are blunt and they told me straight up, ‘Your video game sucks!’ It is true that it was a research protocol and not visually appealing.”
He worked with students at the École Centrale de Paris and developed a prototype of a more engaging video game. The company Act’image then collaborated with the Pasteur Institute to develop a 3D version of the game. A partnership with Microsoft’s HoloLens glasses has produced a mixed-reality version of the game. Unlike virtual reality, which takes users into an imaginary immersive experience, mixed reality superimposes holograms on the real world.
The result was Pop’Balloons, a game that allows young people with ASD to safely explore their real environment. Holograms of balloons are virtually placed in a room and the goal of the game is to burst them. The faster the balloons pop, the higher the score. Children with autism can explore the room at their own pace and repeat the experience as many times as necessary.
An open access version was then developed with financial support from the Orange Foundation, which enabled Dumas to work on the game in Montreal when he was appointed professor at UdeM.
Towards precision medicine
A collaboration with Canadian video game studio Eidos helped improve the game. And the connection with Mila, the Quebec Artificial Intelligence Institute, made it possible to develop more advanced machine learning techniques. IVADO’s dedicated collaborative research team has facilitated links between UdeM and the video game industry, which has the expertise to develop more engaging serious games.
See also
“From the video game, we were able to go as far as clinical stratification — dividing study participants into subgroups — for precision medicine trials in mental health,” Dumas said. “This is what we do at my Precision Psychiatry and Social Physiology Laboratory: try to understand physiology and the basis of sociality in order to create psychiatric approaches tailored to the individual.”
A paper reporting the preliminary results of clinical stratification based on video game data has just been accepted by NeurIPS, an international conference on machine learning and computational neurology, being held this year in New Orleans.
A game accessible to everyone
Can a game be created that is accessible to everyone, including neurodiverse players? Can it be given to all autistic children without some ripping off their headphones? “The first step is to validate the accessibility parameters,” says Romain Trachel, a machine learning specialist at Eidos-Montréal, one of the few video game studios with an accessibility department. “We are talking about a population that often has more developed perception than other people.”
Mariem Hafsia, a master’s student at UdeM, added: “The first step is to make sure the game is comfortable for children with autism. Will they be comfortable with the device? Will they struggle to grasp the concept of the game? Will they find it engaging? Maybe playing with a balloon doesn’t mean the same thing to children with atypical development.”
The hope is that this first game can serve as a model for the development of more inclusive video games. Pop’Balloons may also be used in the future to conduct neuropsychological assessments in a non-medical setting.
“The game will not necessarily be used to make diagnoses that require a psychologist or doctor,” Dumas said. “It can support the health care system, which is very overloaded and has long waiting times for evaluations. The game can save time for healthcare professionals; they wouldn’t have to spend hours performing a clinical evaluation.”
Who knows? One day, the game may also be used in personalized telemedicine.
“It could have a dashboard where a clinician can track the patient and see their performance history,” Dumas suggested. “The patient does not have to come back to the clinic every time. The game itself can guide them to the most appropriate levels of play to help them improve. It would be an adaptive video game that could serve as a psycho-educational intervention.”
About this ASD research news
Writer: Press office
Source: University of Montreal
Contact: Press Office – University of Montreal
Picture: The image is credited to the University of Montreal
