A woman who had suffered a brainstem stroke and was severely paralyzed was able to communicate via a digital avatar thanks to research from UC San Francisco and UC Berkeley that resulted in the development of a brain-computer interface (BCI).
Speech and facial expressions have never before been artificially created from brain signals. The device can also translate these impulses into text at a rate of around 80 words per minute, which is a significant advance above currently available commercial technology.
In the near future, Edward Chang, MD, chair of neurological surgery at UCSF, hopes that this most recent research breakthrough—known as a brain computer interface, or BCI—will lead to an FDA-approved system that enables speech from brain signals. It was published on August 23, 2023, in Nature.
“Our goal is to restore a full, embodied way of communicating, which is really the most natural way for us to talk with others,” said Chang, a member of the UCSF Weill Institute for Neuroscience and the Jeanne Robertson Distinguished Professor in Psychiatry.
We are a lot closer to offering patients a practical option thanks to these developments.
In a man who had similarly gone through a brainstem stroke many years earlier, Chang’s team had previously shown that it was feasible to translate brain signals into writing. The latest study illustrates a more ambitious goal: translating brain signals into the richness of speech as well as the facial expressions that people use to animate themselves during conversation.
On the woman’s brain’s surface, Chang implanted a paper-thin rectangle of 253 electrodes over regions his research has found to be crucial for speaking. The electrodes blocked the brain signals that, in the absence of the stroke, would have reached her face, tongue, jaw, and laryngeal muscles. The electrodes were connected to a bank of computers by a cable that was hooked into a connector fastened to her skull.
The participant and the researchers spent weeks educating the artificial intelligence algorithms in the system to recognize her particular brain signals for speech. Repeating various phrases from a repertoire of 1,024 words repeatedly was necessary to train the computer to identify the brain activity patterns connected to the sounds.
The researchers developed a technique that decodes words from phonemes rather than teaching the AI to recognize complete words. These are the components of speech that combine to produce spoken words, just as letters combine to form written words. Four phonemes make up the word “hello,” for instance: “HH,” “AH,” “L,” and “OW.”
This method only required the computer to learn 39 phonemes in order to decode any English word. This made the system three times faster and improved its accuracy.
The text decoder was created by graduate students Sean Metzger and Alex Silva in the combined Bioengineering Program at UC Berkeley and UCSF. “The accuracy, speed, and vocabulary are crucial,” said Silva. It’s what enables a user to eventually speak almost as quickly as we do and to conduct discussions that are far more lifelike and everyday.
The researchers used a clip of her speaking at her wedding to customize an algorithm for synthesizing speech so that it sounds as her voice did before the injury.
The team used software created by Speech Graphics, a producer of AI-driven facial animation, to animate the avatar by simulating and animating facial muscle movements.
The researchers developed specialized machine-learning algorithms that enabled the company’s software to interact with signals coming from the woman’s brain as she attempted to speak and translate them into facial movements on the avatar’s face, including opening and closing of the jaw, pursed lips, tongue movements, and expressions of happiness, sadness, and surprise.
Gopala Anumanchipalli, PhD, a professor of electrical engineering and computer sciences at UC Berkeley, and graduate student Kaylo Littlejohn are collaborating on the project. “We’re making up for the connections between the brain and vocal tract that have been severed by the stroke,” Littlejohn said.
The development of a wireless version that does not require the user to be physically attached to the BCI is a crucial next step for the team.
According to co-first author David Moses, PhD, an adjunct professor of neurological surgery, “giving people the ability to freely control their own computers and phones with this technology would have profound effects on their independence and social interactions.”
