Brain implant lets paralyzed people speak and gesture through a digital avatar

What it's about

Walter has ALS (amyotrophic lateral sclerosis). In this progressive neurological disease, the nerve cells that control voluntary muscles gradually die off. People with ALS lose the ability to speak, swallow and move, and eventually to breathe. Thinking and understanding usually stay intact. For Walter, it took about six months for slightly slurred speech to become completely unintelligible. After that, he was left with email, text-to-speech apps and telephone relay services.

"You try to keep communication going somehow, and you boil it down to the bare essentials. There's simply no room left for emotions and gestures."

During an ALS Association webinar, Walter heard about a study run by neurosurgeon Edward Chang at the University of California, San Francisco (UCSF). He contacted the research team himself, even though taking part meant brain surgery. He also saw it as a chance to push forward technology that could help future patients, even if he never got to use a finished version himself. Popular Mechanics wrote up his story. The study itself appeared in the journal Nature Neuroscience.

What's new: a brain-computer interface (BCI, a system that translates brain signals directly into commands for a computer) uses a single implant to read out both what someone wants to say and the gesture they want to make while saying it. A personal virtual avatar then performs both. Walter calls it his "digital twin."

How the system works

The system is built on electrocorticography (ECoG). An array of electrodes sits directly on the surface of the brain and measures the electrical activity underneath. The implant covered a large part of the sensorimotor cortex. This region of the cerebral cortex controls movement, including the movements of the lips, tongue and larynx during speech. Different areas take the lead for different parts of the body. That's exactly what lets machine learning models (programs that learn patterns from lots of examples) tell whether someone is trying to speak or to gesture.

The team started working with Walter in early summer 2025. First, the system learned to recognize attempted speech and attempted movements separately. Only then were the two combined. For Walter, the breakthrough came when the avatar got both right at the same time: "What a huge milestone!"

The surprise: speech and gesture interact in the brain

Along the way, the team ran into an unexpected problem. Samantha Brosler, a UCSF researcher and one of the study's first authors, explains:

"We were surprised that models trained separately on speech and gestures performed worse when participants attempted both at the same time."

The culprit is overlapping signals. Some brain regions are active during both speaking and gesturing, and their activity patterns change when the two happen together. Jennifer Collinger, a neuroprosthetics researcher at the University of Pittsburgh who wasn't involved in the study, calls this context-dependent neural activity: the brain activity behind a wave can look different depending on whether you're talking at the same time.

So the UCSF team trained two decoders in parallel, meaning two models that translate brain signals into an output: one for speech and one for gestures. The training data came from recordings in which participants attempted speech and gestures first on their own and then together. If someone says "Hello" while waving, that produces data for both models: one learns the sentence, the other learns the movement. This joint training improved recognition. The models could also recognize some combinations that never appeared in the training data. That matters, because there's no way to teach a system every possible pairing of words and movements.

The numbers and the study's limits

The 100 combinations included some that would rarely come up in a real conversation. That was deliberate: the researchers wanted to see how reliably the system could recognize both at once. For participants paralyzed by a brainstem stroke, the team also ran offline tests, meaning analyses of previously recorded data rather than live operation. According to the report, these produced usable results too.

Collinger finds the results encouraging but stresses the limits. The system doesn't hold open-ended conversations. It only recognizes predefined sentences, shows the decoded speech as text, and triggers the matching avatar animation when it detects a gesture.

One scene from the lab shows what errors can mean. When Walter was asked whether he wanted coffee, his avatar waved goodbye instead of nodding or giving a thumbs-up. In that case it was funny. In other situations, though, the wrong gesture could change the meaning of an entire conversation. That's why Collinger considers user feedback essential. Brosler can imagine future versions letting users confirm or correct decoded outputs before they go out, for example with eye tracking, which lets you control a device with your gaze.

Still a long way from everyday life

For now, the implant is wired to external computers. To work in everyday life, Brosler says, it would need wireless transmission, a wearable device and lower latency. Getting there means solving several technical problems: components need to shrink, brain signals have to be recorded reliably over the long term, and the device has to use less power and give off less heat.

Walter describes the work as "a learning curve on both sides, for the avatar and for the patient." His ideal would be a system that reads his signals reliably. Going wireless and less invasive would make it far more appealing to patients. In one test dialogue, the system decoded his speech and gestures without a single error:

"The avatar's response to my 'thoughts' moved me deeply. I couldn't stop laughing with pure joy."

Our take

Chang's group at UCSF is one of the leading teams working on speech neuroprostheses. Until now, the field has mostly focused on pulling words out of brain signals as quickly and accurately as possible. What's new here is the focus on everything beyond words: gestures, facial expressions, body language. The most interesting scientific finding is probably that speech and gesture are more tightly linked in the brain than expected. Decoders that learn the two separately aren't enough.

Still, don't overstate the results. The study had just three participants and only 20 preset expressions, and roughly one in three outputs was wrong in real-time testing. Add the brain surgery and the cable connection, and this is still a long way from a product for patients. Collinger sums up what really matters:

"Restoring the ability to communicate isn't just measured in words per minute."

In her view, independence, relationships, taking part in community life and being able to express your own personality matter just as much. One open question is how much control users will get over what their digital twin says and does on their behalf.