A non-invasive AI system, a world-first, can turn silent thoughts into text while only requiring users to wear a properly fitted cap.
The Australian researchers who developed the technology, called DeWave, tested the process using data from more than two dozen subjects.
Participants read silently while wearing a cap that recorded their brain waves via electroencephalogram (EEG) and decoded them into text.
With further improvements, DeWave could help stroke and paralysis patients communicate and make it easier for people to direct machines such as bionic arms or robots.
“This research represents a pioneering effort in translating raw EEG waves directly into language, marking a significant advance in this field,” says computer scientist Chin-Teng Lin of the University of Technology Sydney (UTS).
Although DeWave only achieved a little over 40% accuracy based on one of the two sets of measurements in the experiments conducted by Lin and colleagues, this is a 3% improvement. compared to the previous standard for thought translation from EEG recordings.
The researchers’ goal is to improve accuracy to around 90 percent, which is comparable to conventional language translation methods or speech recognition software.
Other methods of translating brain signals into language require invasive surgeries to implant large and expensive electrodes or MRI machines, making them impractical for everyday use – and they often have to use eye tracking to convert brain signals into word-level chunks.
When a person’s eyes move from one word to another, it is reasonable to assume that their brain takes a short break between processing each word. Translating raw EEG waves into words – without eye tracking to indicate the corresponding target word – is more difficult.
Different people’s brainwaves don’t represent all breaks between words in the same way, making it difficult to teach AI how to interpret individual thoughts.
After extensive training, DeWave’s encoder transforms EEG waves into a code that can then be associated with specific words based on their proximity to entries in DeWave’s “codebook.”
“It is the first to incorporate discrete coding techniques into the brain-to-text translation process, thereby introducing an innovative approach to neural decoding,” says Lin.
“Integrating large language models also opens new frontiers in neuroscience and AI.”
Lin and his team used trained language models that included a combination of a system called BERT with GPT, and tested it on existing datasets of people with Eye tracking and brain activity recorded while reading text.
This helped the system learn to match brainwave patterns with words, and then DeWave was trained further with a large open source language model that essentially creates sentences from the words.
frameborder=”0″allow=”accelerometer; autoplay; write to clipboard; encrypted media; gyroscope; picture in picture; web sharing” allow full screen>
DeWave performed best in translating verbs. Names, on the other hand, tended to be translated as pairs of words meaning the same thing rather than exact translations, such as “the man” instead of “the author.”
“We think this is because when the brain processes these words, semantically similar words can produce similar brain wave patterns,” says first author Yiqun Duan, a computer scientist at UTS.
“Despite the challenges, our model produces meaningful results, aligning keywords and forming similar sentence structures.”
The relatively large sample size tested takes into account that the distributions of EEG waves vary widely, suggesting that the research is more reliable than previous technologies that have only been tested on very small samples.
There is still work to be done and the signal is rather noisy when EEG signals are received through a cap rather than electrodes implanted in the brain.
“Translating thoughts directly from the brain is a valuable but challenging endeavor that warrants significant ongoing effort,” the team writes.
“Given the rapid advancement of large language models, similar coding methods linking brain activity to natural language deserve increased attention.”
The research was presented at the NeurIPS 2023 conference and a preprint is available on ArXiv.
Gn En tech