Thought-to-speech: Scientists say their AI can read out what happens in your brain

Neuro-engineers at New York’s Columbia University say they have created a system that can translate human thoughts into recognizable speech, which would revolutionize not just medicine but communication.

By monitoring brain activity of subjects, the researchers at Columbia’s Mortimer B. Zuckerman Mind Brain Behavior Institute were able to train artificial intelligence to translate thought patterns into intelligible sentences, says a paper published on Tuesday in the journal Scientific Reports. The authors see patients with speech impaired by disease or trauma as first adopters of the incipient technology.

“We've shown that, with the right technology, these people's thoughts could be decoded and understood by any listener,” said Dr. Nima Mesgarani, the paper’s senior author.

After the researchers’ early attempts to translate brain activity into recognizable speech failed, they turned to a computer algorithm that can generate speech, called a vocoder. The algorithm improves the more it is “trained” by recordings of human speech.

“This is the same technology used by Amazon Echo and Apple Siri to give verbal responses to our questions,” said Dr. Mesgarani, who is also a professor at Columbia's Fu Foundation School of Engineering and Applied Science.

The vocoder was trained to interpret brain activity with the help of Dr. Ashesh Dinesh Mehta, a neurosurgeon at Northwell Health Neuroscience Institute in Long Island, and the paper’s co-author.

“Working with Dr. Mehta, we asked epilepsy patients already undergoing brain surgery to listen to sentences spoken by different people, while we measured patterns of brain activity,” said Dr. Mesgarani. “These neural patterns trained the vocoder.”

After this training was done, the next phase started. The patients listened to a person reading numbers from 0 to 9 while the algorithm scanned the brain activity and tried to translate it into sound. The result was a robotic voice reading numbers, which human listeners could understand and repeat with about 75-percent accuracy.

This may seem quite modest, but Dr. Mesgarani said such a result is “above and beyond any previous attempts.” The researchers plan to improve the system further, so that it could take as input brain patterns of a person thinking about speech, not actually listening it.

“This would be a game changer. It would give anyone who has lost their ability to speak, whether through injury or disease, the renewed chance to connect to the world around them,” Dr. Mesgarani said.

The technology will also need to work with more complex words and sentences to become more practical. The ultimate goal for the team is to create an implant that would synthesize speech directly from thought.

Think your friends would be interested? Share this story!