Scientists at the Weizmann Institute in Rehovot, Israel, have developed an artificial intelligence tool that can recreate what a person sees with astonishing accuracy simply by analyzing images of the human brain. The technology can also work in the opposite direction: predicting brain activity based on what people see. The scientists say their findings could help with research on the human brain and could also be useful for people suffering from locked-in syndrome, or help recreate the content of dreams.
Image source: Unsplash, Sasun Bughdaryan
In the figure above, the left side of each pair is what people actually see, and the right side is what the artificial intelligence algorithm generated based on the analysis of the subjects’ brain images. The work of Israeli scientists has received positive reviews. Judy Illes, a neuroethicist and professor of neuroscience at the University of British Columbia in Canada, said of the work of her Israeli colleagues:magnificent”. “The idea of using this approach to treat patients with neurological diseases is very exciting.”” said Iles.
However, other scientists believe that similar methods could be used to reveal people’s thoughts and mental images, even without their consent. “The results look very impressive. But what if there was a way to quietly extract the information you’re thinking about?<...> That 150 years of science fiction could become reality at any time is shocking in many ways. “” said Tommy Sprague, a neuroscientist at the University of California, Santa Barbara.
Neuroscientists have been looking for years to use functional magnetic resonance imaging (fMRI) to recreate what people see and what’s going on in their minds. The result of the first such attempt was a blurry image that made it difficult to make out anything. Over the years, the situation has improved as tools for fMRI scanning and result interpretation have been optimized.
The Israeli scientists began their work by analyzing publicly available functional magnetic resonance imaging brain data. Other researchers have collected images of volunteers who were shown hundreds of images inside fMRI machines. The scientists also analyzed data sets obtained from high-resolution scanners. The data showed participants’ brain activity while viewing different images.
Scientists have previously worked on developing technology that could recreate what a person sees based on brain scans. Various tools were used for this purpose, but none of them gave similar results. To achieve this goal, scientists must start by training artificial intelligence models. To collect training data, eight volunteers each viewed approximately 9,000 images in a high-resolution functional magnetic resonance imaging scanner.
Image source: technologyreview.com
The main feature of the decoder created by the scientists is that it works in two directions simultaneously: it predicts the structure of the image (such as the position of the colors) as well as the content of the image (such as a bunch of bananas on a plate). This approach allows AI models to produce clearer images of what humans see. However, to improve a system’s accuracy, scientists need far more data than is available. To overcome this problem, they trained another artificial intelligence model – an encoder that can predict brain activity based on images. Thereafter, the encoder and decoder are used together, improving both tools in parallel.
Here’s how it works. First, take a new image, such as a leopard. The encoder generates predictions of functional magnetic resonance imaging (fMRI) images of the brain of a person viewing the images. Next, the resulting image is reconstructed using the decoder. Initially, the reconstructed image bears little resemblance to the original image. However, repeatedly training a model in this way can significantly improve quality over time.
This approach also allows scientists to train the model on as many images as needed, including those not shown to a person lying in an fMRI scanner. Sources revealed that approximately 70% of the training data were images that did not initially match the fMRI images.
By combining data from multiple studies, scientists have also been able to identify areas of the brain that may function the same in all people. For example, one area responds to images of food and another to images of sports themes. The scientists do not rule out that the technology they create will allow “Learn something new about the brain”.
As for the encoder, with minimal calibration it can handle newcomers’ pictures. However, it typically takes about 40 hours of fMRI data per subject to train the model before the algorithm can predict what a person is seeing. For the decoder, one hour of data is enough for training. The scientists presented the results of their work last month at the Cognitive Computational Neuroscience conference in New York.
In the future, the researchers plan to move beyond images to video and audio. Ultimately, they hope to teach artificial intelligence to reconstruct a person’s thoughts or imagination, as well as the content of dreams. However, the possibility of creating such technology will inevitably raise questions about the privacy of human thoughts, since we are talking about using artificial intelligence to reconstruct thoughts or memories without human consent.
If you find an error, select it with your mouse and press CTRL+ENTER.
