A new AI model developed by the Weizmann Institute of Science, called Brain-IT, can ‘read’ your brain activity and reconstruct what you see with striking speed and accuracy.

The stadium is full, the ball is in the air – and all eyes are fixed on the batter poised to swing his baseball bat. It is a scene rich in detail and symbolic meaning, instantly recognisable to fans.

A ‘mind-reading’ AI model called Brain-IT, developed in the Weizmann lab of Professor Michal Irani can reconstruct this scene based solely on the brain activity of a person viewing it.  Brain-IT has learned to identify brain activity patterns that recur across brain regions of different people as they process images, allowing it to learn to ‘read’ a new person with unprecedented speed and extraordinary accuracy.

In recent years, scientists have succeeded in restoring the ability to speak to people with severe paralysis, using personalised AI models trained on tens of hours of recordings of their brain activity. These technologies are life-changing, but they are still far from reading minds, since they are tailored to specific individuals and require extensive training. Brain-IT, by contrast, recognises patterns of brain activity that are shared by all of us, and translates them into images.

“There exist nowadays models that translate brain activity into images, and they can even produce impressive reconstructions that preserve the semantic meaning of the image reasonably well,” Irani said.

“However, they tend to make mistakes in basic features such as composition and colour. The new model we developed outperforms them in reconstructing both the content of the image and its details. What’s more, while every other model requires dozens of hours of brain scans to learn to ‘read’ a new person, our model needs only one hour.”

The road to developing Brain-IT – which was selected this year to be presented at the International Conference on Learning Representations (ICLR), one of the world’s leading AI conferences – was challenging. Irani’s team, in Weizmann’s Computer Science and Applied Mathematics Department – Roman Beliy, Amit Zalcher, Jonathan Kogman and Navve Wasserman – had to contend with an acute shortage of data.

Training such an AI model to achieve optimal performance takes a vast number of images, paired with functional magnetic resonance imaging (fMRI) scans of brain activity recorded while participants view those images. But to create such datasets, researchers must place large numbers of participants inside claustrophobic MRI machines for extended periods. Accordingly, the largest dataset currently available to researchers in the field contains fMRI scans from no more than eight individuals, each of whom viewed several thousand images – not a large amount when it comes to training AI algorithms.

The solution Irani and her team devised for this scarcity of examples resembles a two-way bilingual dictionary. Instead of developing only a ‘decoder’ that learns to reconstruct an image from an fMRI scan, they also developed an “encoder” that learns to predict an fMRI scan from an image.

“We realised that by translating back and forth – from a random image that had never been viewed in an fMRI machine, to a predicted brain scan, and then back to the image we started with – the models would effectively build themselves a massive dataset,” Irani explained.

“The plan was to feed the encoder images that had never been shown to a person in an MRI scanner, generate estimated brain scans for them, and pass those scans to the decoder, which would try to reconstruct the original image from them. In this way, during training, the models would learn to generate scans that encode images effectively, even though those fMRI scans had never actually been performed.”

The main challenge in developing the encoder was the variation between individual brains: The same image can evoke a different pattern of activity in different people. The solution was to break the problem down into its components. Instead of trying to encode an image into an entire brain scan, the researchers divided the brain into roughly 40,000 ‘brain voxels’ – tiny three-dimensional units of volume – and measured how each responded to different images. At the same time, they used an AI model to extract basic visual features, such as colour and location, from the images, alongside semantic features that reflect the image’s meaning – for example, a food item or a face. At this level of detail, patterns of brain activity that recurred across different brains and were associated with specific image features began to emerge – a discovery that made it possible to build a universal brain encoder.

The road to mind-reading
The brain encoder turned out to be far more than a technical step on the way to developing the decoder.

“During training, the encoder naturally identified 128 functional regions that are shared by all people and perform specific roles in image processing,” Irani explained.

“Some of them are familiar to neuroscientists, but others are entirely new. For example, we discovered a division of roles within the brain region that processes images of places – the PPA – with one part responding to indoor scenes and another to outdoor scenes,” she said.

“The encoder’s strength lies in its ability to identify regions that perform similar functions in different brains, even when their anatomical locations differ,” she added.

“Another advantage is its ability to test different features in a controlled way. For example, you can feed it the same image in colour and in black and white, identify the differences between the predicted fMRI scans, and from this infer where colour is encoded in the brain. In the future, the encoder could also help us understand how different regions work together and how perception takes shape.”

Irani’s lab is now working to extend these ‘mind-reading’ methods to the decoding of auditory information as well. Video could be another frontier.

“What remains especially challenging is decoding video – for example, during dreaming,” she explained.

“Dozens of images change every second, while an fMRI scan takes about two seconds. If we overcome all these obstacles, it’s possible that in the future we may even be able to read dreams.”