Scientists Take a Big Step Toward Movies Based Directly on a Filmmaker's Visual Imagination

‎A Sept. 22, 2011 post in Gizmodo discusses the implications of a recent experiment by UC Berkeley scientists, who have developed a method for capturing visual activity inside the human brain (how the visual cortex reacts to a movie - or any visual stimulation - even a dream) and - and here's the "Oh My God They Can Do That?" part - these scientists can reconstruct that brain activity into digital video clips.

Here's how this amazing experiment worked:

Three volunteers were placed inside a Magnetic Resonance Imaging system where their brain activity (blood flow through their brains' visual cortex) was monitored and recorded as they watched Hollywood movie trailers. By connecting the shape and motion information from these test movies to specific brain actions, the UC Berkeley researchers gathered very specific information about how the visual activity presented on the screen corresponded to each subject's brain activity.

Next another large group of videos (millions of frames from random YouTube videos) were shown to the same three subjects. The MRI machines again monitored brain activity - but this time the researchers compared the brain activity from the movie trailers to the activity in response to the YouTube videos - looking for similarity. In other words, the researchers used software they had developed to pick the frames from the YouTube videos that caused a brain activity most similar to the activity from the trailers that the subjects had previously watched.

After all this viewing (many hours inside an MRI machine) the experimental software picked the hundred YouTube movies that caused the most similar brain response to the original stimulation - and then blended those 100 videos into a single new film for each subject.

In the video at the top of this post you can compare the film the subjects had actually seen with a new video created (out of pieces of random YouTube videos) based solely on brain activity.

What's truly mind-blowing is that new films were produced by each subject's visual cortex - and these films, while crude, suggest that scientists now have a technique for taking brain activity and making it into motion pictures. In the near future, an MRI of a subject's brain might be used to create a motion picture based only on a dream or an artist's conscious efforts at visualization.

“This is a major leap toward reconstructing internal imagery,” said Professor Jack Gallant, a UC Berkeley neuroscientist and coauthor of the study published in the journal Current Biology. “We are opening a window into the movies in our minds.”

One aspect of this recent scientific breakthrough (the ability to express inexpressible ideas) may have been what T. S. Eliot was thinking about when he wrote the following 100 years ago: "[A]s if a magic lantern threw the nerves in patterns on a screen." The Love Song of J. Alfred Prufrock is a modernist masterpiece of regret and anxiety - completed in 1911, but first published in 1915 - where the title character struggles with the difficulty of expressing himself.

As a tool for people who are physically unable to communicate (e.g., those who cannot control the movement of their muscles because of injury or illness), the UC Berkeley technique holds great promise. But a cynic might ask: If given to would-be filmmakers, will the UC Berkeley breakthrough lead to great art - or will the power to bypass craft and simply project images from within give everyone the ability to share their ordinariness and mediocrity?

Thanks to novelist Kevin Holohan for making the connection to Eliot's poem.