Shape of Story?

My friend Brian Clark (who honors civility in discourse) posted this video and seems to think well of the people behind this effort...

Sorry for the incivility Brian (I'm afraid there's more coming), but this video and the blogpost that accompanied it really pissed me off.

Yes. I realize that the folks at Second Story are proud of the tools they've developed for focus groups - and they probably think that their efforts are helpful, or at worst harmless, but I strongly disagree.


For starters, the Portland Second Story team actually - without irony - describe their work as "Capturing Emotions." 

I understand the impulse - but I'm taking this opportunity to discourage these people from hunting for or capturing emotions.  They are not qualified and might inadvertently hurt people who are.

So tonight I'm taking the time to criticize Second Story, their methods and their goals - even at the risk of offending Brian Clark (who I really admire), not because I want to hurt anyone's feelings, but because insults, even when inadvertent, sometimes must be answered.

First, a bit of background... Years ago, I worked at a research company that had its own specially-outfitted theater in Hollywood where movies and TV shows were tested with similar trappings of cutting-edge science.

(Forgive me fellow filmmakers of the late 70s. I was new to LA and hungry. I needed a job. And the account I worked on was NBC under Freddie Silverman - not arthouse films.)

My work at ASI bothers me to this day. And now a group called Second Story in Portland seem to be volunteering to take filmmakers down the same destructive path I was able to get off many years ago...


Am I the only person who thinks the Portland methodology and their ideas about storytelling, filmmaking and audiences are.... profoundly misguided?

Back in the day, our plush theater in Hollywood had a big knob attached by a wire to every seat. Members of a recruited test audience could twist their knob from "very good" to "very boring."

Like tapping an iPad - our knob technology was said to be unobtrusive.  And the results had the gloss of technology.


Seems to me that requiring ANY conscious action while watching a film destroys the fundamental relationship the filmmaker wants with the audience: You're about to experience a wondrous moment of innocence and beauty - so don't forget to tap your iPad...

And that's not all...

If this video accurately represents what they're doing up in Portland, they've replicated yet another flaw in our 1970s methodology...

Specifically? The moderated discussion.

Having a moderator (who in our 1970s tests also introduced the process to the audience in a tightly scripted way) inserts an expert (one with subtle power and unavoidable biases) into the conversation.

Is this what they're doing in Portland?

Capturing artificially stimulated data points with an intentionally crude device - with the hypothesis that somehow the simple act of tapping might actually capture spontaneous human responses (to something as complex and ephemeral as the individual's moment-to-moment response to a movie?) - and that those data points, when transformed into a diamondy graphical representations, might stimulate useful conversation - under the supervision of a human moderator....


As long as everyone understands that the Portland data (like the data we captured back in the 70s) isn't measuring a normal response to a movie.

Remember, instead of watching a movie, you're asking the test audience (see - already they're one adjective removed from a normal audience) to do stuff while they watch the movie.

But what's the harm?

Here's the harm:

The only way data you collect from such a test can be evaluated is by comparison to data from other similar tests (not the real experience of seeing a movie). Otherwise there is no standard.  You need a comparison to know what the taps (or knob turns) mean.

Which (in my real life experience) leads to ridiculous equivalencies and comparisons - e.g., we know how audiences respond to a really engaging sequence in a Mr. Magoo cartoon, so it's obvious that Days of Heaven isn't measuring up here...

I'm not making this up.  We really used a Mr. Magoo cartoon to establish a baseline for every film we tested...


We didn't, as I recall, actually test Days of Heaven. But if we had been asked to, we would have.

And every male actor was compared in our research to a baseline gold standard - the Fonz.

I know.  The Portland guys are better than that.  But how much better?

To sum up...

In my opinion:

1) tapping a screen - or engaging in any behavior with a measuring device -  in a crowd of fellow tappers, knowing you're going to be quizzed on your taps, is not going to correlate to anything that a filmmaker with something to say in her film would find worth measuring, and

2) a filmmaker who listens to a moderated discussion built on top of self-conscious behavior deserves the insights she gets, and

3) even if meaningful responses to a film could be accurately recorded and quantified and delivered in this way... I'd still want to know why.

The goal of this process might seem valid to Second Story...

But "capturing emotions" is what filmmakers do - and the people at Second Story have simply failed to demonstrate that they have the tools or the sensitivity...

No matter how good their moderator's script - he's inserting himself into a relationship where he has not earned the privilege.

Am I alone in thinking that Second Story is putting their tap-collecting process where it doesn't belong?

When it works, the theater is a place of transcendence. And putting a moderator in a brown suit near the front - or a device in my hands I'm supposed to keep tapping- ain't gonna help me transcend.

You only have to read what Second Story wrote about themselves and their methods to see how inept they are with the human stuff.

I know they mean well. But they're really not helping...

UPDATE: April 26th, 2014 Apparently I'm not the only producer who thinks the social science behind testing motion pictures is flawed... Here's a link to a May 9th, 2012 post by Curb Your Enthusiasm producer Gavin "I certainly don’t think the information gained by a test is worth the $25,000 that it costs" Polone.

No comments:

Randy Finch's Film Blog:

Thoughts from a film producer about making and distributing films.