By The Numbers


Part One: My skeptical response to the New York Times story on using data analysis to guide Hollywood screenwriters.  Part Two: Imagining Art In The Age Of Mechanical Production.

(Above: A Paint By Numbers Van Gogh)

* * *

The New York Times ran a story on Monday about data analysis shaping script development in Hollywood.  The gist of it is that studios are looking for their own Nate Silvers to forecast the likely fate of the writing in front of them.  With the right numbers on their side, the story goes, producers will be able to corral writers into making hits, despite whatever pesky aspirations towards individuality or artistry they may have.

The Times piece is deeply silly.  But it is poking around terrain of real interest and import.  Let me first address why it is silly.

1. The article is little more than a soft profile of Vinny Bruzzese and his Worldwide Motion Picture Group.  It purports to describe a trend and yet never really leaves Bruzzese’s office or his argument (the issue as he fashions it).  If data crunchers are really descending on Hollywood – and for all I know they may be – why are other firms or private individuals in the game not interviewed?

The other quotations in the article come from one skeptical, relatively unknown screenwriter, a producer of The Lincoln Lawyer, an unnamed executive who worked on Abraham Lincoln: Vampire Hunter in an unspecified capacity, and an unnamed Oscar-winning writer (for what?  The article does not say).

2. The article reveals basically nothing of how this data crunching works.  Here’s the supposed meat:

For as much as $20,000 per script, Mr. Bruzzese and a team of analysts compare the story structure and genre of a draft script with those of released movies, looking for clues to box-office success. His company, Worldwide Motion Picture Group, also digs into an extensive database of focus group results for similar films and surveys 1,500 potential moviegoers.”

How do they go about comparing story structure?  Does it differ from standard studio analysis (already quite formulaic)?  So they’ve got a database of focus group responses.  Do you know who also has that?  The studios.  Even if Bruzzese’s database is significantly deeper than the studios, what would that tell us about their data mining?

If the stat-heads have radically redesigned focus testing, or if they’ve pioneered a totally new approach to combing through that data and applying it to new productions, that might be a story.  But no such specifics are included in this article, and so there’s not much to talk about.  I mean, you tell me if you see anything new in the excerpted paragraph.

About those quoted producers, from earlier: they refuse to specify what advice Bruzzese gave that was helpful (Lincoln Lawyer) or which parts of his advice were ignored to the film’s detriment (Abraham Lincoln: Vampire Hunter).  So what are we even doing here?

Granted, it may be in Bruzzese’s interest to protect his company’s methods, and certainly the studios hope to maintain the illusion that they are not creating their films by algorithm, but it should be Brooks Barnes’ (the journalist’s) job to have something to say here worthy of publication.

3. In a formula describing a movie’s box-office performance, isolating the variables is a near-impossible task.

Bruzzese gives several examples of lessons the data have suggested: targeting demons make more money than summoned demons, protector super-heroes make more than cursed super-heroes, and bowling alleys are a box-office drag.

But how does he know that “summoned demons” are the problem?  What if the movies are just bad?  How does he even distinguish between “cursed” and “protector” super-heroes?  Aren’t Batman and Spider-man both?  And what could bowling alleys possibly have to do with anything?  The mathematical value of a bowling alley scene – supposing such a thing to exist (the value, that is) – would surely change as it is re-contextualized into other scripts, other films.

The two most celebrated popularizers of modern data analysis in previously shamanistic cultures are Billy Beane and Nate Silver.

Nate Silver, writing for his FiveThirtyEight blog, became the go-to forecaster for the presidential election, and ended up with the most successful prediction of the major public prognosticators.  A comparison of his work to what Bruzzese is attempting to do may be revealing.

Silver, working in the field of presidential elections, basically has to predict which way the electoral vote will fall.  That comes down to predicting the likely outcome of the popular vote in 50 states.  The popular vote itself comes down, for all practical purposes, to two options (Republican vs. Democratic nominee).  How did Silver guess correctly in every case?

He followed the polls.  The primary engine of Silver’s system is averaging polls.  The polls are weighted, yes, which involves some judgment and math, but effectively he took the data from major polling groups (which is available to everyone), combined it, and followed precisely what it said.  He said this himself many times on the blog.  People should not have been surprised, he indicated, by either Obama’s victory or his final electoral count, because in the vast majority of cases the polls had indicated such a result for months.

This kind of analysis is vastly simpler than predicting the success of a Hollywood movie.  Silver already knows the likely outcome in all but a dozen states.  He already knows the percentages of party affiliation in the remaining states.  He has numbers of the likely voters.  And again, there’s only really two options.  Hollywood doesn’t have good numbers on any of that.

A likely voter and a target audience are two wholly different categories.  The prospective audience for a movie can only be estimated by its rating, its connection to similar movies (especially if it is a sequel), and guesswork based on elements like star-power, subject matter, and artistic quality.  And even if you were to accurately gauge the potential audience for a giant summer movie like John Carter, how would you get from there to a reasonable prediction of how many of those people would actually go out and see it?

And remember, at the scripting stage the question isn’t even simply “would x percent of A (where A stands for the total prospective audience) attend this movie?”  The question is actually, “assuming all other values remain constant, how would changing element Y in this script affect the percentage of A likely to attend the finished film?”  Yeah, I doubt Nate Silver would want to touch that.

The Billy Beane comparison is just as problematic for Bruzzese.  Bean’s story was the basis for Michael Lewis’s book Moneyball (and the Brad Pitt film).  It’s the story of the campaign by which advanced metrics gained respect in baseball.

The problem, again, is that numbers play a lot nicer in baseball than they do in Hollywood.  Baseball can be broken down very effectively into isolated events.  A single batter takes on a single pitcher.  The batter has a place in the line-up and a certain arrangement of runners on-base or not.  The pitcher is left-handed or right-handed and has thrown a certain number of pitches to this point.  The stats file into order.  I don’t mean to diminish the work of baseball statisticians — it’s obviously difficult and more complicated than just recording a bunch of numbers.  My point is that there are clear events involving individual players.  The variables can be isolated.

A particular element of a script can’t even be isolated from its own script, let alone from the myriad other factors involved in the production, marketing, distribution, and exhibition of a movie.

Plus there’s the problem of sample size.  A single baseball team plays 162 games in a season.  The major Hollywood studios did not release that many movies all together last year.  Not all of their movies are even comparable.  The current blockbuster, wide-release market only goes back to the mid-70s.  But the increased prominence of international markets is an even more recent trend (1990s, really), and the vogue for CGI-heavy action/adventure movies based largely on comic books really can’t be drawn back much farther than the past decade.  So you have to be highly skeptical of any comparisons drawn to movies made before 1980, and you should be at least moderately skeptical of comparisons to movies made before 2000.

Common sense suggests there is simply no way for Bruzzese (or any statistician, for that matter) to stabilize the values he is targeting.  Therefore, it is likely he is mistaking signal for noise and drawing spurious conclusions.  Of course Bruzzese may have an answer for that, but if he does, he’s not saying in the article.

* * *

Now that I’ve explained why this particular article is useless (and why Bruzzese is probably a fraud – a savvy one, making the most of credulous, possibly illiterate studio execs), I’d like to talk about the real implications of a data-driven approach to making marketable art.

I suspect the common assumption about a numbers-driven approach to the production of art is that it will lead to a diminishment of originality in the spheres where it is employed.  There can be no numbers to support the production of something never before seen.

Indeed, if the use on Broadway and in Hollywood of the crude analytic tools already available is any indication, we should expect an even greater reliance on sequels, prequels, and spinoffs of branded properties and adaptations of established successes in other media.

What else would the diminishment of originality consist of?  Well, the narrowing of genre for one thing.  If the woman with the numbers is telling you that classical romantic comedies are a higher-risk, lower-reward prospect than buddy-cop movies, which genre are you going with from now on?  If musicals without choruses are half as likely to turn a profit as those with choruses, where do you find angels for the former?  The ideal end-point would be maybe only a handful of genres, at different expense-tiers, selected to maximize yield from an established audience.

It’s not static, of course.  Audiences do change, and the numbers would eventually encourage change along with the audience (though the mechanism would operate slowly).  What you’d likely hear is more of an already common refrain: “we’re not making x right now” (where x = a particular genre of story).

But artists have always managed to make dazzling, original work within strict genre limitations.  It’s easy to pick from history the abundant variety in musical compositions based on Christian religious themes, or in paintings of nudes, or short stories in the contemporary naturalistic mode.  The narrowing of permissible genres would not be enough to stifle creativity.  The narrowing of permissible use within genre, however, could be quite damaging.

The bulk of originality in art is the novel combination of existing materials.  “Books are made of other books,” as Cormac McCarthy put it.  What happens when the data limit what pieces you are allowed to use to make your detective story?  After all, this seems to be the direction Bruzzese’s work is headed.  Does a horror movie do better with or without comic relief?  Is it more profitable to get political satire out of your action movie?  What is the expected earnings value of a heist scene in a road-trip movie?

Cross-pollination of genre is what keeps the genres themselves alive.  If the borders between stories become fixed, then we will start to feel the lack of novelty.

Neither Hollywood nor Broadway is unaware of the value of mixing.  That’s why you can expect a romance plot – however cursory – to appear in the middle of just about any story.  But a data-driven approach will seek to define what combinations are acceptable.  The borders will lose the gaps that animate them.

Let me pause for a moment here to comment that it is unclear whether advanced numbers-crunching is any worse a method than those currently employed.  Producers obviously already try to reach the same conclusions, only with less sophisticated tools.  That’s where you get absurdities like the claim that you can’t make money with stories about women (followed by the tedious cover stories about Bridesmaids proving – at last! – that women can make a hit comedy).  Is a producer with dumb numbers and gut feelings any worse than a computer with smart numbers and no feelings?  Hard to say.

From an artistic standpoint, the prevailing theory seems to be that Golden Ages occur when the men with money frankly admit they have no idea what’s going on.  This is the story we tell ourselves about film in the 1970s and television in the 2000s.  Producers throw up their hands and give control to the artists.  What reality there is behind the myth, I don’t know, but it would seem to suggest that a producer who doubts his instincts a quarter of the time is better (for the artists) than a computer that never doubts.

Returning to the genre-mixing problem, and the porousness of borders, it seems highly improbable that even a computer take-over of Hollywood and Broadway would be able to fully banish creative alchemy.  Implied in the development of advanced analytic tools is the acceleration of a trend old beyond reckoning: the movement of destructive energy to the margins.

If the capital is a place of laws, then those who respect no laws will live outside the walls for a time.  In simpler terms, independent productions will persevere, and renewing energy will, as usual, be drawn from outside in.  Your Tennessee Williams, Sam Shepards, Steven Soderberghs (to name only very famous examples) will make their work cheaply, quietly for a time, but once they grow to a certain strength, they will be absorbed into the center, giving it new life.

Here’s a radically simplified image: in the forest beyond the city are spirits so wild they may hardly be touched.  In the hills between the forests and farms live scattered people, who go to the city for trade but who have also wandered the woods and spent time among the spirits.  In the city are people mostly happy behind their walls but who must venture out from time to time or else go mad with order.  The spirits in the dark of the forest are the avant-garde: they are hardly seen by the city-dwellers and can’t be absorbed except by destroying the woods in which they live (in which case they simply move).  The hill-people are artists occupying a median space.  They are influenced by the avant-garde, and their work lies, at first, outside the center.  But they are also influenced by the center.  And if they experience success, if they grow beyond a certain point, a thriving city must certainly absorb them.

Imagine now that access to the center is governed by computer algorithms.  While such a mechanism might initially firm up the walls, turning back more hill-dwellers and threshold artists, it will necessarily admit them once they are clothed in the numbers of success.

As I said above, the situation is not static.  The computer will turn back rule-breakers until the moment they prove that breaking a certain rule makes money.  At that point the computers will re-write the rules (move the walls), letting some new blood in and forcing dedicated outsiders to move further out.

This drama is old, old, old.  And the role of hard-data is bivalent, at least.  On one hand, yes, the certainty of numbers fixes the walls and polices the streets, making it much harder for change to come from within.  On the other hand, a computer is neither embarrassed nor stubborn: when a verity is challenged, the computer will not sit there talking about gut feelings or the old ways.

* * *

The rise of advanced metrics in sports, politics, and culture is matched by the ascendance of a rather new expression of popular will.  While Bruzzese is going on about his experiment in crunching numbers, others are essentially crowd-sourcing their producing decisions.

Amazon recently produced ten pilot episodes for prospective television shows.  Rather than submit these programs to the usual battery of internal tests and closed-doors decisions based on Nielsen ratings, Amazon is letting viewers decide, through clicks, views, and surveys, which shows should go to series.

Amazons tactic is related in spirit to the increased commoditization of youtube success.  Youtube views and likes are about as unmediated an expression of public will you’re going to find.  Hugely successful video-makers now find themselves with sponsored channels and the chance to make more (and more sophisticated videos), as well as actual money.

Then there’s Kickstarter.  Successful campaigns have shown that crowd-funding can obviate the need for project approval from a central authority.  Movies, games, plays, and web-series have all been funded by online donations from friends, fans, and curious supporters.

Assuming advances on both sides, how will these two trends – statistical analysis and crowd-sourcing – interact?  It appears from a distance that they represent an opposition.  The man with a forecasting model is a soothsayer or royal advisor, holding esoteric knowledge.  The internet hordes are the public, making no claim to knowledge but expressing profound preferences.  One is dispersed and democratic.  The other concentrated and effectively autocratic.

Again, there’s ambiguity in these forces.  The public may be getting us a new Veronica Mars movie, but how effective is it at supporting projects by unknown artists?  And when it comes to the Amazon pilots, is the tyranny of the majority (to borrow from J.S. Mill) necessarily an improvement on the impartiality of the numbers?

An actual inquiry into this question is certainly beyond the scope of this article.  Honestly, it’s just beyond the scope of my knowledge.  It felt foolish to avoid bringing it up though, since any hypothetical about the future of funding popular entertainment is bound to involve more than one driving force.

* * *

The second half of this post has been an imagination exercise.  Though the Times article from Monday is entirely unconvincing, what if it were possible to assess a work of art’s market potential at the draft stage?

I’d like to make a few more points about such a case, before I drop.

In 1936, in his landmark “Art In The Age Of Mechanical Reproduction”, Walter Benjamin wrote that, “works of art are received and valued on different planes. Two polar types stand out; with one, the accent is on the cult value; with the other, on the exhibition value of the work.”  The long-term effect of mechanical reproduction has been to diminish the cult value of art and to increase the emphasis on exhibition value.  “By the absolute emphasis on its exhibition value,” said Benjamin, “the work of art becomes a creation with entirely new functions among which the one we are conscious of, the artistic function, later may be recognized as incidental.”  Would anyone argue that the art function has not become a secondary, incidental concern in the creation of Hollywood and Broadway?

A world where computers can accurately predict our entertainment desires is one that further accepts that art is the same as commerce.  While this is an obvious point to major producers, audiences and artists don’t like to be reminded of it.  And yet we hardly bat an eye while referring to a new play or movie as a product.  We talk freely of getting our money’s worth.  We’ve already swallowed the idiom and therefore the worldview.

At an imagined point where producers (again, an economic term) can take the guesswork out of a trip to the theater, we as audiences will no longer be lining up for a new experience.  We will be lining up for a new dose.

But this arrangement may be unstable.  We don’t like to be predictable, and more than that, we do like to be surprised.  Which leads me to the uncertainty principle: the more accurately a computer is able in theory to predict a project’s success, the less accurate the predictions in general will become.  When novelty drops below a certain threshold (therefore when machine certainty rises past the same point), the audience will lose interest in the work, and therefore predictions will cease to match reality.  In fact, the prediction itself will effectively cause the lack of predictability.

Now that’s a hopeful thought.  As a counter-point, we can look to the increasing public awareness and understanding of advertising: we are as savvy as ever about ways we are being sold products, but instead of becoming indignant and refusing the overtures, we congratulate ourselves on our savvy (and congratulate the advertisers on their post-modern savvy for playfully engaging our awareness (in a winking “I know that you know that I know” game) and move on in the same direction.  I’d like to think the uncertainty principle is real and powerful, but Iron Man 3 just had the second-greatest opening weekend in history.

There’s a last underlying problem here, so basic as to escape notice.  Producers and audiences do not, in fact, share objectives.  If producers claim to seek financial success above all, then they hope to obtain that by pleasing their audience.  This ideally occurs opening weekend or in the earliest portion of a run, but must occur (to meet the producer’s need) during the producer’s lifetime.  The audience meanwhile, hopes to be pleased.  But this does not have to happen on any schedule.  The audience for a work of art exists precisely as long as that work of art is accessible.  Therefore a work of art that is a complete failure to the producer (it makes no money while she is alive) may eventually be an unbounded success to an audience.

This is a long way of saying that, from an audience perspective, it is actually good for financial failures to be produced.  It is good for art to be funded even if it has no chance of success in its own time.  The history of art is littered with examples of genius unappreciated.  Van Gogh sold only two paintings during his lifetime.

If we rig the system to only give us what we want, we are paradoxically depriving ourselves.

Stephen Foglia, Literary Manager