Measuring Impact: The Importance of Evaluation for Documentary Film Campaigns

Abstract

Introduction Documentary film has grown significantly in the past decade, with high profile films such as Fahrenheit 9/11, Supersize Me, and An Inconvenient Truth garnering increased attention both at the box office and in the news media. In addition, the rising prominence of web-based media has provided new opportunities for documentary to create social impact. Films are now typically released with websites, Facebook pages, twitter feeds, and web videos to increase both reach and impact. This combination of technology and broader audience appeal has given rise to a current landscape in which documentary films are imbedded within coordinated multi-media campaigns. New media have not only opened up new avenues for communicating with audiences, they have also created new opportunities for data collection and analysis of film impacts. A recent report by McKinsey and Company highlighted this potential, introducing and discussing the implications of increasing consumer information being recorded on the Internet as well as through networked sensors in the physical world. As they found: "Big data—large pools of data that can be captured, communicated, aggregated, stored, and analyzed—is now part of every sector and function of the global economy" (Manyika et al. iv).  This data can be mined to learn a great deal about both individual and cultural response to documentary films and the issues they represent. Although film has a rich history in humanities research, this new set of tools enables an empirical approach grounded in the social sciences. However, several researchers across disciplines have noted that limited investigation has been conducted in this area. Although there has always been an emphasis on social impact in film and many filmmakers and scholars have made legitimate (and possibly illegitimate) claims of impact, few have attempted to empirically justify these claims. Over fifteen years ago, noted film scholar Brian Winston commented that "the underlying assumption of most social documentaries—that they shall act as agents of reform and change—is almost never demonstrated" (236). A decade later, Political Scientist David Whiteman repeated this sentiment, arguing that, "despite widespread speculation about the impact of documentaries, the topic has received relatively little systematic attention" ("Evolving"). And earlier this year, the introduction to a special issue of Mass Communication and Society on documentary film stated, "documentary film, despite its growing influence and many impacts, has mostly been overlooked by social scientists studying the media and communication" (Nisbet and Aufderheide 451). Film has been studied extensively as entertainment, as narrative, and as cultural event, but the study of film as an agent of social change is still in its infancy. This paper introduces a systematic approach to measuring the social impact of documentary film aiming to: (1) discuss the context of documentary film and its potential impact; and (2)  argue for a social science approach, discussing key issues about conducting such research. Changes in Documentary Practice Documentary film has been used as a tool for promoting social change throughout its history. John Grierson, who coined the term "documentary" in 1926, believed it could be used to influence the ideas and actions of people in ways once reserved for church and school.  He presented his thoughts on this emerging genre in his 1932 essay, First Principles of Documentary, saying, "We believe that the cinema's capacity for getting around, for observing and selecting from life itself, can be exploited in a new and vital art form" (97). Richard Barsam further specified the definition of documentary, distinguishing it from non-fiction film, such that all documentaries are non-fiction films but not all non-fiction films are documentaries. He distinguishes documentary from other forms of non-fiction film (i.e. travel films, educational films, newsreels) by its purpose; it is a film with an opinion and a specific message that aims to persuade or influence the audience. And Bill Nichols writes that the definition of documentary may even expand beyond the film itself, defining it as a "filmmaking practice, a cinematic tradition, and mode of audience reception" (12). Documentary film has undergone many significant changes since its inception, from the heavily staged romanticism movement of the 1920s to the propagandist tradition of governments using film to persuade individuals to support national agendas to the introduction of cinéma vérité in the 1960s and historical documentary in the 1980s (cf. Barnouw). However, the recent upsurge in popularity of documentary media, combined with technological advances of internet and computers have opened up a whole new set of opportunities for film to serve as both art and agent for social change. One such opportunity is in the creation of film-based social action campaigns. Over the past decade, filmmakers have taken a more active role in promoting social change by coordinating film releases with action campaigns. Companies such as Participant Media (An Inconvenient Truth, Food Inc., etc.) now create "specific social action campaigns for each film and documentary designed to give a voice to issues that resonate in the films" (Participant Media).  In addition, a new sector of "social media" consultants are now offering services, including "consultation, strategic planning for alternative distribution, website and social media development, and complete campaign management services to filmmakers to ensure the content of nonfiction media truly meets the intention for change" (Working Films). The emergence of new forms of media and technology are changing our conceptions of both documentary film and social action. Technologies such as podcasts, video blogs, internet radio, social media and network applications, and collaborative web editing "both unsettle and extend concepts and assumptions at the heart of 'documentary' as a practice and as an idea" (Ellsworth). In the past decade, we have seen new forms of documentary creation, distribution, marketing, and engagement. Likewise, film campaigns are utilizing a broad array of strategies to engage audience members, including "action kits, screening programs, educational curriculums and classes, house parties, seminars, panels" that often turn into "ongoing 'legacy' programs that are updated and revised to continue beyond the film's domestic and international theatrical, DVD and television windows" (Participant Media).  This move towards multi-media documentary film is becoming not only commonplace, but expected as a part of filmmaking. NYU film professor and documentary film pioneer George Stoney recently noted, "50 percent of the documentary filmmaker's job is making the movie, and 50 percent is figuring out what its impact can be and how it can move audiences to action"  (qtd. in Nisbet, "Gasland"). In his book Convergence Culture, Henry Jenkins, coined the term "transmedia storytelling", which he later defined as "a process where integral elements of a fiction get dispersed systematically across multiple delivery channels for the purpose of creating a unified and coordinated entertainment experience" ("Transmedia"). When applied to documentary film, it is the elements of the "issue" raised by the film that get dispersed across these channels, coordinating, not just an entertainment experience, but a social action campaign. Dimensions of Evaluation It is not unreasonable to assume that such film campaigns, just like any policy or program, have the possibility to influence viewers' knowledge, attitudes, and behavior. Measuring this impact has become increasingly important, as funders of documentary and issue-based films want look to understand the "return on investment" of films in terms of social impact so that they can compare them with other projects, including non-media, direct service projects.  Although we "feel" like films make a difference to the individuals who also see them in  the broader cultures in which they are embedded, measurement and empirical analysis of this impact are vitally important for both providing feedback to filmmakers and funders as well as informing future efforts attempting to leverage film for social change. This type of systematic assessment, or program evaluation, is often discussed in terms of two primary goals—formative (or process) and summative (or impact) evaluation (cf. Muraskin; Trochim and Donnelly). Formative evaluation studies program materials and activities to strengthen a program, and summative evaluation examines program outcomes. In terms of documentary film, these two goals can be described as follows: Formative Evaluation: Informing the Process As programs (broadly defined as an intentional set of activities with the aim of having some specific impact), the people who interact with them, and the cultures they are situated in are constantly changing, program development and evaluation is an ongoing learning cycle. Film campaigns, which are an intentional set of activities with the aim of impacting individual viewers and broader cultures, fit squarely within this purview. Without formulating hypotheses about the relationships between program activities and goals and then collecting and analyzing data during implementation to test them, it is difficult to learn ways to improve programs (or continue doing what works best in the most efficient manner).  Attention to this process enables those involved to learn more about, not only what works, but how and why it works and even gain insights about how program outcomes may be affected by changes to resource availability, potential audiences, or infrastructure. Filmmakers are constantly learning and honing their craft and realizing the  impact of their practice can help the artistic process. Often faced with tight budgets and timelines, they are forced to confront tradeoffs all the time, in the writing, production and post-production process. Understanding where they are having impact can improve their decision-making, which can help both the individual project and the overall field. Summative Evaluation: Quantifying Impacts Evaluation is used in many different fields to determine whether programs are achieving their intended goals and objectives. It became popular in the 1960s as a way of understanding the impact of the Great Society programs and has continued to grow since that time (Madaus and Stufflebeam). A recent White House memo stated that "rigorous, independent program evaluations can be a key resource in determining whether government programs are achieving their intended outcomes as well as possible and at the lowest possible cost" and the United States Office of Management and Budget (OMB) launched an initiative to increase the practice of "impact evaluations, or evaluations aimed at determining the causal effects of programs" (Orszag 1). Documentary films, like government programs, generally target a national audience, aim to serve a social purpose, and often do not provide a return on their investment. Participant Media, the most visible and arguably most successful documentary production company in the film industry, made recent headlines for its difficulty in making a profit during its seven-year history (Cieply). Owner and founder Jeff Skoll reported investing hundreds of millions of dollars into the company and CEO James Berk added that the company sometimes measures success, not by profit, but by "whether Mr. Skoll could have exerted more impact simply by spending his money philanthropically" (Cieply). Because of this, documentary projects often rely on grant funding, and are starting to approach funders beyond traditional arts and media sources. "Filmmakers are finding new fiscal and non-fiscal partners, in constituencies that would not traditionally be considered—or consider themselves—media funders or partners" (BRITDOC 6). And funders increasingly expect tangible data about their return on investment. Says Luis Ubiñas, president of Ford Foundation, which recently launched the Just Films Initiative: In these times of global economic uncertainty, with increasing demand for limited philanthropic dollars, assessing our effectiveness is more important than ever. Today, staying on the frontlines of social change means gauging, with thoughtfulness and rigor, the immediate and distant outcomes of our funding. Establishing the need for evaluation is not enough—attention to methodology is also critical. Valid research methodology is a critical component of understanding around the role entertainment can play in impacting social and environmental issues. The following issues are vital to measuring impact. Defining the Project Though this may seem like an obvious step, it is essential to determine the nature of the project so one can create research questions and hypotheses based on a complete understanding of the "treatment". One organization that provides a great example of the integration of documentary film imbedded into a larger campaign or movement is Invisible Children. Founded in 2005, Invisible Children is both a media-based organization as well as an economic development NGO with the goal of raising awareness and meeting the needs of child soldiers and other youth suffering as a result of the ongoing war in northern Uganda. Although Invisible Children began as a documentary film, it has grown into a large non-profit organization with an operating budget of over $8 million and a staff of over a hundred employees and interns throughout the year as well as volunteers in all 50 states and several countries. Invisible Children programming includes films, events, fundraising campaigns, contests, social media platforms, blogs, videos, two national "tours" per year, merchandise, and even a 650-person three-day youth summit in August 2011 called The Fourth Estate.  Individually, each of these components might lead to specific outcomes; collectively, they might lead to others. In order to properly assess impacts of the film "project", it is important to take all of these components into consideration and think about who they may impact and how. This informs the research questions, hypotheses, and methods used in evaluation. Film campaigns may even include partnerships with existing social movements and non-profit organizations targeting social change. The American University Center for Social Media concluded in a case study of three issue-based documentary film campaigns: Digital technologies do not replace, but are closely entwined with, longstanding on-the-ground activities of stakeholders and citizens working for social change. Projects like these forge new tools, pipelines, and circuits of circulation in a multiplatform media environment. They help to create sustainable network infrastructures for participatory public media that extend from local communities to transnational circuits and from grassroots communities to policy makers. (Abrash) Expanding the Focus of Impact beyond the Individual A recent focus has shifted the dialogue on film impact. Whiteman ("Theaters") argues that traditional metrics of film "success" tend to focus on studio economic indicators that are far more relevant to large budget films. Current efforts focused on box office receipts and audience size, the author claims, are really measures of successful film marketing or promotion, missing the mark when it comes to understanding social impact. He instead stresses the importance of developing a more comprehensive model. His  "coalition model" broadens the range and types of impact of film beyond traditional metrics to include the entire filmmaking process, from production to distribution. Whiteman (“Theaters”) argues that a narrow focus on the size of the audience for a film, its box office receipts, and viewers' attitudes does not incorporate the potential reach of a documentary film. Impacts within the coalition model include both individual and policy levels. Individual impacts (with an emphasis on activist groups) include educating members, mobilizing for action, and raising group status; policy includes altering both agenda for and the substance of policy deliberations. The Fledgling Fund (Barrett and Leddy) expanded on this concept and identified five distinct impacts of documentary film campaigns. These potential impacts expand from individual viewers to groups, movements, and eventually to what they call the "ultimate goal" of social change. Each is introduced briefly below. Quality Film. The film itself can be presented as a quality film or media project, creating enjoyment or evoking emotion in the part of audiences. "By this we mean a film that has a compelling narrative that draws viewers in and can engage them in the issue and illustrate complex problems in ways that statistics cannot" (Barrett and Leddy, 6). Public Awareness. Film can increase public awareness by bringing light to issues and stories that may have otherwise been unknown or not often thought about. This is the level of impact that has received the most attention, as films are often discussed in terms of their "educational" value. "A project's ability to raise awareness around a particular issue, since awareness is a critical building block for both individual change and broader social change" (Barrett and Leddy, 6). Public Engagement. Impact, however, need not stop at simply raising public awareness. Engagement "indicates a shift from simply being aware of an issue to acting on this awareness. Were a film and its outreach campaign able to provide an answer to the question 'What can I do?' and more importantly mobilize that individual to act?" (Barrett and Leddy, 7). This is where an associated film campaign becomes increasingly important, as transmedia outlets such as Facebook, websites, blogs, etc. can build off the interest and awareness developed through watching a film and provide outlets for viewers channel their constructive efforts. Social Movement. In addition to impacts on individuals, films can also serve to mobilize groups focused on a particular problem. The filmmaker can create a campaign around the film to promote its goals and/or work with existing groups focused on a particular issue, so that the film can be used as a tool for mobilization and collaboration. "Moving beyond measures of impact as they relate to individual awareness and engagement, we look at the project's impact as it relates to the broader social movement … if a project can strengthen the work of key advocacy organizations that have strong commitment to the issues raised in the film" (Barrett and Leddy, 7). Social Change. The final level of impact and "ultimate goal" of an issue-based film is long-term and systemic social change. "While we understand that realizing social change is often a long and complex process, we do believe it is possible and that for some projects and issues there are key indicators of success" (Barrett and Leddy, 7). This can take the form of policy or legislative change, passed through film-based lobbying efforts, or shifts in public dialogue and behavior. Legislative change typically takes place beyond the social movement stage, when there is enough support to pressure legislators to change or create policy. Film-inspired activism has been seen in issues ranging from environmental causes such as agriculture (Food Inc.) and toxic products (Blue Vinyl) to social causes such as foreign conflict (Invisible Children) and education (Waiting for Superman). Documentary films can also have a strong influence as media agenda-setters, as films provide dramatic "news pegs" for journalists seeking to either sustain or generation new coverage of an issue (Nisbet "Introduction" 5), such as the media coverage of climate change in conjunction with An Inconvenient Truth. Barrett and Leddy, however, note that not all films target all five impacts and that different films may lead to different impacts. "In some cases we could look to key legislative or policy changes that were driven by, or at least supported by the project... In other cases, we can point to shifts in public dialogue and how issues are framed and discussed" (7). It is possible that specific film and/or campaign characteristics may lead to different impacts; this is a nascent area for research and one with great promise for both practical and theoretical utility. Innovations in Tools and Methods Finally, the selection of tools is a vital component for assessing impact and the new media landscape is enabling innovations in the methods and strategies for program evaluation. Whereas the traditional domain of film impact measurement included box office statistics, focus groups, and exit surveys, innovations in data collection and analysis have expanded the reach of what questions we can ask and how we are able to answer them. For example, press coverage can assist in understanding and measuring the increase in awareness about an issue post-release. Looking directly at web-traffic changes "enables the creation of an information-seeking curve that can define the parameters of a teachable moment" (Hart and Leiserowitz 360). Audience reception can be measured, not only via interviews and focus groups, but also through content and sentiment analysis of web content and online analytics. "Sophisticated analytics can substantially improve decision making, minimize risks, and unearth valuable insights that would otherwise remain hidden" (Manyika et al. 5). These new tools are significantly changing evaluation, expanding what we can learn about the social impacts of film through triangulation of self-report data with measurement of actual behavior in virtual environments. Conclusion The changing media landscape both allows and impels evaluation of film impacts on individual viewers and the broader culture in which they are imbedded. Although such analysis may have previously been limited to box office numbers, critics' reviews, and theater exit surveys, the rise of new media provides both the ability to connect filmmakers, activists, and viewers in new ways and the data in which to study the process. This capability, combined with significant growth in the documentary landscape, suggests a great potential for documentary film to contribute to some of our most pressing social and environmental needs. A social scientific approach, that combines empirical analysis with theory applied from basic science, ensures that impact can be measured and leveraged in a way that is useful for both filmmakers as well as funders. In the end, this attention to impact ensures a continued thriving marketplace for issue-based documentary films in our social landscape. References Abrash, Barbara. "Social Issue Documentary: The Evolution of Public Engagement." American University Center for Social Media 21 Apr. 2010. 26 Sep. 2011 ‹http://www.centerforsocialmedia.org/›. Aufderheide, Patricia. "The Changing Documentary Marketplace." Cineaste 30.3 (2005): 24-28. Barnouw, Eric. Documentary: A History of the Non-Fiction Film. New York: Oxford UP, 1993. Barrett, Diana and Sheila Leddy. "Assessing Creative Media's Social Impact." The Fledgling Fund, Dec. 2008. 15 Sep. 2011 ‹http://www.thefledglingfund.org/media/research.html›. Barsam, Richard M. Nonfiction Film: A Critical History. Bloomington: Indiana UP. 1992. BRITDOC Foundation. The End of the Line: A Social Impact Evaluation. London: Channel 4, 2011. 12 Oct. 2011 ‹http://britdoc.org/news_details/the_social_impact_of_the_end_of_the_line/›. Cieply, Michael. "Uneven Growth for Film Studio with a Message." New York Times 5 Jun. 2011: B1. Ellsworth, Elizabeth. "Emerging Media and Documentary Practice." The New School Graduate Program in International Affairs. Aug. 2008. 22 Sep. 2011. ‹http://www.gpia.info/node/911›. Grierson, John. "First Principles of Documentary (1932)." Imagining Reality: The Faber Book of Documentary. Eds. Kevin Macdonald and Mark Cousins. London: Faber and Faber, 1996. 97-102. Hart, Philip Solomon and Anthony Leiserowitz. "Finding the Teachable Moment: An Analysis of Information-Seeking Behavior on Global Warming Related Websites during the Release of The Day After Tomorrow." Environmental Communication: A Journal of Nature and Culture 3.3 (2009): 355-66. Jenkins, Henry. Convergence Culture: Where Old and New Media Collide. New York: New York UP, 2006. ———. "Transmedia Storytelling 101." Confessions of an Aca-Fan. The Official Weblog of Henry Jenkins. 22 Mar. 2007. 10 Oct. 2011 ‹http://www.henryjenkins.org/2007/03/transmedia_storytelling_101.html›. Madaus, George, and Daniel Stufflebeam. "Program Evaluation: A Historical Overview." Evaluation in Education and Human Services 49.1 (2002): 3-18. Manyika, James, Michael Chui, Jacques Bughin, Brad Brown, Richard Dobbs, Charles Roxburgh, and Angela Hung Byers. Big Data: The Next Frontier for Innovation, Competition, and Productivity. McKinsey Global Institute. May 2011 ‹http://www.mckinsey.com/mgi/publications/big_data/›. Muraskin, Lana. Understanding Evaluation: The Way to Better Prevention Programs. Washington: U.S. Department of Education, 1993. 8 Oct. 2011 ‹http://www2.ed.gov/PDFDocs/handbook.pdf›. Nichols, Bill. "Foreword." Documenting the Documentary: Close Readings of Documentary Film and Video. Eds. Barry Keith Grant and Jeannette Sloniowski. Detroit: Wayne State UP, 1997. 11-13. Nisbet, Matthew. "Gasland and Dirty Business: Documentary Films Shape Debate on Energy Policy." Big Think, 9 May 2011. 1 Oct. 2011 ‹http://bigthink.com/ideas/38345›. ———. "Introduction: Understanding the Social Impact of a Documentary Film." Documentaries on a Mission: How Nonprofits Are Making Movies for Public Engagement. Ed. Karen Hirsch,  Center for Social Media. Mar. 2007. 10 Sep. 2011 ‹http://aladinrc.wrlc.org/bitstream/1961/4634/1/docs_on_a_mission.pdf›. Nisbet, Matthew, and Patricia Aufderheide. "Documentary Film: Towards a Research Agenda on Forms, Functions, and Impacts." Mass Communication and Society 12.4 (2011): 450-56. Orszag, Peter. Increased Emphasis on Program Evaluation. Washington: Office of Management and Budget. 7 Oct. 2009. 10 Oct. 2011 ‹http://www.whitehouse.gov/sites/default/files/omb/assets/memoranda_2010/m10-01.pdf›. Participant Media. "Our Mission." 2011. 2 Apr. 2011 ‹http://www.participantmedia.com/company/about_us.php.›. Plantinga, Carl. Rhetoric and Representation in Nonfiction Film. Cambridge: Cambridge UP, 1997. Trochim, William, and James Donnelly. Research Methods Knowledge Base. 3rd ed. Mason: Atomic Dogs, 2007. Ubiñas, Luis. "President's Message." 2009 Annual Report. Ford Foundation, Sep. 2010. 10 Oct. 2011 ‹http://www.fordfoundation.org/about-us/2009-annual-report/presidents-message›. Vladica, Florin, and Charles Davis.  "Business Innovation and New Media Practices in Documentary Film Production and Distribution: Conceptual Framework and Review of Evidence." The Media as a Driver of the Information Society. Eds. Ed Albarran, Paulo Faustino, and R. Santos. Lisbon, Portugal: Media XXI / Formal, 2009. 299-319. Whiteman, David. "Out of the Theaters and into the Streets: A Coalition Model of the Political Impact of Documentary Film and Video." Political Communication 21.1 (2004): 51-69. ———. "The Evolving Impact of Documentary Film: Sacrifice and the Rise of Issue-Centered Outreach." Post Script 22 Jun. 2007. 10 Sep. 2011 ‹http://www.allbusiness.com/media-telecommunications/movies-sound-recording/5517496-1.html›. Winston, Brian. Claiming the Real: The Documentary Film Revisited. London: British Film Institute, 1995. Working Films. "Nonprofits: Working Films." Foundation Source Access 31 May 2011. 5 Oct. 2011 ‹http://access.foundationsource.com/nonprofit/working-films/›.