2.1 Individual Case Studies

One main approach to measuring the impact of documentaries is through case studies, i.e. collections of quantitative metrics and/or anecdotal reports about a single production. Two examples are the assessment of “Legacy” (Applied_Research_Consulting_LLC, 2002), and the Working Films’ evaluation of “Blue Vinyl” (Barrett & Leddy, 2008). Also, BritDoc has released a series of high quality impact assessment studies (http://britdoc.org/real_good/evaluation). Such evaluations approximate the influence of a documentary by considering (a combination of) the following indicators:

• Cumulative counts of the number of screenings, video distributions, or people reached through campaign activities.
• Comments from individual viewers; analyzed qualitatively on a case by case basis.
• Lists of key organizations participating in the documentary–related campaign. Connections between these organizations are typically not considered.
• A few instances of policy adoption.

Overall, case studies can be useful in highlighting the outcomes of a specific documentary. However they do not generalize to other productions. In other words, this approach fails to ensure that the same methodology is applicable across productions and genres such that findings for multiple films could be compared.

A widely adopted technology in this field that supports people in visually exploring and analyzing the data they have gathered and entered about their production is Sparkwise (http://sparkwi.se/), which offers a visually highly appealing and professional graphical user interface. Also, the Harmony Institute has recently releases of StoryPilot; an impact assessment technology for documentaries that allows for conduct assessment across productions (http://www.storypilot.org/).

2.2 Previously Proposed Frameworks

Various major media institutes and foundations, including the Center for Social Media, the Fledgling Fund, the Knight Foundation and the Rockefeller Foundation, have proposed systematic frameworks for impact assessment (Barrett & Leddy, 2008; Clark & Abrash, 2011; Figueroa, 2002; KnightFoundation, 2011). Each of these organizations has released such a framework, which typically involves the measurement of impact along five to seven dimensions that entail the following: the aforementioned quantitative metrics plus influence on the individual, community and societal level.

The main limitation with these frameworks is their normative and theoretical character, which means that applying them for practical purposes might require adaptations and changes to make them actionable. Furthermore, the indicators recommended in prior frameworks are highly similar to the anecdotal evidence mentioned in the case studies section. In terms of methodology, these frameworks typically combine simple cumulative frequency counts (number of screenings, viewers, website visitors and supportive organizations) with analyses of small samples of narrative descriptions from self–reports from participants.

Even more relevant with respect to our project is the fact that some of the proposed frameworks include indicators related to capturing social networks: for example, “interorganizational collaboration” (Fledgling Fund), “network building” (Center for Social Media), and “network cohesion” (Rockefeller Foundation) have been mentioned as key ingredients. However, there are no further details on how to collect, analyze, interpret and leverage such network data. Even where core network metrics, such as density and centrality, are mentioned (Rockefeller Foundation), these terms are simply introduced as possible metrics without providing information or practical guidance for how to use these metrics in an evaluation process.

2.3 Academic Research

The majority of scholarly work on this topic is confined to studying psychological effects of documentaries on individual viewers. Thus, most scholarly publications consider documentaries as a subcategory of mass media. A few exceptions exist: Whiteman (2004) uses a political science perspective to study several factors that affect a documentary’s impact. However, since his framework heavily depends on qualitative analysis such as observations and content analysis, it is highly similar to the first two groups of approaches.

Summarizing the reviewed families of assessment approaches, we conclude that although various types of approaches have been suggested and applied, most of them are similar in that they jointly consider traditional frequency counts on a large scale and qualitative indicators on a small scale. Several proposals have emphasized the importance of taking social networks and the content of information associated with network members into consideration. At least in the domain of assessing the impact of documentaries, these strategies are waiting to be put into action. The work presented herein is a step into this direction.

3.1 Theoretical Framework: CoMTI

There is little prior evidence that conclusively confirms or negates impact of media in general (Sparks, 2012). Even with advanced research designs, evidence for a causal relationship between media and impact (in that direction) remains vague. Several lab experiments have successfully shown short–term impacts. However, such highly controlled lab study settings are a limitation to the generalization of any findings to real–world situations. More importantly, the small–scale and typically point–wise nature of such studies often prevents longitudinal insights. Overall and despite many open questions about media impact, scholars agree that media content affects our perception and behavior in certain, maybe latent, ways (Bryant & Oliver, 2008; Laughey, 2007). A large common denominator of media effects research is the belief that humans can be affected by media stimuli. The holistic process of how stimuli influence people has been dissected into five categories; all of which were originally suggested by Laswell in his model of communication (Johnson & Klare, 1961; Lasswell, 1948). Most theories of media effects fit into one or more of these categories (Laughey, 2007). We use the Laswell model as a backbone for our theoretical framework by empirically identifying: What has been said (content) on which channel (medium) to whom (target) and with what effects (impact)? The Who dimension is partially entailed in the medium dimension, and will also be considered when we extract (groups of) stakeholders from network data, and by bringing text mining methods to the medium dimension. In the Lasswell formula, communication happens in order to influence a target audience. Thus, communication is conceptualized as a persuasive process (McQuail, 2010). This aligns with the goal of documentaries to lead to change in people’s knowledge and/ or behavior.

Applying the provided definition of media use, we argue that a documentary is not some one–way communication where some agent (seeks to) transfer ideas or messages to others in order to achieve certain effects, but rather a two–way process in which senders and receivers interact with each other: receivers’ responses and reactions to senders’ input form dynamic feedback loops. This inherently reciprocal and iterative process is represented in our framework as shown in Figure 1, and is essential to overcome Lasswell’s conceptualization which has been criticized for it’s a linear, one–way direction of communication flow. Such feedback loops have high practical implications as film producers and engagement workers can leverage them to model the landscape of stakeholders and discourse associated with the theme of a documentary prior to and during release in order to identify relevant social agents and themes to connect with. This helps to strategically allocate scarce resources.

Figure 1: CoMTI framework with a Feedback Loops

We have synthesized the indicators of impact as suggested by prior work into a framework that we named CoMTI (content, medium, target, and impact). The framework is shown in table 2. This model is organized along the main dimensions of impact assessment and respective methods as explained below:

• Dimension: a component or process through which a documentary can achieve impact.
• Level: a set of sub–categories of evaluation criteria per dimension.
• Index: a set of evaluation factors per level.
• Analytics: suitable methods for discovering meaningful results per index category.
• Item: a set of specific features to be measured per index.

The framework is grounded in a set of theories, and allows for large–scale, multi–level analysis by considering the following features:

• Theoretical foundation: CoMTI is based on empirically and rigorously tested theories from domains including diffusion of innovation and information, media effects, marketing, social and semantic networks and collective action.
• Domain expertise: CoMTI incorporates concepts specific to documentary evaluation that were suggested to us in consultations with subject matter experts from this domain.
• Analytical Comprehensiveness: considered analytical methods and metrics originating from statistics, network analysis and text analysis.
• Multi–modal units of analysis: CoMTI considers the entity types of people, organization and information.
• Integrated approach: CoMTI combines traditional strategies for measuring documentary impact (frequency counts and qualitative analysis) with additional methods (network analysis, text analysis).

This framework entails a variety of stimuli that have been associated with cognitive, attitudinal and behavioral changes over time on the individual, communal, societal and global level. In this context, we consider a documentary as a special kind of media product. When it comes to identifying the impact of media content on people, prior work can be divided into three categories (Laughey, 2007):

• Direct impact: media content can have powerful influence on the knowledge and behavior of the audience.
• Indirect impact: media content is one of several factors that affect peoples’ behavior and cognition.
• Null impact: media content does not have any significant influence on peoples’ cognitive and behavior.

The proposed framework assumes that (some portion of) the impact of a documentary can be measured; and that this impact can be direct, indirect or not evoked. Also, we conceptualize the entire process of making and distributing a documentary as a communication process, where participants exchange information and knowledge via behavioral signals, including natural language (Griffin & McClish, 2003). The CoMTI framework borrows elements from verified outcomes of media studies, but is also unique in the following three ways:

• While most studies of media effects focus on one or two phases of the Lasswell’s formula, our framework models the whole communication processes around a documentary.
• The proposed framework overcomes the linear, sender–driven, one–way flow of communication.
• The proposed framework is tailored towards measuring the impact of documentaries by integrating dependent variables into measurable indices.

Table 1: CoMTI Framework for Impact Assessment

In the next section, we briefly detail every dimension of the CoMTI framework.

3.1.3 Target

In marketing, the size of the reachable target audience matters as it determines for instance the cost–per–person of an advertisement. However for documentaries, this rationale does not apply, mainly because producers have no tangible metric for assessing effectiveness other than the number of pairs of eyes that have watched a film. Thus, the size of the audience can translate into impact, but needs to be complemented with additional factors (Barrett & Leddy, 2008; Clark & Abrash, 2011; Figueroa, 2002; John & James, 2011).

Another issue related to the target dimension is audience diversity: the more heterogeneous the audience, the broader the reach. Studies in risk communication, marketing, social influence and diffusion have shown that audiences who are homogeneous in terms of age, sex, income, education or physical proximity can limit the ripple effect of communication (Page, 2007; Prell, 2012; Rogers, 2003).

A classical finding from media effect studies is that ideas flow from media to opinion leaders to the rest of the world (E. Katz & Lazarsfeld, 2006; Lundgren & McMakin, 2011). In the CoMTI framework, formal opinion leaders, e.g. media editors and professional critics, are distinguished from informal opinion leaders, such as popular bloggers and grass–root organizations. The latter type of influencers can be identified from social media data via social network analysis (Hansen, Shneiderman, & Smith, 2010; Watts, 2007).

One common feature of previous efforts to measure documentary impact is the focus on advocacy (Barrett & Leddy, 2008; Clark & Abrash, 2011; John & James, 2011). Established communities of practice can be powerful change agents because members of tightly knit groups are subject to group norms (Drazin & Schoonhoven, 1996; Rogers, 2003). The importance of communities as change agents justifies their inclusion as a separate indicator in CoMTI.

Data for measuring the indices for the Target dimension mainly come from statistical reports by documentary producers, web analytics, surveys and archival records. For identifying informal opinion leaders, social network analysis can be used.

3.1.4 Impact

In the ComTI framework, impact is measured as a weighted function over four stimulus dimensions that are associated with cognitive, attitudinal and behavioral changes over time on the individual, communal, societal, and global level. Sometimes, a change might be clearly associated with a stimulus, e.g. the creation of a new piece of legislature or the adoption of a policy (Barrett & Leddy, 2008).

Studies in diffusion, risk communication and social contagion generally list four levels of the range of impact: individual, communal, societal and global (Kasperson et al., 1988; Lundgren & McMakin, 2011; Marsden, 1998; Rogers, 2003). In prior conceptualization of range, impact is assumed to start on the individual level and branch out to the next larger level; implying a linear diffusion mechanism from small to large scale. We do not make this assumption, but acknowledge the fact that impact might diffuse between any of these layers, maybe in an iterative or reverse fashion.

Research on human perception and behavior has identified the following sequential processes through which individuals experience change: knowledge, persuasion and decision (Rogers, 2003; Slovic, Finucane, Peters, & MacGregor, 2004). Knowledge is generated when an individual is exposed to new stimuli or information and develops an understanding of them. Persuasion means that an individual forms a positive or negative opinion towards stimuli or information. Decision follows if an individual becomes engaged in activities that lead to accepting or rejecting the given inputs. There is no common agreement on how to collect data corresponding to each these stages. KAP surveys have been used for several decades to provide information on the knowledge, attitudes and practices of health behavior and innovation adoption (Launiala, 2009).

The CoMTI framework conceptualizes the phase of potential documentary impact as involving cognitive, attitudinal and behavioral factors, and suggests corresponding indices. We choose the term cognitive because the mental activities related to knowledge acquisition are mainly of cognitive nature. Persuasion denotes the intent of communicators to induce attitudinal change in a direction desired by the senders. Attitudinal is neutral in that it does not imply any directionality of change. Behavior can be distinguished from cognition and attitude in that it represents tangible changes expressed in words or activities. We do not assume a strictly sequential order of these stages and allow for interaction effects.

In explaining changes in cognition, attitude and behavior, the network concept is vital. Numerous studies have shown that perceptions, feelings and behavior initiated by one member of a network can influence other network participants (Christakis & Fowler, 2007; De Nooy, Mrvar, & Batagelj, 2011; Marsden & Friedkin, 1993; Scherer & Cho, 2003). As discussed for the Medium dimension, social media and other forms of interpersonal interaction can be more influential for cognitive and behavioral changes than mass media exposure. Furthermore, empirical reports on measuring the impact of documentaries have listed the network of viewers or alliances of advocacy organizations as a sign of increased capacity (Barrett & Leddy, 2008; Clark & Abrash, 2011; John & James, 2011). For example, the degree of connectedness of the audience can be used to gauge the degree of cohesion of members for collective action. The sheer act of forming connections to others can be part of some behavioral change.

The temporal aspect of impact is an understudied issue. Many impact studies have relied on surveys and experiments from a single point in time, or use a survey with a before/after (watching and documentary) design (Bryant & Oliver, 2008; Sparks, 2012). The CoMTI framework incorporates the temporal aspect of impact by measuring indices at multiple points in time. In summary, the CoMTI framework considers spatial, temporal and phase–related aspects of change.

Data for measuring the Impact indices can be obtained through intensive mining of unstructured and semi–structured natural language text data, e.g. from the social web. Text mining and network analysis technique need to be used to extract instances of relevant entity classes (including people, organization and information) and detecting patterned relationships between them.

In summary, the CoMTI framework bridges the gap between theory and practice by offering a mapping from clearly defined, practically relevant and theoretically grounded indicators of impact to a) crucial dependent variables, i.e. relevant dimensions of impact and b) appropriate method for capturing, representing and analyzing these signals based on real–world data.

3.2 Methodology

Based upon our review of prior work and the proposed theoretical framework, we conclude that enabling a reliable, efficient, broad and deep understanding of documentary impact requires the capturing and analysis of the web of stakeholders and content associated with (the theme of) a movie. This implies the combination of two types of techniques:

• Social network analysis, which helps to map and assess the structure, functioning and dynamics of the web of stakeholders (Borgatti, Mehra, Brass, & Labianca, 2009; Wasserman & Faust, 1994).
• Techniques from Natural Language Processing (NLP) and Text Mining, which help to identify (the valence of) salient concepts and themes originating from or shared by stakeholders (McCallum, 2005).

In order to account for these requirements, we have selected specific techniques from the abovementioned family of methods to develop the following three–step methodology for assessing documentary impact: creating and analyzing a baseline model, a ground truth model, and a change model, which ultimately get compared to each other. These steps are explained in detail below.

Baseline model: First, we map the public discourse around the main issue(s) addressed in a movie prior to a film’s initial public screenings or release. This results in a baseline model. The main purpose with this step is to understand the given ecosystem of people and themes associated with the main issue of a film. This helps to identify where impact would be possible. The main issues of a film can be identified in a data driven way, e.g. by conducting topic modeling on the film transcript, and/ or be elicited from the film maker, producer or funder. Based on our initial experience, the outcomes from both strategies do not necessarily match. We decided to go with the issues identified by subject matter experts on the film since these are the topics on which they want to motivate some change. Once these issues are identified, we use ConText (http://context.lis.illinois.edu/) (Diesner, 2014), a tool we built for this project, to a) collect social media data (currently from from Facebook, Twitter and YouTube) and b) processing media coverage that we collect from external sources, currently via e.g. LexisNexis. The same tool is then used to construct a) social networks of agents mentioned in the bodies of unstructured, natural language text data, in structured meta–data and as social media account holders, and b) (semantic networks of) salient key terms, themes and sentiments that explicitly or implicitly occur in the text data and meta–data. The resulting networks are then used as input to conducting social network analysis. This allows us to assess the structure and reasoning about the underlying dynamics of these networks, and to identify (clusters of) key individuals and organizations that are relevant with respect to different dimensions of power and influence. We also inspect the retrieved semantic networks and key terms to identify main themes and trends in the discourse. Practitioners from the film domain can utilize this step to understand the given opportunity space, which helps them to link their campaign work to relevant stakeholders and messages, supporting them in strategically allocating resources and tapping into existing social capital and public awareness. Scientifically, this model is necessary to be able to plot any change against a baseline.

Ground truth model: Second, we extract the film’s main message or themes from genuine material from the documentary, more specifically from the transcript. This helps to understand the issues that are actually addressed in the film. We apply the same text mining techniques as in step one, but to the film transcript. This results in a ground truth model, i.e. the message that a documentary can communicate. We understand that there is much more to a film than what is contained in the transcript, namely the cast, images, sound and other aesthetic elements, which are not yet considered in our methodology.

Change model: Third, we measure whether the film has moved the needle on a given issue, i.e. we try to capture the measurable impact of a film. For this purpose, we reassess the key players and public discourse related to, i.e. co–mentioned with, the film from after its release onwards. For this step, the same types of data collection and analysis techniques as in step one are being used. First, we compare these networks and salient themes to the baseline model to identify any changes in stakeholders and language use that co–occur with the mentioning of the movie. This is change associated with the film. Second, we compare the networks and terms constructed in this step to the ground–truth model. The differences between both models indicate divergences in the perception of movie by its makers versus the public. Step three can be repeated several times throughout the life–cycle of a film to monitor changes in impact.

3.3 Technology

Conducting such analyses in a scalable and robust fashion requires an automated computational solution. To facilitate the outlined methodology, we have been developing ConText (http://context.lis.illinois.edu/) (Diesner, 2014). ConText is a publicly available tool that supports a) the construction of different types of network data based on unstructured, semi–structured and structured natural language text data (Diesner J, Aleyasen, Mishra, Schecter, & Contractor, 2014) and b) the joint consideration of any such text data and network data.

ConText has a graphical user interface to ease adoption by non–technical people. We also provide sample data and training material to empower non–technical people, e.g. film makers, impact producers and funders can conduct this type of impact assessment on their own.

The application domains for ConText are not limited to impact assessment: we have designed ConText as a general applicability tool for researchers and practitioners who conduct work related to the digital humanities and computational social sciences; with the limitation being that the evaluation criteria for impact from the CoMTI framework might not apply.

3.4 Illustrative Example

This section provides an illustrative example of the proposed computational solution to documentary impact assessment. The film we look at here is the The House I Live In, a documentary by Eugene Jarecki first screened at Sundance in 2012.

Baseline model: For this assessment, the funder of the film informed us that the main issue upon which the movie aims to have an impact is “mandatory minimum sentence” (MMS). We collected the international press coverage on this topic prior to film release from LexisNexis (N=167 articles). We used ConText to parse, deduplicate and preprocess these data. This set of steps transforms raw download data into to a curated corpus and metadata database.

Figure 2 shows a semantic network generated from the meta–data of the media coverage of MMS. This network was generated in ConText by linking any two index terms per article that occur within and across user–selected entity classes—in this case “subject”—and that meet or exceed the user–specified relevance score that LexisNexis provides. The visualization was generated in Gephi (https://gephi.github.io/). This image shows that the public discourse around MMS centers on legal issues including criminal controlled substance crime, criminal offenses and justice departments.

Figure 2: Media discourse on mandatory minimum sentencing prior to movie release (semantic networks of meta–data)

Figure 3 provides a summarizing visualization of the themes emerging from the bodies (as opposed to meta–data) of these news articles. This image was generated by applying topic modeling to the data and visualizing the main words for the main topics as a word cloud. These outcomes suggest that the media frame MMS as (a) a social issue centered on people and (b) a legal issued centered on drug abuse and sentencing.

Figure 3: Baseline model: Media discourse on mandatory minimum sentencing prior to movie release (visualization of topic modeling of text bodies of media coverage)

Ground truth model: Comparing the salient themes from the media coverage (Figure 3) to those prevalent in the film’s transcript (Figure 4) shows a large common denominator: both texts (sets) portray MMS as a social issue. However, while media is more focused on prisons and violence, the film itself is more about politics related to drugs.

Figure 4: Ground-truth model: message that the documentary can convey (visualization of topic modeling of film transcript)

Change model: We assessed the media discourse on the actual movie after its release—again based on articles from LexisNexis (N = 167, selected same number of articles for comparability) that we converted into semantic networks based on the meta–data (Figure 5) and text bodies. Our results indicate that the press coverage is mainly centered on screening announcements and the director, but hardly addresses MMS—the main issue that the movie aims to have an impact on. While we as academics might consider this as a limitation, we were informed at the Sundance Producing summit where we presented our assessment of this film in 2013 that producers may aim to position a movie as a piece of art first and a communication vehicle for some issue later. This calls for a more long–term cycle of evaluation, which is supported by our methodology and technology.

Figure 5: Media discourse on The House I Live In after movie release (semantic networks of meta–data)

To capture the public reaction to the movie, we also collected and analyzed social media data using ConText and NodeXL (http://nodexl.codeplex.com/) (Hansen et al., 2010). In the following images, accounts are displayed as nodes if they have more than 200 followers, and node size and hue increase with the number of followers. Mapping followers and followees of @DrugWarMovie—the handle for The House I Live In—shows that even though the film was successful in attracting a substantial number of followers (N = 2,804), many of them are not that important or influential themselves on Twitter (small number of accounts displayed, small node size) (Figure 6). In fact, the visually represented accounts are less and smaller in size than the accounts which the film account is following, even though the film is following less accounts (1,735) than it has followers. This indicates an asymmetry between following key players (successful) and attracting key players (less successful).

Figure 6: Twitter—sphere for @DrugWarMovie

Zooming closer into the intersection of followers and followees (Figure 7) shows that most of these accounts are organizations involved with legalizing certain drugs. Only a few types of stakeholders that we consider as relevant in this content domain are involved in the public discourse on Twitter—more precisely one retired politician, two government workers, twelve small media companies and 33 NGOs.

Figure 7: Intersection of followers and followees (red = relevant types of account, purple = any other type of account)

We do not assume that all social media platforms to lead to the same (impression of) impact. Thus, we are also looking at another social networking service—Facebook: the semantic networks built from co–occurring and highly salient terms (defined in terms of TFIDF) that appear in the posts of the film’s fanpage suggests that the person posting those notes mainly addresses “watching the movie”, “release of the movie” and “war on drugs” (Figure 8). This represents classic campaign work. However, the user base (comments and replies to posts) not only picks up on these topics, but also brings new ones to the table, mainly related to the prison system and people of color. This finding suggests that it takes an engaged campaign worker to get a discussion started (missing on Twitter for this particular movie). Once this has been achieved, one possible form of impact is the public engaging with this topic and taking it into new directions. By looking at only one social media platform would not have allowed for gaining this differentiated view.

Figure 8: Co–occurrence of salient terms from posts on Facebook fan page for The House I Live In

Figure 9: Co–occurrence of salient terms from comments on Facebook fan page for The House I Live In