Skip to navigation – Site map

HomeIssues4:1Developing Survey Methods for Col...

Developing Survey Methods for Collecting Individual Policy Narratives: A case study of climate change narratives using an engaged convenience sample

Wesley Wehde and Mildred Perreault

Abstract

We examine the assumption of individuals as ‘homo narrans’ and the effect of their demographics and beliefs in shaping policy narratives, using climate change as a case. To do so we use a version of the Narrative Policy Framework codebook to analyze open-ended survey responses in a highly liberal and knowledgeable convenience sample. We use two different ways of collecting narratives in surveys. The first method relies on priming participants to consider certain narrative elements, while the other primes participants to consider the overarching narrative instead of individual elements. In line with public opinion research, we find that conservative ideology and media choices are associated with the use of fewer victims and policy problems. Finally, we find that wording a question in a way that primes overall narratives and problems produces a more reliable result in terms of complete policy narratives than do questions geared toward other narrative elements. We conclude that open-ended survey questions provide a valuable and underutilized, but limited, source of data for scholars interested in the NPF.

Top of page

Full text

Introduction

1Policy narratives are engaging, powerful, and specific accounts of how individuals and groups view the political and policy world. How interest groups create and maintain narratives that support their policy positions has been studied extensively using the Narrative Policy Framework (Gupta et. Al., 2018; Shanahan et al., 2013). Similarly, scholars of the NPF have used survey experiments to examine the effect of narrative elements on policy support in public opinion (Jones, 2014a, 2014b; Lybecker et al., 2013). However, there has been less work directly examining how individuals, themselves, create policy narratives and how those narratives may differ systematically between individuals.

2Thus, in this article, we combine these streams of NPF research and methodologies to examine individual differences in climate change narratives. We used a version of the codebook from meso-level NPF studies applied to open-ended survey responses seeking to elicit individual policy narratives about climate change. We also examined different ways of eliciting narratives and narrative elements. Our first set of survey questions focuses on methods of eliciting more complete narratives while the second set of survey questions focuses on eliciting specific narrative elements. Using these methods, we ask and answer a series of research questions: 1) How can open-ended survey questions be used to elicit complete policy narratives among an engaged sample? 2) How are demographic characteristics associated with differences in climate change narratives? 3) How are political characteristics and beliefs associated with differences in climate change narratives? 4) How is media usage associated with differences in climate change narratives?

3First, we find that problem-based survey questions and questions designed to elicit more complete narratives do, in fact, achieve this. Questions geared toward other specific narrative elements are less likely to result in complete narratives and should be used in conjunction with each other to understand complete climate change policy narratives. These findings provide direct evidence for the central NPF assumption of humans being, at an individual level, ‘homo narrans’ (Jones et al., 2014). Substantively, we draw on research on climate change beliefs to examine what factors may help to explain differences in climate change policy narratives among a sample highly knowledgeable about the causes of climate change. We find evidence that political beliefs, specifically ideology, are related to narrative choices, specifically usage of policy problems and victim characters. These findings are generally in line with work examining meso-level interest group narratives in environmental policy domains where political beliefs are often more apparent (Shanahan et al., 2013; Gupta et al., 2018).

4Our findings are also generally in line with previous research in public opinion on climate change in the US, although our sample is generally more knowledgeable than the mass public. In our highly liberal sample, more conservative respondents identify fewer problems and victims across our open-ended survey questions. We also find evidence that media choices are related to narrative element choices. Specifically, more frequent use of local newspapers is associated with identifying fewer villains. Combining existing studies, using both content analysis and survey methods with our NPF approach, we examine the role of specific media sources because of their known climate biases and find that more frequent use of Fox News network as a source is associated with identifying fewer policy problems (Boykoff & Boykoff, 2004, 2007; Carmichael & Brulle, 2018).

5In this article, we begin by reviewing the two streams of NPF research. We also review research on climate change beliefs, focusing on the role of demographics, political beliefs, and media use. We then describe our survey dataset, including the different measures we used to collect policy narratives. We then present the results from our analyses of that dataset; finally, we conclude with the implications of our research and directions for future research.

Literature Review

6Policy narratives are one way to summarize and understand how individuals and groups think about and process policy problems. These narratives generally reflect individuals’ ideologies and experiences (Stone, 2002; Jones & Song, 2014). Policy narratives are built and processed through cultural views of the world, through the individual’s reality, and through congruence or incongruence with characters in the narrative at hand (Jones & McBeth, 2010; Jones & Song, 2014). While this is a relatively common approach in other policy domains, Moezzi, Janda & Rotman (2017) argue, in a special issue on the topic, that the methods and forms of narratives and storytelling be integrated into research on climate change and energy policy. Policy narratives based on climate change may be used to analyze the complex interconnections between public policies and policy actors, as well as events, circumstances, and effects (Shanahan et al., 2018). These narratives can also be used to understand underlying ideologies and belief systems. In order to study narratives, however, one must decide how to measure them. In this article, we measure narratives through the methods of the Narrative Policy Framework which is described below.

Narrative Policy Framework

7The Narrative Policy Framework, or NPF, is a framework of the policy process which is centered on the importance of narratives. Specifically, the framework draws on qualitative and quantitative methodologies, as well as positivist and post-positivist approaches, to describe one specific way of studying policy narratives (Sievers & Jones, 2020). Scholars of the NPF have suggested that policy narratives contain at least one character and some reference to the policy domain being studied (McBeth et al., 2014). Other key narrative elements include the setting and the plot (Gray & Jones, 2016).

8Two primary streams of research have been conducted in the NPF to date. The first is the use of the NPF at the meso-level (Shanahan et al., 2013; Shanahan et al., 2018). These studies often use the NPF and its associated codebook to categorize and analyze public documents, including press releases, newspaper articles, or social media posts (Shanahan et al., 2013; McBeth et al., 2012; Gupta et al., 2018). This codebook typically includes key narrative characters such as heroes, villains, and victims, while sometimes including other categories such as beneficiaries or allies. NPF codebooks also include a moral or policy solution, policy problems, and strategies such as benefit and cost diffusion. Studies in this vein most commonly compare how narratives are built across winning vs. losing groups or coalitions in a policy subsystem (Shanahan et al., 2013; Gupta et al., 2014; Gupta et al., 2018). However, more recently, scholars have argued that the winning vs. losing dichotomy does not explain narrative choices; it is rather the case that policy preferences, regardless of coalition success, structure narratives (Gottlieb et al., 2018; Chang & Koebele, 2020).

9The other dominant stream uses survey experiments to measure the effect of varying narrative types and elements on policy preferences (Lybecker et al., 2013). Research in this vein has demonstrated that hero characters are important in understanding climate change risk perceptions and policy preferences (Jones, 2014b). A follow-up article by Jones (2014a) finds that narratives are not more persuasive than “just the facts” on climate policy preferences. However, narrative transportation is associated with positive affect toward hero characters, which his previous research finds is associated with persuasion. Related research by Shanahan, McBeth and Hathaway (2011) finds that narratives can strengthen policy preferences when congruent with previous preferences and can persuade preferences in the opposite direction when incongruent; however, this experiment only considered two different narrative treatments and had no non-narrative control.

10A more recent stream of NPF research seeks to use qualitative methods, especially interviews and focus groups, to examine how narratives are created at the individual level (Gray & Jones, 2016). Gray and Jones (2016) propose a combination of deductive NPF coding, combined with more traditional qualitative methods such as inductive grounded theory coding. Research using interviews has documented how underlying democratic values and belief systems can be observed in the narratives that people proffer when asked (Gray & Jones, 2016; McMorris et al., 2018; Smith-Walter et al., 2020).

11Most relevant to our research is a series of studies using a survey of Norwegian citizens with open-ended questions about climate change lifestyles (Tvinnereim & Fløttum, 2015; Tvinnereim et al., 2017; Langaas et al., 2019; Gjerstad & Fløttum, 2021). In the first of these studies, Tvinnereim and Fløttum (2015) find that Norwegian respondents are more likely to think of the social aspects of climate change than are their counterparts in the US (Smith & Leiserowitz, 2012). These studies also show that respondents’ images or narratives of climate change vary systematically by demographic characteristics (Tvinnereim & Fløttum, 2015; Tvinnereim et al., 2017). Finally, most recently, Gjerstad and Fløttum (2021) apply some of the coding strategies of the NPF to these open-ended survey questions. They find that first person singular pronouns are most commonly employed as a hero character, while first person plural pronouns are often cast as the villain. We follow, and extend upon, the methods of Gjerstad and Fløttum (2021) through a more comprehensive NPF coding of open-ended survey questions, resulting in a mixed-methods design. Open-ended survey questions allow for rich data collection similar to interviews but are more immediately and directly connected to other concepts of interest, measured quantitatively, such as ideology. This leads us to our first research question:

RQ1: Can open-ended survey questions be used to elicit complete policy narratives? Does question format affect the content and structure of elicited policy narratives?

Demographic and political determinants of climate change beliefs

12To examine climate change policy narratives, we review previous research on climate change beliefs more broadly so as to inform our analyses. Beginning with individual demographics and political beliefs, Hornsey, Harris, Bain and Fielding’s (2016) meta-analysis finds that demographic effects, other than age, are important but have typically small effects on climate change beliefs. On the other hand, political party and ideology have large effects on climate change beliefs. Importantly, those identifying as liberal are more likely to have stable and more certain beliefs regarding climate change, while the beliefs of those who identify as moderate or conservative are often highly varying and inconsistent (Bloodhart et al., 2015; Jenkins-Smith et al., 2020). This leads us to our second research question:

RQ2: How, if at all, are political characteristics and beliefs associated with differences in the elements present in climate change policy narratives?

13Research has drawn attention to the identities of individuals and the importance of relationships between race, gender, and climate change beliefs. Specifically, previous research has identified the white male effect, which presumes that white men are more likely to accept wide-ranging and high levels of risk compared with other adults (McCright & Dunlap, 2011). On the other hand, those who are part of more vulnerable populations, such as women and people of color, are less likely to benefit from power, control, technologies, and institutions (Flynn et al., 1994; Kahan et al. 2007; McCright & Dunlap, 2011), leading those with less power to view the world, and climate change, as more threatening and unjust. This leads us to our third research question:

RQ3: How, if at all, are demographic characteristics associated with differences in elements present in climate change policy narratives?

142.3 Media and climate change

15Finally, scholars have examined the relationship between media use and climate change beliefs (Bloodhart et al., 2015; Zhao et al., 2014). Some previous research has examined the content of local news coverage and engagement around environmental crises. Environmental concerns have been covered by journalists in the United States because they impact on communities geographically (Friedman, 2004; Bonfadelli, 2010). However, local journalists have largely failed to portray climate change in a way that demonstrates its local impacts (Rahmstof, 2012).

16Other research has examined the effect of national news on climate change beliefs. In one study, most respondents were skeptical of climate scientists after reading a collection of news articles on the topic (Leiserowitz et al., 2011). Similarly, a study done by Painter and Ashe (2012) found that newspaper articles published between 2007 and 2010 featured climate ‘skeptics’ as sources in more than a third of US and UK articles. Looking at specific sources, Feldman et al. (2012) found that viewers of Fox News and readers of the Wall Street Journal were less likely to view climate change as a problem. This may be because more conservative media sources, such as these, are less likely to refer to the scientific status of climate change as a consensus than are more liberal media sources (Boykoff & Boykoff, 2004, 2007).

17Thus, media sources work in interaction with political ideology and identity to create self-reinforcing feedback loops that create confirmation bias spirals. Ideologues choose media sources that conform to their beliefs and then those media sources provide information that strengthens those prior beliefs (Stroud, 2011; Zhao, 2009). Results from a study by Carmichael and Brulle (2018) suggest that these effects can be found for a variety of very specific media sources, ranging from the NYT and WSJ to Sean Hannity, the Colbert show, and the Huffington Post. Each of these sources is related to climate change beliefs, even after accounting for underlying political dispositions such as ideology and partisanship. This leads us to our final research question, below. We then proceed to describe the data we use to answer these research questions.

RQ4: How, if at all, are media sources associated with differences in the narrative elements present in in climate change policy narratives?

Data and Methods

18We examine individual differences in climate change narratives using an online survey distributed through a variety of channels. Respondents were recruited from multiple channels, including an email list-serv at a medium-sized (between 10,000 and 20,000 students) public university in the Southeast (a region including Alabama, Florida, Georgia, Kentucky, Maryland, Mississippi, North Carolina, South Carolina, Tennessee, Virginia, and West Virginia), environmental groups in the same region, and personal and regional social media pages on Facebook, Twitter, and Reddit. Importantly for any study of climate change or environmental policy, this region is diverse geographically, including portions of the Mississippi River delta and plains, the Appalachian Mountains, and the Gulf and Atlantic coasts. Respondents were incentivized through a drawing of ten $25 Amazon gift cards. The resulting sample is a convenience sample, n = 88. These surveys were collected between November 2020 and February 2021. Due to the convenience nature of the sample and the fact that the sample is relatively small, generalization from the data should be limited. However, considering that this is an exploratory study using primarily qualitative methods, we are not interested in broad generalization and are therefore less concerned about our sample size and characteristics.

19Reflecting the nature of a convenience sample, survey respondents are overwhelmingly female (71%), liberal (84% are slightly liberal to strongly liberal on a seven-point scale), educated (64% have a bachelor’s degree or higher), and white (90%). This is surprising given that the regions from which most respondents are recruited – the Southeast and Appalachia – are highly conservative, less educated, and relatively diverse. However, it is likely that the university context of much of the data is behind these disparities. Only the prevalence of white respondents reflects the population of the most targeted region of Appalachia, which was approximately 81.4% white as of 2017 (Pollard & Jacobsen, 2019). In addition to demographic questions, we asked respondents close-ended questions about climate change and news sources. Our sample, unsurprisingly, overwhelmingly (95%) believes that climate change is caused by human actions. Thus, we are primarily investigating the narratives of people who are knowledgeable about the causes of climate change.

Quantitative Findings

20We collected the narratives using seven separate questions, recreated below in Table 1. We coded each separate question using a version of the NPF codebook which is in the Appendix. We then analyzed the counts of each element across all seven questions due to time constraints, as well as examining differences between question styles.

Table 1: Question Wordings and Rates of Narrative Responses

Question Wording

Proportion of Responses that Qualify as a Narrative (Policy referent + Character)

Q1: Please describe what in your local environment is special to you.

0.068

Q2: Does thinking about climate change make you angry, sad, hopeful, etc? Describe your emotional responses to climate change as you think about yourself, your family, and your community.

0.50

Q3: Please describe how you imagine the future if we do not take significant action to address climate change.

0.63

Q4: Describe how you would like your community to address the climate crisis.

0.18

Q5: Describe any problem in your community you associate with climate change.

0.36

Q6: Describe who you think is being hurt by climate change in your community.

0.24

Q7: Describe who is to blame for the hurt and problems you associate with or are caused by climate change in your community

0.31

Source: The Authors

21The first four questions in Table 1 were more open-ended questions intended to elicit complete narratives, while the final three were intended to elicit specific narrative elements: policy problem, victim, and villain, respectively. As Table 1 demonstrates, Q2 and Q3 were most effective at eliciting a complete narrative with at least one character and policy referent (Shanahan et al., 2013). On the other hand, Q1 – regarding what is special in a community – was least likely to contain a complete NPF policy narrative. Instead, this question elicited responses that contained settings and not many other narrative elements. Finally, our questions designed primarily to elicit character responses resulted in policy narratives in between one quarter and one third of respondents. Difference of proportions tests suggest that the proportion of responses to Q2 and Q3 which qualify as a narrative are greater than the proportion of responses that qualify as a narrative resulting from Q5, Q6, and Q7 for all combinations at p < 0.05 (except Q2 vs. Q5, p = 0.09). In general, these results suggest, for our convenience sample, that problem-focused questions were more effective at eliciting narratives than were other types of questions. Table 1 answers both parts of RQ1 with a strong affirmative. Survey questions can elicit complex policy narratives, but the structure of those questions affects the subsequent narrative structures and the elements observed.

22For our subsequent analyses, we are interested in the prevalence of narrative elements across the various survey questions. Thus, we use the sum of each narrative element across all seven questions. Additionally, coders could identify a narrative element multiple times within a given response and therefore the total, maximum number of a particular element can exceed 7. We also included a measure of media use ranging as follows: 1 = Never, 2 = Less than once a week, 3 = About once a week, 4 = Several times a week, 5 = About once a day, 6 = Several times a day. The descriptive statistics for these variables are in Table 2.

Table 2: Descriptive Statistics of Media Use

Statistic

N

Mean

St. Dev.

Min

Max

Fox News

60

1.9

1.4

1.0

5.0

CNN

62

3.4

1.6

1.0

6.0

MSNBC

61

2.6

1.5

1.0

6.0

New York Times

62

3.8

1.4

1.0

6.0

Local Newspapers

60

3.6

1.6

1.0

6.0

Wall Street journal

61

2.4

1.3

1.0

6.0

Washington Post

60

2.9

1.6

1.0

6.0

Local TV

61

2.9

1.5

1.0

6.0

NPR

63

3.8

1.7

1.0

6.0

Source: The Authors

  • 1 Gwet’s AC (Wongpakaram et al., 2013) is a measure of inter-rater reliability that corrects for agre (...)

23As you can see in Table 2, the most used news sources are the New York Times and the NPR. Again, this reflects our highly educated, liberal-leaning sample. The least used news source is, by far, Fox News. This table also shows that we have some missing data for these measures, so the overall sample size is slightly smaller for these questions. Table 3 then presents the maximum number of each narrative element, as well as intercoder reliability measures by element1.

Table 3: Details on Coded Narrative Elements

Element

Maximum Number

Percent Agreement

Gwet’s AC

Heroes

3

97.7

0.976

Villains

7

91.6

0.882

Victims

8

92.9

0.881

Beneficiaries

2

96.3

0.961

Policy Problem

6

83.8

0.706

Time

3

94.8

0.943

Solution

4

93.0

0.907

Policy Surrogates

4

91.4

0.901

Science

2

98.4

0.984

Costs

4

88.0

0.834

Benefits

3

96.9

0.968

Source: The Authors

24Table 3 suggests that the most common elements are victims, villains, and problems. Given that these are the elements we targeted with specific questions, this is not surprising and suggests that our targeting may have worked. Table 3 also demonstrates the general patterns in individual climate change policy narratives. Climate change is, first and foremost, identified as a narrative problem, often without an accompanying solution. Thus, it is also unsurprising that climate change policy narratives focus on villain and victim characters. It is likely that these descriptive content findings are in part driven by the nature of our convenience sample, which is highly liberal and knowledgeable about climate change. However, these findings also provide evidence for the NPF model of the individual as ‘homo narrans’ in a more direct manner than do most existing studies. Additionally, when analyzing the data by element, levels of inter-rater reliability are acceptable and generally high.

25Having described our policy narratives and answered RQ1, we then answer RQ2 by examining the relationship between ideology and narrative elements. These analyses found that two out of 11 possible relationships were statistically significant at the p < 0.05 level. Ideology, from strongly liberal to strongly conservative, is negatively associated with the number of victims (Pearson’s correlation = -0.29; p = 0.020) and policy problems (Pearson’s correlation = -0.26, p = 0.042) across all seven climate change narrative questions. These correlations are visualized below in Figure 1. Correlation is signified by the solid black line, while the dashed lines represent locally estimated regressions which can reveal nonlinearities in the bivariate relationships. Finally, the boxplots along each axis represent the distribution of each variable.

Figure 1: Relationship between Ideology and Narrative Elements

Figure 1: Relationship between Ideology and Narrative Elements

Source: The Authors

26Figure 1 also demonstrates the limits of our data in terms of the most conservative ideologies for which we do not have data. Despite this, Figure 1 suggests that these negative relationships are relatively strong across the range, although less pronounced for the most liberal respondents, forming the majority of our data.

27We now turn to examining the answer to RQ3 regarding the relationship between demographics and climate change policy narratives. We find some evidence that demographics are related to differences in climate change narratives and the use of specific narrative elements. Regarding gender, we find that female respondents (mean = 0.3) use a higher number of heroes on average (Welch two sample t-test; p = 0.047) than do male respondents (mean = 0.06). On the other hand, male respondents (mean = 1.3) use policy surrogates in higher average numbers (Welch two sample t-test; p = 0.046) than do females (0.65). Regarding education, respondents with a bachelor’s degree or greater report an average of 2.8 villains across the seven questions, while respondents with less than a bachelor’s degree report only 1.6 villains across all seven narrative measures (Welch two sample t-test; p = 0.004). More educated respondents also use policy problems, solutions, and surrogates in their climate change narratives on average more frequently (means equal to 3.6, 1.5 and 1.2, respectively) relative to less-educated respondents (means equal to 2.5, 0.91 and 0.27, respectively) using a Welch two sample t-test (p-values equal to 0.014, 0.013 and 0.00017, respectively). For age, we find a positive and statistically significant correlation with the index of policy solutions (correlation = 0.29, p = 0.029) and policy surrogates (correlation = 0.45, p = 0.00062). For all other demographic and narrative element combinations, including for race where we compared white and non-white respondents, we find no statistically significant differences.

28Finally, we addressed RQ4 and examined the relationship between media use and narrative element usage. Of all possible relationships between information source usage and narrative elements, only three were statistically significant at the p = 0.05 level. Specifically, we found statistically significant correlations between the frequency of local newspaper use and the number of villains (Pearson’s correlation = -0.27; p = 0.039) and the frequency of Fox News use and the number of identified policy problems (Pearson’s correlation = -0.28; p = 0.030). We also found a statistically significant correlation (0.027; p = 0.030) between reported use of public radio, such as NPR, and the number of policy surrogates used. These relationships are visualized, in the same manner as Figure 1, in Figure 2 below.

Figure 2: Relationship between Media Use and Narrative Elements

Figure 2: Relationship between Media Use and Narrative Elements

Source: The Authors

29Given the correlation between more conservative ideology and local newspaper use (correlation = 0.25; p = 0.054), the negative relationship between local newspaper use and villains is expected. Similarly, Fox is a typical conservative news source and therefore associated with fewer identifications of policy problems related to climate change. Finally, NPR is a source more commonly used among liberal and educated individuals; thus, the positive relationship between NPR use and policy surrogates, an element connecting climate change policy to other policy areas such as health and education, does not come as a major surprise. Because of the limited sample size, we chose to examine bivariate relationships and to present only statistically significant results. Thus, we did not find significant relationships between narrative elements and most information sources, nor with most of the demographic characteristics examined – race, gender, age, and education. However, we believe that the significant relationships that we have found are interesting and consistent with existing theory and research, potentially validating the utility of our proposed method. Ultimately, our findings are suggestive, not definitive, and descriptive, not causal, and should primarily serve as motivation for future research on these topics.

Qualitative Responses

30In addition to the scaled and coded survey responses, we qualitatively analyzed the open-ended questions using inductive methods. Specifically, the researchers read through the short answers and open-ended responses using an inductive thematic approach, examining the answers individually, across questions, and as a group. Specifically, we used descriptive and process coding methods (Miles et al., 2013). These were then coded and summarized in a second cycle, which resulted in four thematic areas.

31Thematically, the participants emphasized: (1) how specific places (narrative settings) influenced their understandings of the environment and climate change; (2) how previous beliefs informed their understanding of environmental issues; (3) how specific challenges could be tackled (plot); and (4) which of these inform the way in which they think about, and share information about, climate change.

32Many participants shared personal experiences and connections to natural settings in the short-ended individual responses. These were often detailed and provided specific locations that the respondents had visited. Knowing that the survey was from a pool of participants mainly residing in the Appalachian mountain region, this was not surprising.

Connecting to locations

33When asked about what they considered special about their environment, many residents discussed their love for their region, for mountains, for outdoor spaces, such as brooks or creeks, near their homes, and for the changing seasons and wildlife (Question 1). These included potentially specific locations such as “Desoto Bridge”, but also more general places, such as “forests”, “parks”, “streams near my home”, and “mountain trails and views”. These answers help to provide the distinct and specific settings, as a narrative element, in which respondents’ climate policy narratives occur.

34For many, this concern for nature and creatures often spilled into other moral or ethical issues, such as concern for the poor and for long-term property values in coastal regions. One particular participant shared:

“I spend a lot of time hiking/camping outdoors with my children. The conservation of outdoor space is important for recreation. I also live in an area where environmental inequality is a significant problem, and I believe we have a moral/ethical obligation to address environmental racism/justice issues.” (Participant 5)

35This connection to space and place is often credited as a reason for specific policy preferences or solutions on the micro-level. Connecting climate change to poverty and income inequality was one of the most commonly present policy surrogates. These uses of policy surrogates often reflect inadvertent causal stories where respondents identify climate change as a policy problem that is both a cause and effect of income inequality and poverty, more generally. These connections to physical space and place also inform how people think of their communities and associated social aspects.

Previous beliefs and emotional responses

36In addition, several participants said that the phrase “climate change” often angered or overwhelmed them, or left them feeling that they could not make a difference (Question 2). For example, one respondent stated:

“Thinking about climate change makes me feel sad as well as angry. It's sad because the climate doesn't seem to be getting better and that's devastating for our future. It makes me angry because we are to blame for the reason the climate has changed so much and it seems like we as (a) whole aren't doing enough to change.” (Participant 17)

37Many of the responses to the question about emotion, such as this one, also identified villains and victims. In this example, we see that the respondent views people (“we”) as the villains to blame for the problem, while the victims are future generations, often referred to as our children and grandchildren.

38 Another respondent, reflecting on the importance of their general beliefs and personality as they relate to climate change, states: “I’m a generally optimistic person, but not when it comes to climate change.” (Participant 16) This same participant called for the depoliticization of the issue as a prerequisite for a policy solution. This participant’s responses provide an interesting example of ‘homo narrans’ where the individual recognizes how their self-identification does not align with their created and preferred policy narrative. These responses generally reflect the dominant narratives amongst our liberal and educated sample. However, some counter narratives were also present. One respondent stated their previous belief that “climate change is cyclical” (Participant 9). For them, climate change follows an accidental plot – they stated: “Climate change is a natural process that human activity cannot influence.” While this is just one example, other respondents stated similar claims, presenting a plot counter to that of most policy narratives provided by our engaged, liberal, and educated sample.

Solutions-based approaches

39Study participants also discussed what they thought might be policy solutions to climate concerns. Many discussed clean air and water, reductions in carbon emissions, taxes, and recycling initiatives. Some even discussed locally sourced agriculture, local government initiatives, and other approaches to climate change education:

“By protecting and restoring the forested infrastructure to improve their ability to sequester carbon” (Participant 30)

“Support for a carbon tax and dividend program or a cap and adapt policy at the federal level. The chamber of commerce needs to realize that these are necessities [...] The city hired a resiliency coordinator to plan, but more people and resources need (to be) dedicated to the issue.” (Participant 10)

“I would like to see my community decrease its single-use plastic and Styrofoam use, demand more availability of recycling programs, and look to green energy alternatives. I would also like to see more electric vehicle charging stations.” (Participant 20)

40Many participants also recognized the challenges of the local and national narratives around climate change, particularly in relationship to the political polarization associated with climate science and advocacy. While the beauty of the region and its preservation were not seen by participants as politically charged issues, taxes and the allotment of government resources were seen as a sticking point in several open-ended responses. For example, one participant discussed their anger at government and corporations not acting (Participant 23), while others emphasized personal responsibility (Participant 26).

Sharing and action

41People who participated in the study also discussed the importance of how their beliefs and actions could affect the perspectives of others. For policies to change, respondents noted that people had to share their experiences and change how people conceptualized climate change:

“[...] change starts with the individual, but due to social norms, local policies, and larger state and national policies, change has been stagnant” (Participant 25)

“I think it's a shared responsibility. Individuals need to do better, but corporations will continue to produce as long as there is demand and profit to be made.” (Participant 26)

42The use of terms such as “social norms” and “shared responsibility” reflect what Stone (1989) calls historical or structural causes. These respondents identify the ways in which norms reproduce themselves and their outcomes and are therefore difficult to change. When referring to the importance of corporations, these respondents may also be invoking institutional causes in their policy narratives, following Stone’s distinction between institutional and structural causes. These responses, and others like them, center on the power structures that lead to status quo bias and narratives thereof.

43Finally, with this study taking place during the COVID-19 pandemic, responses sometimes included comments concerning the COVID-19 crisis and how it related to climate change. An example of this is the mix of emotions that comes from being hopeful that the pandemic will bring more attention to some systemic issues regarding science and policy (in response to Question 2), but there is also concern that it will overshadow climate conversations:

“It [climate change] is horrifying because, much like the COVID-19 pandemic, there is much we could do if we just cared for our global world and other people more than our immediate personal interests.” (Participant 71)

44In conclusion, the approach that many participants took to the survey shows their genuine interest in unpacking the issues in their regions dealing with climate change, while at the same time not neglecting the role that they can play. The emphasis on individual responsibility, commonly associated with more conservative ideologies, may reflect the more individualistic nature of the Appalachian region in which most participants reside. It also reflects a preference for what Stone (1989) calls intentional causal stories, and the importance of this plot type, as identified by Soremi (2019), because these narratives emphasize a set of purposeful actions with intended consequences. However, despite this emphasis – as demonstrated above – our respondents also frequently discussed the need for change from politicians and corporations, who were often cast as the villains in the narrative. These narratives, on the other hand, rely on inadvertent causes in their plot – respondents identify that those with power take purposeful action that often has unintended consequences for climate change. Reflecting the overall prevalence, or lack thereof, of heroes in our data, most of these example narratives do not have clear hero characters. This is especially interesting given the importance of heroes in survey experimental research on climate change narratives (Jones, 2014b). This may be a result of the highly engaged, liberal sample which is more likely to view climate change as a problem.

Discussion

45Methodologically, our findings suggest that the way in which researchers choose to elicit narratives matters. Specifically, we find that more complex questions are more likely to result in complete policy narratives with a policy referent and character (Shanahan et al., 2013). Additionally, questions that center on the problem element are also more effective at eliciting complete policy narratives. However, questions which focus on specific narrative elements can also be used to elicit complete policy narratives, when used in combination.

46Substantively, we find that there are important individual differences in how climate change narratives are created. These substantive differences provide one form of validation of our open-ended survey technique. Instead of prompting respondents to choose between pre-selected narratives (Kirkpatrick & Stoutenborough, 2018), we used open-ended survey questions to elicit their narrative choices. Examining systematic narrative differences in narratives based on demographics, we find that men – in line with previous research – are more likely than women to externalize climate change solutions (Tvinnereim et al., 2017). Interestingly, our findings regarding age and education differ somewhat from those found in the Norwegian Citizen Panel Data. We find that age is positively associated with a focus on climate solutions, as opposed to causes (Tvinnereim & Fløttum, 2015). Additionally, while education is not often strongly associated with different images or narratives in previous studies (Tvinnereim & Fløttum, 2015; Tvinnereiam et al., 2017), we find education to be – strongly and statistically – significantly associated with narrative differences for more elements than other demographics. One possible explanation for these differences may be the different institutional structures governing climate policy in the US and Norway. However, this does not entirely explain why our findings regarding gender are similar but other demographic findings are different. Thus, different relationships between demographic factors and political ideologies in these countries may also contribute to differences in the findings. Future research should use comparative methods to examine the importance of institutional contexts in shaping individual policy narratives and the assumptions of ‘homo narrans’.

47Regarding political beliefs and information use, the individual differences observed are theoretically predictable with respondents who are more ideologically conservative referencing fewer policy problems and victims of climate change than more ideologically liberal individuals (McCright & Dunlap, 2011). We find similar patterns for media use, with users of more conservative sources referencing fewer policy problems and villains (Feldman et al., 2012). We believe that these findings provide feedback suggesting that our approach to collecting individual narratives provides indications of face validity.

48Our qualitative analyses suggest that these elicited climate narratives are also complex. These responses were often detailed and thoughtful in their nature, eliciting connections between their individual responsibility to the local environment while recognizing that climate change must be addressed by a combination of individual – as well as structural and societal – change. One potential reason for the complexity of climate change narratives may be the fact that climate change is what some have called a long-duration crisis (DeLeo et al., 2021). Similar to previous qualitative research using the NPF, we find that our climate change narratives reflect underlying belief systems about democracy and democratic norms (Gray & Jones, 2016; Smith-Walter et al., 2020) While our responses cannot match the depth of data provided by interviews, we believe that open-ended survey questions may be appropriate for collecting narratives with highly engaged samples. Doing so will save the researcher time as interviews will not need to be recorded and transcribed and data will already be in text format. Additionally, open-ended survey responses are directly attached to “meta-data” of interest when examining how individual differences are associated with narrative differences. We believe that this method may prove useful to NPF researchers interested in combining the qualitative and quantitative traditions of the framework.

49However, our study is subject to a number of key limitations. First, we have a highly engaged, highly biased, and small non-representative sample. Therefore, all associations that we present are bivariate and suggestive in nature. Our findings are certainly not conclusive and we hope that future research, using larger and more representative samples, can replicate our findings. To do so will require significant investment of resources as narrative coding is still a time- and labor-intensive process. One potential threat to studies of the sort suggested is the lower level of attention often found in common representative samples. Therefore, while we are unable to generalize broadly from our findings, we do believe that there are benefits to our sample resulting from the highly engaged nature of respondents. The sample also provides a different perspective than most studies concerning climate change knowledge. A related path of research would be to attempt to replicate our findings using other highly engaged samples.

50Second, our study focuses on climate change narratives. Climate change is a highly polarized issue in the United States, and our sample is highly engaged with the climate issue. Respondents in our sample have much higher levels of belief in the anthropogenic causes of climate change than does the general population. Individual narratives on other policy issues may not result in similar levels of individual differences. Thus, future research on climate change narratives may seek out the narratives of a small, highly engaged sample that is less likely to hold these beliefs. Additionally, future research on policy narratives more broadly ought to consider adapting survey measures such as those presented here and applying these methods to a wide variety of policy domains. Future research can examine if similar patterns of individual narrative differences in policy narratives emerge for other issues. These studies should focus both on other highly polarized issues, as well as on less controversial issues.

Conclusion

51While primarily exploratory in nature, our research suggests that open-ended survey questions are able to elicit policy narratives. Our data also suggest that survey question wording matters for eliciting narratives, with questions that focus on plots or problems being more likely to result in full policy narrative responses. Among a liberal and engaged sample, we find that climate policy narratives are overall often focused on villains, victims, and problems. We expect that the average climate policy narrative would differ greatly in a more conservative, or representative, sample.

52For our engaged sample, policy narratives were often complex and provided interesting insights into how the respondents structured their beliefs about climate change. Our quantitative results suggest that climate change narratives do differ according to demographics, political beliefs, and media use. These findings, especially those regarding ideology and media use, are in line with existing research and theory on climate change policy preferences and thus act as a sort of validity check on our method (Hornsey et al., 2016; Carmichael & Brulle, 2018). Our qualitative results suggest that key themes in climate change narratives for our sample include connection to place, desire and reluctance to take individual action, and preference for societal and structural plots (Stone, 1989).

53The NPF relies on the assumption that the individual is a ‘homo narrans’. Existing research provides evidence for related assertions by documenting policy narratives in organizational communications (ultimately created by groups of individuals). Others have observed that individuals process narratives provided to them in straightforward and predictable ways, suggesting again the ‘homo narrans’ model. However, we argue that open-ended survey questions, especially those priming policy problems and narrative structure, provide one relatively simple and straightforward way to test this assumption directly. We find that, when asked through online surveys, humans can produce policy narratives and do so in ways that are shaped systematically and predictably by factors including lived experiences, demographics, political beliefs, and their exposure to others’ narratives through media.

Top of page

Bibliography

Bloodhart, B., Maibach, E., Myers, T., & Zhao, X. (2015). Local climate experts: The influence of local TV weather information on climate change perceptions. PLoS One, 10(11).
https://doi.org/10.1371/journal.pone.0141526, e0141526.

Bonfadelli, H. (2010). Environmental sustainability as challenge for media and journalism. In M. Gross & H. Heinrichs (Eds.), Environmental sociology (pp. 257-278). Springer., Dordrecht.

Boykoff, M. T., & Boykoff, J. M. (2004). Balance as bias: Global warming and the US prestige press. Global environmental change, 14(2), 125-136.

Boykoff, M. T., & Boykoff, J. M. (2007). Climate change and journalistic norms: A case-study of US mass-media coverage. Geoforum, 38(6), 1190-1204.

Carmichael, J. T., & Brulle, R. J. (2018). Media use and climate change concern. International Journal of Media & Cultural Politics, 14(2), 243-253.

Chang, K. T., & Koebele, E. A. (2020). What drives coalitions' narrative strategy? Exploring policy narratives around school choice. Politics & Policy48(4), 618-657.

DeLeo, R. A., Taylor, K., Crow, D. A., & Birkland, T. A. (2021). During Disaster: Refining the Concept of Focusing Events to Better Explain Long-Duration Crises. International Review of Public Policy3(1).

Feldman, L., Maibach, E. W., Roser-Renouf, C., & Leiserowitz, A. (2012). Climate on cable: The nature and impact of global warming coverage on Fox News, CNN, and MSNBC. The International Journal of Press/Politics, 17(1), 3-31.

Flynn, J., Slovic, P., & Mertz, C. K. (1994). Gender, race, and perception of environmental health risks. Risk analysis, 14(6), 1101-1108.

Friedman, S. (2004). And the beat goes on: The third decade of environmental journalism. The environmental communication yearbook, 1, 175-187.

Gjerstad, Ø., & Fløttum, K. (2021). Climate change lifestyle narratives among Norwegian citizens: A linguistic analysis of survey discourse. European Policy Analysis.

Gottlieb, M., Bertone Oehninger, E., & Arnold, G. (2018). “No fracking way” vs.“drill baby drill”: A restructuring of who is pitted against whom in the narrative policy framework. Policy Studies Journal, 46(4), 798-827.

Gray, G., & Jones, M. D. (2016). A qualitative narrative policy framework? Examining the policy narratives of US campaign finance regulatory reform. Public Policy and Administration, 31(3), 193-220.

Gupta, K., Ripberger, J. T., & Collins, S. (2014). The strategic use of policy narratives: Jaitapur and the politics of siting a nuclear power plant in India. In M. D. Jones, E. A . Shanahan, & M. K. McBeth (Eds.), The science of stories: Applications of the Narrative Policy Framework in public policy (pp. 89-106). Palgrave Macmillan.

Gupta, K., Ripberger, J., & Wehde, W. (2018). Advocacy group messaging on social media: Using the narrative policy framework to study Twitter messages about nuclear energy policy in the United States. Policy Studies Journal, 46(1), 119-136.

Hornsey, M. J., Harris, E. A., Bain, P. G., & Fielding, K. S. (2016). Meta-analyses of the determinants and outcomes of belief in climate change. Nature climate change, 6(6), 622-626.

Howarth, A. (2012). Participatory politics, environmental journalism and newspaper campaigns. Journalism studies, 13(2), 210-225.

Jenkins-Smith, H. C., Ripberger, J. T., Silva, C. L., Carlson, D. E., Gupta, K., Carlson, N., ... & Dunlap, R. E. (2020). Partisan asymmetry in temporal stability of climate change beliefs. Nature Climate Change, 10(4), 322-328.

Jones, M. D. (2014a). Communicating climate change: Are stories better than “just the facts”?. Policy Studies Journal, 42(4), 644-673.

Jones, M. D. (2014b). Cultural characters and climate change: How heroes shape our perception of climate science. Social Science Quarterly, 95(1), 1-39.

Jones, M. D., McBeth, M. K., & Shanahan, E. A. (2014). Introducing the narrative policy framework. In The science of stories (pp. 1-25). Palgrave Macmillan, New York.

Jones, M. D., & McBeth, M. K. (2010). A narrative policy framework: Clear enough to be wrong?. Policy Studies Journal, 38(2), 329-353.

Jones, M. D., & Song, G. (2014). Making sense of climate change: How story frames shape cognition. Political Psychology, 35(4), 447-476.

Kahan, D. M., Braman, D., Gastil, J., Slovic, P., & Mertz, C. K. (2007). Culture and identity‐protective cognition: Explaining the white‐male effect in risk perception. Journal of Empirical Legal Studies, 4(3), 465-505.

Kirkpatrick, K. J., & Stoutenborough, J. W. (2018). Strategy, narratives, and reading the public: Developing a micro‐level theory of political strategies within the narrative policy framework. Policy Studies Journal, 46(4), 949-977.

Langaas, R. F., Fløttum, K., & Gjerstad, Ø. (2019). Expressing one’s conceptions of lifestyle in a climate perspective. Cahiers de praxématique, 73.

Leiserowitz, A., Maibach, E., Roser-Renouf, C., & Smith, N. (2011). Global warming’s six Americas in, May 2011. Yale University and George Mason University.

Lybecker, D. L., McBeth, M. K., & Kusko, E. (2013). Trash or treasure: recycling narratives and reducing political polarisation. Environmental Politics, 22(2), 312-332.

McBeth, M. K., Jones, M. D., & Shanahan, E. A. (2014). The narrative policy framework. Theories of the policy process, 3, 225-266.

McBeth, M. K., Shanahan, E. A., Arrandale Anderson, M. C., & Rose, B. (2012). Policy story or gory story? Narrative policy framework analysis of Buffalo field campaign's YouTube videos. Policy & Internet, 4(3-4), 159-183.

McCright, A. M., & Dunlap, R. E. (2011). Cool dudes: The denial of climate change among conservative white males in the United States. Global environmental change, 21(4), 1163-1172.

McMorris, C., Zanocco, C., & Jones, M. (2018). Policy Narratives and Policy Outcomes: An NPF Examination of Oregon's Ballot Measure 97. Policy Studies Journal, 46(4), 771-797.

Miles, M. B., Huberman, A. M., & Saldana, J. (2014). Qualitative data analysis: A methods sourcebook. Sage.

Moezzi, M., Janda, K. B., & Rotmann, S. (2017). Using stories, narratives, and storytelling in energy and climate change research. Energy Research & Social Science, 31, 1-10.

Painter, J & Ashe, T. (2012) Cross-national comparison of the presence of climate scepticism in the print media in six countries, 2007–10 Environ. Res. Lett. 7 044005

Pollard, K., & Jacobsen, L. A. (2019). The Appalachian Region: A Data Overview from the 2013-2017 American Community Survey. Chartbook. Appalachian Regional Commission.

Rahmstorf, S. (2012). Sea-level rise: towards understanding local vulnerability. Environmental Research Letters, 7(2).
https://doi.org/10.1088/1748-9326/7/2/021001, 021001.

Shanahan, E. A., Jones, M. D., & McBeth, M. K. (2018). How to conduct a Narrative Policy Framework study. The Social Science Journal, 55(3), 332-345.

Shanahan, E. A., Jones, M. D., McBeth, M. K., & Lane, R. R. (2013). An angel on the wind: How heroic policy narratives shape policy realities. Policy Studies Journal, 41(3), 453-483.

Sievers, T., & Jones, M. D. (2020). Can power be made an empirically viable concept in policy process theory? Exploring the power potential of the Narrative Policy Framework. International Review of Public Policy2(1), 90-114.

Smith, N. and Leiserowitz, A. (2012). The rise of global warming skepticism: Exploring affective image associations in the United States over time. Risk Analysis: An International Journal32(6), 1021-1032.

Smith-Walter, A., Jones, M. D., Shanahan, E. A., & Peterson, H. (2020). The stories groups tell: campaign finance reform and the narrative networks of cultural cognition. Quality & Quantity, 54(2), 645-684.

Soremi, T. (2019). Storytelling and Policy Transfer. The Case of Disaster Risk Reduction Policy Transfer to West Africa. International Review of Public Policy1(2), 194-217.

Stone, D.A. (1989). Causal Stories and the Formation of Policy Agendas. Political Science Quarterly, 104(2), 281-300.

Stone, D. A. (2002). Policy paradox: The art of political decision making (Vol. 13).WW Norton.

Stroud, N.J. (2011). Niche news: The politics of news choice. Oxford University Press.

Tvinnereim, E., & Fløttum, K. (2015). Explaining topic prevalence in answers to open-ended survey questions about climate change. Nature Climate Change5(8), 744-747.

Tvinnereim, E., Fløttum, K., Gjerstad, Ø., Johannesson, M. P., & Nordø, Å. D. (2017). Citizens’ preferences for tackling climate change. Quantitative and qualitative analyses of their freely formulated solutions. Global Environmental Change46, 34-41.

Wongpakaran, N., Wongpakaran, T., Wedding, D. et al. (2013). A comparison of Cohen’s Kappa and Gwet’s AC1 when calculating inter-rater reliability coefficients: a study conducted with personality disorder samples. BMC Medical Research Methodology, 13(61).
https://doi.org/10.1186/1471-2288-13-61

Zhao, X. (2009). Media use and global warming perceptions: A snapshot of the reinforcing spirals. Communication Research, 36(5), 698-723.

Zhao, X., Maibach, E., Gandy, J., Witte, J., Cullen, H., Klinger, B. A., ... & Pyle, A. (2014). Climate change education through TV weathercasts: Results of a field experiment. Bulletin of the American Meteorological Society, 95(1), 117-130.

Top of page

Notes

1 Gwet’s AC (Wongpakaram et al., 2013) is a measure of inter-rater reliability that corrects for agreement by chance and is more stable, relative to the more commonly used Cohen’s Kappa, when prevalence of an element is low. Additionally, we calculated intercoder reliability by element as our quantitative analyses use each type of narrative element, across all questions, as the unit of analysis.

Top of page

List of illustrations

Title Figure 1: Relationship between Ideology and Narrative Elements
Credits Source: The Authors
URL http://journals.openedition.org/irpp/docannexe/image/2480/img-1.png
File image/png, 49k
Title Figure 2: Relationship between Media Use and Narrative Elements
Credits Source: The Authors
URL http://journals.openedition.org/irpp/docannexe/image/2480/img-2.png
File image/png, 54k
Top of page

References

Electronic reference

Wesley Wehde and Mildred Perreault, Developing Survey Methods for Collecting Individual Policy Narratives: A case study of climate change narratives using an engaged convenience sampleInternational Review of Public Policy [Online], 4:1 | 2022, Online since 30 June 2022, connection on 16 August 2022. URL: http://journals.openedition.org/irpp/2480; DOI: https://doi.org/10.4000/irpp.2480

Top of page

About the authors

Wesley Wehde

Texas Tech University Department of Political Science
wwehde@ttu.edu

Mildred Perreault

East Tennessee State University
perreault@etsu.edu

Top of page

Copyright

CC BY 4.0

Creative Commons - Attribution 4.0 International - CC BY 4.0

https://creativecommons.org/licenses/by/4.0/

Top of page
  • Logo IPPA – International Public Policy Association
  • DOAJ - Directory of Open Access Journals
  • Journal supported by the Institut des Sciences Humaines et Sociales (CNRS)
    CNRS - Institut national des sciences humaines et sociales
  • OpenEdition Journals
Search OpenEdition Search

You will be redirected to OpenEdition Search