- 1 This work was supported by the the European Union’s Horizon 2020 Research & Innovation Action under (...)
1Research possibilities utilising text data are expanding (Baden et al., 2022), and political text produced by citizens is among the most commonly employed text data within political and communication science. We1 understand citizen-produced political text (CPPT) broadly as a text produced by citizens, offline or online, in a written or retranscripted form, relating to a political process, policy or civic issues (Gelovani et al., 2022). Research questions addressed through such data are diverse, ranging from the analyses of millions of search engine queries of netizens relating to anti-corruption campaigns in China (Zhu and Wang, 2020) to more qualitative examinations of two dozen in-depth interviews with leftist activists in Texas (USA) to understand their internal debates when disagreeing (Venegas, 2022). CPPT studies allow to focus on citizens’ views, opinions, and experiences rather than elites’ and provide invaluable insights into political communication research.
2Despite the richness and importance, utilising CPPT presents several interconnected challenges. Research tools and methodologies, developed primarily for English-language text, have limited applicability to non-English text (Baden et al., 2022). Researchers outside non-Western regions struggle with collaboration opportunities (Matassi and Boczkowski, 2021). Access to comprehensive social media data – the most common CPPT data source – is a privilege of a few research teams collaborating with social media companies (Lazer et al., 2020). This paper aims to assess the challenges and inequalities faced by those interested in employing CPPT, regardless of the method or source of data. It identifies challenges researchers experience from accessing data to publications and dissemination, but it excludes ethical challenges that have been extensively explored elsewhere (Sormanen and Lauk, 2016). We approach the problem from a generalisable and globalised perspective to ensure a cross-national understanding.
3Studying citizens’ use of political text is essential to understanding their political positions, participation and agenda-setting. For example, Sebastian Stier and colleagues (2018) identified topics citizens discussed in open-ended responses to a survey during the 2013 German federal election to assess how well candidates reflected these issues. Commenting on news articles (Erzikova and Simpson, 2017), microblogging (Fu and Chau, 2014) or replying to politicians’ tweets (Londoño, 2019) are means for citizens to engage in political conversations. Such forms of primarily social media participation are becoming more commonplace and growing in variety (Vaccari and Valeriani, 2021).
4Given the range of substantive questions addressed through studying text produced by the citizens and the diversity of disciplines involved (Giglietto et al., 2012), the field employs a variety of methodological approaches. Not least in volume – studies using Twitter range from critical discourse analysis of 100 tweets (Kreis, 2017) to social network analysis of 20 million tweets (Deltell et al., 2013). Furthermore, a comprehensive review of more than three thousand manuscripts studying CPPT (Gelovani et al., 2022) indicates that half employ qualitative methods, and a quarter utilise computational or quantitative methods. During the period from 2014 to 2022, social media platforms emerged as the primary, but not exclusive, source of data in these studies.
5The utilisation of big or digital data and computational methods in social sciences have already prompted reflections on how such methods can be applied by social scientists (e. g. boyd and Crawford, 2012; Van Atteveldt and Peng, 2018; Theocharis and Jungherr, 2021; Baden et al., 2022). There is a growing need for comparative studies across platforms, countries, and time (Larsson, 2015). It is also crucial for researchers to enhance their understanding of how platform policy and algorithm modifications, as well as users’ engagement patterns of these platforms, might impact the research approach and outcomes (Karpf, 2020).
6Researchers have identified a number of impediments within the research on CPPT. One such challenge pertains to data accessibility, especially from social media platforms, particularly in the aftermath of the Cambridge Analytica controversy, also called the “APIcalypse” (Bruns, 2019). Facebook drastically limited access to platform data via their Application Programming Interfaces (APIs) and Twitter closed completely free-accessed data in mid-2023. Furthermore, disparities emerge among researchers themselves. Significant variations occur in how data is utilised by those engaged, for example, in ethnographic research in comparison to computational research, despite using the same sources of CPPT data (Giglietto et al., 2012). These differences are often reflected in a lack of reciprocal understanding and appreciation for each other’s work among researchers. Additionally, biases have been identified, where priorities are given to well-established subjects, Western research sites and English-language materials (Baden et al., 2022). Bias in research also emerges because the “Global North” perspective dominates, crowding out other perspectives not seen as the norm (Wasserman, 2020). To gain a deeper understanding of and address the challenges encountered by researchers interested in studying CPPT data, we ask:
-
What challenges confront CPPT researchers or researchers interested in using CPPT data relating to the themes of accessibility, languages and resources?
-
In what way do these challenges vary across researchers due to career stage, country of affiliation, language of interest, or other personal factors?
7The manuscript builds upon two methodological approaches. First, we run a non-representative survey among researchers from diverse disciplines, mainly political science, communication, methods, or linguistic studies. Subsequently, we perform a series of in-depth interviews with a purposefully selected sample of respondents. The repartitions and characteristics of respondents can be found in Tables A1 and A2 of the Supplementary materials.
8The first data source employed in this study is an online survey on a non-representative but strategically recruited sample of researchers actively engaged in text-based research or being interested in working with text, especially CPPT data. The questionnaire asked them about their background, use of text data, challenges faced when using such data, etc. Respondents were recruited through a multifaceted approach, including (i) lists of contacts of authors employing CPPT data in their research identified via extensive literature review (Gelovani et al., 2022); (ii) contacts within the network of OPTED project; (iii) mailing lists of organisers of text analysis related events; (iv) online advertisement by OPTED. The survey received ethics approval from the Research Ethics committee of the University of Exeter. A total of 295 responses were collected from February to August 2022 (Table A1). Respondents could skip questions (total N per question is specified in the results section) or provide additional answers in the text box. Based on their reported usage of CPPT, respondents were categorised into three groups: those who do not use CPPT (N=73; not employed in this manuscript), those interested in using it in future (N=59), and those currently using CPPT (N=163).
- 2 Interviewee citations are edited for language correction.
9A second source of data consists of expert interviews. A purposeful sampling strategy (Palinkas et al., 2015; Patton, 2015) was employed to identify relevant experts and CPPT researchers. Our goal was to achieve a diverse sample encompassing for discipline, methods, language(s) studied, language of publications, career stage, regional variation in institutional affiliation, and country of origin. Additionally, we looked for respondents with experience in CPPT research gained through journal editing, collaboration with data companies, or involvement in research integrity and ethics. We contacted authors who had published CPPT studies and experts identified via workshops, conferences or courses relating to text analysis, or editors and ethical review board members. Twenty-one interviews were conducted in May-July 2022 (Table A2). Interviewees were assured of their anonymity, asked for permission to audio-record and transcribe interviews, and informed of the interview’s purpose in a consent form signed beforehand. A semi-structured questionnaire was developed, with core questions for all interviewees and specific questions tailored to each individual. Interviews ranged from 35 to 90 minutes and were conducted online. Ethical approval for the study was received from the Research Ethics committee of Audencia Business School. Interviewees are given a random identification number when referenced2. The results of the survey were helpful in preparing and conducting the interviews. As the survey was anonymous, the in-depth interviews were not conducted as a direct follow-up. The two datasets are interrelated by the researchers’ shared interest.
10Survey respondents were asked about the challenges they encountered when accessing CPPT data. Table 1 indicates the most significant access-related issues perceived as difficulties by these users.
Table 1. Access issues for survey respondents by CPPT use (N=201).
Challenge |
Interested (N=56) |
Users (N=146) |
Company restriction |
44 |
135 |
Content removal |
34 |
112 |
Finding tools |
33 |
95 |
Finding text |
33 |
91 |
Ethical restrictions |
24 |
80 |
11Social media companies implement access restrictions through various means. Firstly, they do not allow researchers to access all data comprehensively. This is also confirmed in the in-depth interviews, as restrictions by companies often leave interviewees with what they consider suboptimal and less representative data sources. Consequently, the potentially more valuable data remains inaccessible.
- 3 Interviews were kept before the restriction to data access on Twitter introduced in Summer 2023.
“Twitter3 is very infrequently the best solution for a problem. It is just usually the easiest solution for a problem.” (Interviewee_5, OPTED, interview with Authors July 2022).
12The further development of API web services is deemed crucial for 68% of survey respondents. Specifically:
“With Twitter’ API’s, there’s a limit for how many inquiries you can make. And also there’s a limit for how far you can go back. […] you get wildly different results if you do research today, and then three years ago, and you have no ways to verify where [the difference] is coming from” (Interviewee_8, OPTED, interview with Authors June 2022).
13Secondly, the evolving (not in a favourable direction) nature of the restrictions in data access by social media corporations presents a significant challenge. Not only does this restrict what interviewees can currently study because of data becoming unavailable, but it also makes it impossible to access historical data or access only part of a sample of data.
“Constantly what you can scrape, what you cannot scrape, changes and – may I say – with zero regard to researchers. Like they do not consider researchers when putting through these guidelines. They consider industry people.” (Interviewee_4, OPTED, interview with Authors June 2022).
“When I started you could download everything. Everything was scrapable […] but now we can’t have access to the data, so we have to orientate the research with stuff that we can do” (Interviewee_16, OPTED, interview with Authors May 2022).
14Interviewees are concerned about how access will change in the future and impact ongoing projects. Such uncertainty and the power imbalance between researchers and social media corporations lead to high levels of distrust:
“Facebook used to be open. It isn’t anymore. You can get to it through a service called CrowdTangle, which is owned by Meta. But a lot of the people who worked on CrowdTangle have since been placed in other services within Meta, or some have even quit. So we don’t really know what’s going to happen with CrowdTangle” (Interviewee_6, OPTED, interview with Authors June 2022).
- 4 A fear that turned out to be well-founded, as announced in April 2023, available here: https://csma (...)
“what we’ve seen, Twitter opening up the API for researchers a few months ago, I think that was a very big thing. I’m not necessarily sure how long that stays, because, well, it’s always at the whim of the company.” (Interviewee_14, OPTED, interview with Authors May 2022)4
15Thirdly, skills and time requirements to access data continue to be high barriers for entry. Several interviewees expressed frustration that the data-gathering process has become more strenuous than necessary due to how companies (re)structure APIs and the lack of information provided.
16Content removal creates two primary challenges for researchers: it complicates the replicability of analyses and limits the ability to study communication that is most likely to be deleted, such as hate speech, conspiracy theories, anti-government rhetoric, etc. Interviewees experience that posts are removed, made private or edited by the person/page who created the post or because the user deleted the account. Additionally, content can be removed when a page, user, or group is blocked. In certain cases, systemic content removal occurs when, for example, powerful actors censor online discussion on a specific topic.
17Regardless of the underlying reasons for content removal, there is a shared frustration among researchers regarding its impact on the ability to study certain phenomena where deletion is likely.
“Facebook pulls everything that it finds offensive. Then you have the user review process that might pull the next thing and then people pull their own things or edit them later. And yes, I know that you can technically get to the pre-edit version. But let’s face it, who does that?” (Interviewee_5, OPTED, interview with Authors July 2022).
“I cannot possibly be on top of everything at all times. Things may get deleted, I cannot check many thousands of tweets whether or not they got deleted constantly.” (Interviewee_4, OPTED, interview with Authors June 2022).
- 5 As multiple interviewees explain, it is not possible to retroactively access deleted posts using st (...)
- 6 Prior to Twitter closing down the public access to their API in 2023, several tweet scraper tools d (...)
18The techniques vary among researchers when dealing with deleted social media posts. Some believe that as long as they only analyse posts that were available at the time of data collection5 and anonymise them, they are in line with the research standards. Indeed, depending on the source of data, this method can sometimes be the only possible solution when using large datasets. Other interviewees explain the process of “re-hydrating”6 social media posts: during data collection, posts’ IDs, rather than the text, should be archived, and then these IDs can be retrieved within future analysis, ensuring that only currently available posts are included. Yet, even interviewees who are familiar with re-hydrating state that they currently do not engage in this practice or are not sure how it works. The inability to provide complete datasets for replication is a systemic issue, as evidenced in the meta-study, which indicates that only 1.3% of studies provide access to datasets (Gelovani et al., 2022).
19Regarding tools utilised for accessing CPPT, a primary concern revolves around substantial time commitment to master the usage of software, codes, and applications. Moreover, due to the relative novelty of many of those techniques, it is difficult to find sufficient guidelines, often resulting in mistakes that lengthen data collection. One interviewee indicated such constraints:
“I was crawling their website and my crawler was getting that data at a very huge speed, […] so when I was crawling this data the guys from this company reached out to [interviewee’s university] and they blocked my IP for a week or so. And they were like ‘OK. You’re not going to do anymore crawling because they’re very mad at us’. What were we doing? We’re just jamming their servers!” (Interviewee_18, OPTED, interview with Authors July 2022).
- 7 Netvizz, a now defunct Facebook app created by Bernhard Rieder, University of Amsterdam, allowed re (...)
20A persistent challenge lies in the misusage of tools designed for other purposes (such as marketing research) that often do not result in the type of data needed for the CPPT researcher in social sciences. One interviewee expressed a feeling of greater trust in tools developed by discipline-specific researchers, such as Netvizz7. Moreover, tools trained on other text types often perform worse when applied to CPPT-related tasks.
“I think given the structure of this text, it’s not that straightforward. Some of the models cannot be applied or perform very differently with these short texts.” (Interviewee_21, OPTED, interview with Authors June 2022).
21Despite the availability of tools that align with researchers’ needs, their visibility is often limited. Researchers frequently rely on word-of-mouth to learn about the existing tools rather than having a reliable place to browse alternatives. This lack of awareness results in sometimes choosing suboptimal methods for the question at hand.
22The survey findings (Table 2) indicate that researchers were more likely to study text in multiple languages, particularly when English was included.
Table 2. Languages studied (survey respondents N=196).
Languages |
Interested (N=55) |
Users (N=141) |
Multiple incl. English |
43 |
105 |
Only English |
2 |
22 |
Only non-English |
5 |
10 |
Multiple not incl. English |
5 |
4 |
23The ability to access text was a major factor (42%) among survey respondents for choosing which languages to study. Also, the interviewees indicate that access to text varies across languages and regions. The reasons for that are diversified, for example, in certain countries easy-to-study platforms have greater usership or specific regional restrictions exist.
“I see that there are some papers coming out like using Twitter data and different kinds of other forums where they have analysed like millions of tweets […] In the [small country in Europe] context, to find enough data to do it well? I don’t think it’s possible” (Interviewee_3, OPTED, interview with Authors May 2022)
“The problem is that in the North African context, there are no archives. […] the web pages are not stable. And that applies to citizen-produced text, which generally has less durability than print media or audio-visual materials.” (Interviewee_2, OPTED, interview with Authors July 2022).
24Social media platforms are also used differently across communities. For specific topics, open discussions may be prevalent on some platforms rather than on others due to populations inhabiting it or restrictions in content supervision:
“People from my own country, [country in Asia] and in my part of the world. They would not be that vocal on social media […] even if they are vocal, they won’t talk much about these topics that we are interested in” (Interviewee_18, OPTED, interview with Authors July 2022).
25Regional differences in methodological training was also a theme discussed by interviewees:
“We still have a problem here in [country in South America], working especially in human sciences, in working with big data. We are not trained to do that” (Interviewee_11, OPTED, interview with Authors June 2022).
26Generally, the perception is that computational tools work best for English, and the more a language diverges from English, the more likely it is that tools will be unavailable or require significant work to apply. Among survey respondents employing computational methods (N=142), the availability of suitable tools is considered a major problem, however, for “only” 26% of those studying English text but for 45% of those studying the non-English text. Factors such as accent handling, languages using specific symbols, or the scarcity of validated stop-word lists, lead researchers to either abandon certain languages or methods altogether.
“I had quite some students that were using Arabic text, […] there we had also some encoding things just to start with, you know, to get the texts properly into R and work with them when they have very different letters etc. […] they often have different encodings on their computers so when they read even a proper Arabic text into R, for instance, they transfer to very weird systems that they are not aware of.” (Interviewee_21, OPTED, interview with Authors June 2022).
27A concern almost universally expressed among interviewees is the difficulty in analysing and interpreting citizens’ short, unstructured, and less strategic text. This issue becomes more entrenched when translation is required, also limiting the collaboration opportunities for research on languages that are less commonly studied:
“It’s extremely easy to find people who will code specific languages, or content in specific languages. It’s much more difficult to find multiple people in [other] languages in comparison.” (Interviewee_17, OPTED, interview with Authors May 2022).
28Moreover, researchers’ employment of the terminology may not fit the reality of citizens in the regions they study. For example:
“Arab Spring has been standard in Western and English language bibliographies and library searches and even keywords in general and so on. But it is a colonial term that has been externally imposed on the people in the region, and people in the region never use the “Arab Spring” as a term” (Interviewee_2, OPTED, interview with Authors July 2022).
29Interviewees in departments with more extensive linking among disciplines highlighted the benefits of collaboration concerning methodological approaches they were able to employ. Access to data is also impacted by networking possibilities.
30CPPT frequently requires diverse skillsets from multiple disciplines, languages, and methodological experiences, not to mention that time-consuming or costly research can be alleviated when collaborating. However, a recurring pattern emerged from interviews with more junior researchers, who were substantially less likely to collaborate compared with more career-advanced scholars. Moreover, some interviewees experienced exclusion from international networks due to their institution’s location or a lack of international openness within their institutions:
“Sometimes I think if I would have had this network, maybe the problems that I faced in gathering data during my PhD would not have been so pronounced […] I’m a migrant. When migrants are new to a place, they don’t know their ways around things” (Interviewee_18, OPTED, interview with Authors July 2022).
“Opening science, decentralising science, is also about establishing connections with our peers in other places” (Interviewee_15, OPTED, interview with Authors June 2022).
31A bias in favour of the English language is common throughout many aspects of CPPT research. This bias manifests itself in three primary ways.
32First, non-English-speaking countries are often treated as case studies, while English-speaking countries are considered as reflecting the world. This dynamic was observed by interviewees studying European countries but it was even more pronounced in non-European contexts.
33Second, publishing in English journals is frequently perceived more favourably by institutions and it often facilitates the wider dissemination of the research.
“It’s kind of a bet. ‘Oh this I would save to publish in English’, and then the chances of being refused by the publisher, by the journal, are higher.” (Interviewee_11, OPTED, interview with Authors June 2022).
34Third, non-native English speakers may face doubts about their ability to work with CPPT material in English. This scepticism is not as prevalent for English-speaking researchers working with non-English material.
35In a field with such variety of research methods, many interviewees perceived a bias toward quantitative methods as being more valuable or rigorous than qualitative approaches. This perception persists despite the evidence from the CPPT review indicating that qualitative methods are the most commonly used (Gelovani et al. 2022). Additionally, interviewees expressed concern that the methodological choices of peers are often not easy to understand.
36Finally, the respondents highlighted systemic and institutional inequalities in research guidance-setting process. As research progresses and stricter rules become necessary, these inequalities could potentially intensify over time.
“To me it seems that it is fairly easy to incorporate and amplify inequalities between places that have resources, and places that do not have resources. Because if you want to be GDPR compliant, there will be more steps to carry out. If there is institutional support, mostly legal support, then that will be done in a much better way. But not all institutions can provide that, […] and this obviously might lead to some people not embarking on specific research questions, because there are six more steps that need to be carried out.” (Interviewee_17, OPTED, interview with Authors May 2022).
37Our findings point to several trends: 1) some CPPT researchers avoid research questions they are interested in, where data accessibility is perceived as too difficult to navigate; 2) additional resources are essential for specific CPPT projects, including studies of non-English text, but resources are unevenly distributed; 3) inequalities remain in the ability to engage in CPPT research across institutions, regions, and scholars at various career levels.
38The scarcity of collaboration opportunities impacts researchers’ ability to access data, particularly for those working outside of North America and Western/Northern Europe. Resources for translation may be more difficult to acquire when studying languages spoken in some regions of the world than others, making data access more complicated. To address these challenges, a specialised infrastructure is needed to bring together researchers from different disciplines, competencies, and regions to facilitate the exchange of data, context knowledge, and language skills and to recognise tools for data gathering and analysing (Balluff, 2023). Yet such initiatives are scarce and expensive, thus they need substantial international institutional support8.
39Perhaps the greatest challenge in studying CPPT is the limited access to text data produced by citizens, specifically due to restrictions imposed by social media corporations (De Vreese and Tromble, 2023). This limited access is further compounded by the fact that these restrictions are subject to change over time, often without much warning (Bruns, 2019; Venturini and Rogers, 2019; Perriam et al., 2020). Given the reliance on social media data among CPPT researchers, a significant limitation is that individuals who post on social media platforms are not representative of the general population (Hargittai, 2020). There is both a performative aspect to online discussion, and an algorithmic impact that could affect how accurately observed online behavior reflects the theoretical concepts researchers are interested in (Lazer et al., 2021). Despite the challenges, publicly available social media posts continue to be a “low-hanging fruit” (Özkula et al., 2023; Burgess and Bruns, 2015). Yet, best practices for academics to collaborate with policymakers and platforms corporations to ensure reliable and ethical access to data still remain underdeveloped (Dommett and Tromble, 2022). While our study identified primarily social media-based access issues, future studies can benefit from focusing on the type of CPPT usually created outside of social media, i. e. letters to the editors, terrorist manifestos, or protest posters.
40Matters are further complicated for researchers interested in non-English text and non-Western regions, as such projects often require more resources. The additional difficulty placed upon researchers studying non-Western regions and languages are a large part of the motivation for calls to de-westernise and de-colonialise communication research (Suzina, 2021; Ganter and Ortega, 2019). Scholars advocate for promoting greater heterogeneity and comparative social media research (Matassi and Boczkowski, 2021). It remains interesting to observe the future developments of “capabilities and impact of large language models (LLMs) in the wake of ChatGPT” (Jungherr, 2023).
41Other resource inequalities contribute to disproportionate barriers for some researchers. Collaboration reduces the individual cost of CPPT research, but networking possibilities are limited (Giglietto et al., 2012) and unevenly distributed. Multidisciplinary collaboration is further hindered if it is undervalued (Lazer et al. 2020). Establishing sustainable interdisciplinary research groups can be challenging but may be essential (Theocharis and Jungherr, 2021). Unequal networking has also been noted at conferences, where researchers with fewer barriers show reluctance to address the systemic bias experienced by other less-privileged researchers (Ng et al. 2020).
42While these challenges, and the inequalities they exacerbate, are daunting, heightened awareness can facilitate addressing these concerns. For example, conferences ensuring better representation of underrepresented regions could enhance better global networking. Collaboration between developers and language experts, particularly those with diverse alphabets or language structures, will ensure the optimisation of tools to analyse various languages. As knowledge of shared challenges and inequalities among CPPT researchers is growing, so can the potential for developing constructive and effective remedies.