Navigation – Plan du site

AccueilNuméros34On Complex Argumentation Structur...

On Complex Argumentation Structures in Tweets

Robin Schaefer, Sophia Rauh et Manfred Stede
Traduction(s) :
Sur les structures complexes de l’argumentation dans les tweets [fr]

Résumé

Using a corpus of all tweets issued by German members of parliament in a recent 6-year period, we study the formal complexity of the argumentation found in tweet pairs, where the second is a reply to the first. Our complexity measures are based on the number of argumentative units and the relations among them. We suggest a method for identifying potentially complex argumentation and then manually annotating a set of tweet pairs for their argument structure. Despite the length limit of tweets, we find that politicians indeed perform rather elaborate arguments and that the structures require amendments to an annotation scheme that has previously been used in similar corpus projects, though on more traditional types of texts.

Haut de page

Texte intégral

Introduction

1The landscape of social media can change rapidly; Twitter, for example, currently has a mixed image after the change in ownership and the renaming to ”X”. Before, it had been a popular platform for ”serious” communication, for example among scientists, journalists, NGO activists, and politicians. A sizable part of that communication was not just news broadcasting but exchange of opinion and argumentation. We study Twitter use by German politicians in the years 2016 to 2022. Our focus is on assessing the complexity of argumentation in the tweets sent by members of parliament (henceforth: MPs) in that period. The original dataset comprises the complete set of those tweets, but for our current research we narrow it down to the topic of climate change, resulting in some 30,000 tweets that form the base set for our experiments.

2Previous research has explored the complexity of argumentation on Twitter in the sense of being part of a multi-venue discourse with many participants (Aakhus and Lewinski 2017). For example, Greco (2023) studies tweets issued by activists on sustainable fashion in the context of a broader discourse involving companies, consumers, and other stakeholders. In contrast, our notion of complexity here refers to the purely formal complexity of arguments as constellations of claims and premises that are brought forward by the MPs. We examine tweet pairs that arise from using the ”reply to” function of Twitter, i.e., where a tweet directly follows up on the content of some other tweet. We define as complex any constellation that involves more than two argumentative units, i.e., which goes beyond the minimal argument form of one claim and one premise.

  • 1 For comprehensive overviews of the field of argument mining, see Lawrence and Reed (2020) or Stede (...)

3The perspective from which we approach this theme is that of corpus annotation and automatic argument mining (henceforth: AM).1 Over the last decade, AM has developed into a fruitful research discipline that conceives argument analysis as a pipeline of separate tasks, including the detection of claims (Daxenberger et al. 2017) and premises (Rinott et al. 2015). Related tasks focus on the assessment of argument quality (Wachsmuth et al. 2017) and underlying argumentation strategies (Al-Khatib et al. 2016, Schaefer et al. 2023). Traditionally, AM is implemented as supervised learning based on annotated text material, which in turn is based on an annotation scheme that operationalizes the model of argumentation underlying the research.

4The majority of AM work has focused on texts from formal and edited domains, such as legal documents (Palau and Moens 2009), Wikipedia (Levy et al. 2014), student essays (Stab and Gurevych 2014), scientific papers (Green 2018) or news editorials (Al-Khatib et al. 2016). Not surprisingly, such approaches tend to yield much worse results when applied to user-generated text that is found in social media. According to Sňajder (2016), potential challenges in these texts include not only noisy text (which is well known) but also vague claims and vague argument structure. In recent years, AM has developed a growing interest in social media, with Twitter having been in focus. Previous work primarily investigated the annotation and detection of argument components in tweets (Bhatti et al 2021, Wührl and Klinger 2021) and their types (Addawood and Bashir 2016). To the best of our knowledge, however, no research so far studied specifically the complexity of argumentation within tweets and across the boundary of tweet pairs. Hence, our broad research theme is:

RQ 1: Do German MPs use Twitter for argumentative exchanges that involve complex claim/premise constellations?

  • 2 Note that threads can be published incrementally or as one batch. In this work, however, we do not (...)

5To shed light on this question, we conducted an exploratory annotation study of the German MP tweets in our corpus, filtered for the climate change topic. Twitter allows for different options to post a tweet, which is reflected in our study design. We focus on conversations and threads. A conversation is defined as a chain of tweets written by different authors and linked to each other via Twitter’s “reply to” functionality. A thread2, on the other hand, describes a chain of tweets by the same author. While the former may resemble an offline discussion, the latter allows for more complex utterances, and hence argumentation, by an individual person. We hypothesize that these different settings may encourage variation in argumentation structures. We begin our investigation of complexity with just pairs of tweets, leaving the investigation of longer chains for future work.

6We apply two annotation schemes to our data. First, we annotate argumentation structures for claims and premises, henceforth called “argumentative discourse units” (ADU), linked by support and attack relations (see Fig. 1 for an annotated example). Our starting point is the scheme proposed by Peldszus and Stede (2013), which is based on the work of Freeman (1991) and has been used in a variety of AM projects. It had been designed for analyzing short monolingual texts, however, and thus our second – more concrete – research question is:

RQ 2: Can an established annotation scheme for monologue be used for pairs of tweets, or are changes/extensions to the scheme necessary?

7Our second annotation scheme captures the coherence relation between the tweets in a pair, for example, the agreement or disagreement in a pair of type “conversation.” These relations are meant to supplement the argumentation analysis by attending to basic dialog phenomena. As mentioned above, we are generally interested in potential differences between the two types, which leads to our last research question:

RQ 3: Do we find differences between tweet threads and conversations in terms of the complexity patterns of argumentation?

8This paper is structured as follows: in Section 1, we present relevant related work. In Section 2, we describe the data collection and preprocessing, our annotation schemes, and the annotation procedure. This is followed by a qualitative description of interesting annotation examples in Section 3 and a statistical analysis of the data using various measures of argument complexity in Section 4. We provide a discussion of the results in Section 5 and then get to our Conclusion.

1. Related Work

1.1. Argument Mining on Twitter

9Argument mining on Twitter has received some attention over the years, though it still has a particular niche status in the AM community. Bosc et al. (2016) presented the first tweet corpus (DART) annotated for argumentation. They labeled 4,000 English tweets on four topics as argumentative or not and presented the first results on automatic relation identification by pairing tweets and marking “support/attack” relations between them.

10Other early work includes Addawood and Bashir (2016), who annotated an English tweet corpus with a set of semantic evidence types in a two-step procedure. After annotating tweets for being argumentative or not, evidence types were added to the argumentative tweets. Afterward, different sets of linguistic features were extracted for the machine learning models. Building on Bosc et al. (2016), Dusmanu et al. (2017) extended a subset of DART with another tweet set and annotated them for being opinionated or factual, and in the case of factual tweets, for the information source.

Figure 1: An example of complex argumentation. Green depicts claims, while blue shows premises. The numbering represents the ordering of the ADUs in the tweet. The tweet has been translated to English and simplified such that the argumentation structure was not affected.

  • 3 According to Landis and Koch (1977), a κ value of 0.21-0.40 represents “fair”, 0.41-0.60 “moderate” (...)

11While these studies treated argumentation only on the level of full tweets, Schaefer and Stede (2020) published the first (German) tweet corpus annotated for argumentation on the level of ADU spans. While forming an important starting point for our work, a drawback is the somewhat low inter-annotator agreement3 (Cohen’s κ: 0.38-0.45), caused by the subjectivity of making the distinction between claim and evidence. Schaefer and Stede (2022) tried to mitigate this issue with a modified annotation scheme that relies less on the dependency between claim and evidence and more on fine-grained semantic types, like Addawood and Bashir (2016). This resulted in more reliable annotations for their new corpus (Krippendorff’s alpha: 0.40-0.83). Note that none of these corpora were annotated for argumentative relations, which are necessary for analyzing the complexity of argument structures. The present paper improves on this shortcoming.

12Other works with a special focus on premise or claim includes that of Bhatti et al. (2021). They utilize hashtags as claims, e.g., #StandWithPP (“Planned Parenthood”), and annotate tweets containing support with a reason (with a premise), support without reason (without a premise), or no explicit support. With 24,100 tweets, this corpus is the largest AM Twitter resource to date. Wührl and Klinger (2021) built a corpus for claim detection in a biomedical context. 1,200 tweets were annotated for containing explicit or implicit claims. Like earlier projects, the task resulted in relatively low inter-annotator agreement.

1.2. Annotation Schemes

13Starting with the early research of Palau and Moens (2009), annotation schemes used in AM have generally modeled argumentation as tree structures. Palau and Moens labeled conclusions and premises, with the latter divided into the types “support” andagainst.”

14Similarly, Peldszus and Stede (2013) assigned ADUs to the roles of “proponent” and “opponent” (following Freeman 1991) and built trees by means of support and attack relations. In addition to a simple support relation, they also account for “joint” support, where two or more premises support a claim (or premise) together (i.e., independently, they would not form sufficient support). The attack relation, which is used by the text author to deal with potential counterarguments against their position, is split into “rebut” (attacking the validity of an ADU statement) and “undercut” (attacking the support/attack relation that is proposed to hold between two ADUs, i.e., denying the argumentative relevance). For illustration, we show examples used by Peldszus and Stede (2013):

We should tear the building down. It is full of asbestos (Support).

We should tear the building down. It is full of asbestos, and all buildings with hazardous material should be demolished (Joint support).

We should tear the building down. It is full of asbestos. On the other hand, many people like the view from the roof (Rebutting the claim).

We should tear the building down. It is full of asbestos. On the other hand, it could also be cleaned up (Undercutting the support)

15Targeting relatively short texts, Peldszus and Stede (2013) assumed that the tree is rooted in a single “central claim” that constitutes the main message of the text. As a variant approach suitable for longer essays, Stab and Gurevych (2014) added the ADU type “major claim” for the central thesis of the essay. This claim can be supported by various claims throughout the essay and, eventually, premises that support it.

2. Data

  • 4 For data privacy reasons only tweet IDs can be freely distributed.

16In this work, we utilize a corpus of German tweets on climate change written by MPs. As mentioned above, Twitter was/is a relatively popular communication channel for MPs to justify their positions and engage in discussions. Our corpus is part of a more extensive set of tweet IDs provided by Lasser et al. (2022)4, which contains all tweets published by former and current MPs from January 1st, 2016, to March 15th, 2022 (n=754,233). We collected the tweets via the Twitter API in December 2022. In the following, we describe how the corpus was preprocessed to arrive at the subset of tweets to be used in our study on complex argumentation.

2.1. Data Preprocessing

17Our initial dataset contained tweets on various topics. In the first step, we filtered the data for tweets on climate change by using several keywords related to the topic. This led to a subset of 30,242 tweets. To narrow this down to find tweets that are likely to be argumentative, we searched for causal or contrastive coherence relations between text units, as, presumably, complex argumentation tends to contain these relations in particular, for making support and attack.

  • 5 http://connective-lex.info

18A typical signal for coherence relations in texts is connectives (cf. Moeschler 1989 on the role of connectives for argumentation). We therefore extracted discourse connectives that express the Penn Discourse Treebank (Webber et al., 2019) senses “Cause” and “Contrast” from the German connective lexicon DiMLex5 (Stede and Umbach 1998) and identified the tweets that include these connectives. We built a table containing the tweets, the connectives occurring in them, and their frequency. Some of the selected words and phrases also have a non-connective (“sentential”) meaning, so they do not necessarily express a discourse relation. We heuristically removed those connectives whose non-connective reading in German is more frequent, including allein (“alone”), noch (“still”), nur (“only”), dass/daß (“that”), eben (“just”), and wo (“where”). After this step, the table still contained several false candidates, so we performed a manual filtering. From the ranked list of most common connectives in the climate corpus, the first seven have a sentential reading as well: aber (“but”), durch (“through”), da (“as/since”), damit (“so that”), dafür (“for that/but”), denn (“because”), doch (“but”). They are followed by weil (“because”) and deshalb (“therefore”), which always express discourse relations. Some connectives can express multiple relations, like und (“and”), which we also excluded, as its Contrast reading is relatively rare. Finally, to target complex argumentation in particular, we kept only tweets containing at least two connectives of type either Contrast or Cause.

19Our second perspective on the formal complexity of argumentation is to analyze tweets in context. We use tweets that are replies to another one. We assume that the tweet author agrees or disagrees with the prior one and potentially justifies their opinion. Therefore, we searched for the IDs of the tweets that received replies and added these tweets to our set.

20By applying these filtering methods, the original 30,242 tweets were reduced to 111 replies. 20 reply tweets responded to another author (conversation), while the rest of the set extended a tweet by the same author (thread). By manual inspection, we identified several tweet pairs that did not focus on climate change and removed these from the corpus. Our final set has 102 tweet pairs, thus 204 tweets. 20 tweet pairs are conversations and 82 tweets pairs are threads.

2.2. Annotation Schemes

21We apply two annotation schemes to the tweet pairs. The first targets argumentation structure and is inspired by the proposal of Peldszus and Stede (2013) (see Section 1.2). As we are dealing here with very short texts, we slightly modify the scheme and, as the first step, identify ADUs as either claim or premise. We regard a claim as a standpoint toward a particular topic or aspect, while a premise is used to either support or attack a claim; together, the two form an argument. A premise can be further supported (or attacked) by another premise as well. Using the two subtypes of support and attack mentioned in Section 1.2, we annotate four relation types: support-simple, support-joint, attack-rebuttal, and attack-undercut. As ADUs can be only parts of a tweet, annotators also make segmentation decisions.

22In contrast to Peldszus and Stede (2013), we loosen the a priori restriction of having a single central claim per text/tweet and instead leave it to empirical analysis to ascertain whether it holds for our data or not. In addition, we allow premises to support or attack more than one ADU, which violates the argumentation tree structure described above and instead allows for more variant structures. We allow relations between ADUs to also occur across tweet boundaries, but this happened rarely and is not further analyzed in this paper. For illustration of the scheme, see the examples of tweet pairs in Figures 2 and 3 in Section 3.

23Our second annotation scheme models the coherence between two tweets by means of a single inter-tweet relation. We use the classes: continuation (elaboration), question/answer, (partial) agreement, (partial) disagreement, and others. Continuation (elaboration) is defined as the continuation of a line of argumentation or thought that was brought up in the source tweet and is extended in the reply tweet. It may be further elaborated on previously discussed aspects, i.e. more details may be given. However, this is not mandatory. Question/answer applies to situations where a question is asked in the source tweet, and an answer is given in the reply tweet. (Partial) agreement and (partial) disagreement refer to (dis)agreement between source and reply tweet. While similar in meaning, it is important to note that partial agreement and partial disagreement are not identical. For instance, a reply tweet partially agreeing with the source tweet may also contain neutral propositions while showing no disagreement. Finally, “other” is used for cases where no alternative label seems plausible. Annotators may annotate one or more labels per tweet pair. However, “other” is only to be annotated individually.

2.3. Annotation Procedure

  • 6 https://github.com/heartexlabs/label-studio

24We used the tools INCEpTION (Klie et al., 2018) and Label Studio6 for annotating the argumentation structure and the coherence relations, respectively. As this is explorative work on a small set of tweets, we do not evaluate our annotation scheme by using inter-annotator agreement metrics. Instead, we iteratively refined our argumentation structure by 1) independently annotating the tweet set, 2) discussing our results, and 3) adapting our annotations to the improved scheme. We found that our data contained a sufficient amount of recurring patterns that allowed us to create an annotation scheme that successfully captures the main kinds of argumentation structures observable in tweets. With respect to our inter-tweet relation scheme, we found that these were intuitive categories that did not require much adjustment. One annotator labeled all tweet pairs, followed by a second annotator who reviewed the annotations. All authors of this study participated in the annotation.

3. Qualitative Inspection: Examples

25In this section, we briefly discuss example annotations for both conversation and thread pairs before turning to our statistical analysis in Section 4.

  • 7 « Paris » refers to the Paris Agreement, i.e., the climate protection agreement made at the UN Clim (...)

26Figure 2 shows an example of a conversational tweet pair. The source tweet contains an argument with depth and width of 2 (see Section 4.3 for a definition of argument depth/width). The claim states thatGermany is to adopt a climate protection law to keep the promises made in Paris”7 and is jointly supported by two premises (units 2 and 3), i.e., “[lessons can be drawn from the past] [when a lack of commitment did no achieve good results].” The premises jointly support the claim, as they depend on each other.

Figure 2: Conversation Example. Green depicts claims, while blue shows premises. The numbering represents the ordering of the ADUs in the respective tweet. Tweets have been translated to English and slightly simplified (without affecting the argumentation structure).

27The reply tweet contains two claims, which are linked via a premise (unit 4). It begins with a claim stating that “Germany’s climate protection policy was bad for a decade.” This claim gets rebutted by two premises (units 2 and 3), declaring it was actually “great internationally” and had “good national objectives.” However, both rebuttals get undercut by another premise (unit 4) arguing that “no binding instrument for the implementation existed.” By doing so, the author strengthens his initial claim. The tweet ends with another claim, which is backed by the premise that undercuts the rebutting premises of the first argument, i.e., due to the lack of instruments for implementation, “the Social Democratic Party (SPD) included the Climate Protection Act in their election program of 2013.” Interestingly, this tweet does not adhere to a classic argumentation tree structure, i.e., a structure where the argumentation concludes with a single claim. Instead, two individual claims are linked via the same premise. It further shows the extensive use of both attacking types, i.e., attack-rebuttal and attack-undercut. With respect to the inter-tweet relation, the reply tweet (partially) agrees with the source tweet.

28Figure 3 depicts a thread example that shows notable differences from the conversational example. The source tweet contains a rather deep argument consisting of a claim arguing in favor ofCO2 reduction via the means of innovation and certificate trading.” It is supported by a premise according to which “a CO2 tax will not be efficient.” This premise is supported by another premise, i.e.the price of emissions does not affect the climate.” The argument has thus a depth of 3 and a width of 1. With respect to linearization, we find that it deviates from the argumentation structure. First, the relation between both premises is expressed (units 1 and 2) before ending with the claim.

29The reply tweet also contains a single claim, i.e.the amount of CO2 needs to be limited.” In contrast to the source tweet, however, it is supported by four premises. Hence, the tweet has a depth of 2 and a width of 4. First, the claim is backed by two joint premises stating that “[the decisive factor (for the increase in temperature) is the amount of emitted CO2] [and not the price of emitted CO2].” Then, following the expression of the claim. another premise (unit 4) is uttered according to which “the limitation of CO2 will result in a price for emitted CO2 on its own.” Finally, the author states that this will be “the most efficient way to protect the climate” (unit 5). While this example does not include attacking premises, it shows arguments of extensive depth/width. Contrasting with the conversational example, the reply tweet continues the argumentation of the source tweet. This is an often-observed pattern in thread constellations.

Figure 3: Thread Example. Green depicts claims, while blue shows premises. The numbering represents the ordering of the ADUs in the respective tweet. Tweets have been translated to English and slightly simplified (without affecting the argumentation structure).

Tweet

Conversation

Thread

ADUs

Total

5.15

5.91

ADUs

Source

2.35

2.65

ADUs

Reply

2.80

3.27

Claims

Source

1.35

1.49

Premises

Source

1.00

1.16

Claims

Reply

1.70

1.62

Premises

Reply

1.10

1.65

Relations

Total

2.10

2.82

Relations

Source

0.95

1.34

Relations

Reply

1.15

1.50

Support-Simple

Source

0.65

1.13

Support-Joint

Source

0.10

0.09

Attack-Rebuttal

Source

0.15

0.10

Attack-Undercut

Source

0.05

0.00

Support-Simple

Reply

0.70

1.20

Support-Joint

Reply

0.10

0.21

Attack-Rebuttal

Reply

0.20

0.10

Attack-Undercut

Reply

0.15

0.00

Table 1: Means of Absolute Counts of ADUs and Relations

4. Statistical Analysis

30We approach data analysis from the perspectives of our setup of source and reply tweets and the two kinds of reply constellations – different author, i.e., conversations, vs. same author, i.e. threads. Given the two annotation schemes explained in Section 3.2, we present basic statistics of component and relation distributions (Sec. 5.1), analyses of claim positioning (Sec. 5.2), analyses of argumentation depth/width and completeness (Sec. 5.3), and proportions of inter-tweet relations (Sec. 5.4).

4.1. Basic Statistics of ADUs and Relations

31First, we calculate the proportions of argumentative tokens. To this end, we remove punctuation, including @ and # symbols, from the data, before splitting it using a simple whitespace tokenizer. Then, we calculate the proportions of argumentative tokens by source/reply tweet and conversation/thread distinctions. This analysis reveals that conversations contain fewer argumentative tokens (source tweet: 0.73; reply tweet: 0.71) than threads (source tweet: 0.82; reply tweet: 0.84).

32Means of absolute counts of ADUs and relations are presented in Table 1. Conversations show a total mean of 5.15, i.e. roughly five ADUs are used on average. The source tweets exhibit a mean of 2.35, while the reply tweets show a mean of 2.80. Thus, reply tweets contain a higher number of components. A similar pattern can be observed for threads. They show a total mean of 5.91 ADUs. Again, reply tweets have a higher mean (3.27) than source tweets (2.65).

33Turning to individual ADU types in conversations, claims tend to be used more often than premises. This is true for both source tweets (1.35 vs 1.00) and reply tweets (1.70 vs 1.10). Similarly, threads show a higher mean for claims than premises in source tweets (1.49 vs 1.16). However, in reply tweets no notable difference can be observed (1.62 vs 1.65).

34As with ADU counts, reply tweets contain a higher number of relations. This is the case for both conversations (1.15 vs 0.95) and threads (1.50 vs 1.34). The means of absolute relation count for the whole dataset are 2.10 and 2.82 for conversation and thread tweets, respectively. Thus, like ADU counts, tweets in a thread contain more relations.

35Regarding individual relation types, source tweets show a notable difference in means between conversation and thread constellations for support-simple (0.65 vs. 1.13), which also achieves the highest means in general. No such difference can be observed for the other types. Support-joint and attack-rebuttal show means of 0.09-0.15, while attack-undercut obtains lowest scores (0.00-0.05). Reply tweets, however, tend to show more variance. Again, support-simple is the most dominant type for both conversations and threads (0.70 vs 1.20), with notable distances to the other types. The attack variants yield a mean of 0.15-0.20 in conversations while rarely being used (or not at all in the case of attack-undercut) in threads.

36Finally, we calculate the proportions of premises supporting/attacking more than one ADU, as we specifically allow this in our annotation scheme. While proportions are relatively low for both conversations and threads, we still find that conversations tend to contain more instances of premises with outgoing relations to more than one ADU. Also, we find that differences can be observed between source and reply tweets. Premises in source tweets of conversations show a proportion of 0.10, while reply tweets show a proportion of 0.18. Source tweets in threads yield a proportion of 0.07, while reply tweets obtain a score of 0.12.

37To summarize, threads tend to contain more argumentation than conversations, both in terms of tokens and ADUs. Similarly, reply tweets contain more argumentation than source tweets. Claims are more frequently annotated than premises (with the exception of reply tweets in threads). With respect to relations, support-simple is the most prominent type, albeit with substantial differences between conversations and threads. Attack relations are less often used, which is especially the case for reply tweets in threads.

4.2. Claim Positioning

38For positioning of claims (see Table 2), we find that claims usually initiate or end the argumentation in a tweet. This is the case for both source and reply tweets and for both conversations and threads. Moreover, only threads contain arguments that are started and ended by premises.

39Source Tweets: In conversations, a proportion of 0.75 of source tweets begin with a claim, while 0.45 end with it. Focusing on source tweets with more complex argumentation, i.e. tweets with >= three ADUs, we find that a proportion of 0.90 tweets starts with a claim while 0.40 ends with a claim. In contrast, no tweet starts and ends with a premise. While source tweets in a thread show a generally similar pattern, there are some differences to be observed. First, claims are equally distributed at the beginning and end of a tweet with more complex argumentation (0.67). Second, some tweets also begin and end with premises. Both results differ from the proportions we find in conversations.

Tweet

Conversation

Thread

Starting w/ Claim

Source

0.75

0.67

Ending w/ Claim

Source

0.45

0.61

Starting w/ Claim (ADU N >= 3)

Source

0.90

0.67

Ending w/ Claim (ADU N >= 3)

Source

0.40

0.67

Starting and Ending w/ Premise (ADU N >= 3)

Source

0.00

0.14

Starting w/ Claim

Reply

0.80

0.61

Ending w/ Claim

Reply

0.70

0.56

Starting w/ Claim (ADU N >= 3)

Reply

0.70

0.61

Ending w/ Claim (ADU N >= 3)

Reply

0.80

0.53

Starting and Ending w/ Premise (ADU N >= 3)

Reply

0.00

0.15

Table 2: Proportions of Claim Positions

40Reply tweets tend to begin or start with a claim as well. Proportions for the full data set range from 0.56 to 0.80. In threads with more complex argumentation only a reply tweet proportion of 0.15 started and ended with premises. In contrast, reply tweets in conversations do not start and end with a premise. Both author constellations show the opposite pattern to that of source tweets. In reply tweets, conversation tweets end more frequently with a claim (0.80 vs. 0.70), while thread tweets start more often with a claim (0.61 vs. 0.53).

4.3. Argumentation Depth/Width and Completeness

41We define the depth of an argument structure as the maximum number of ADUs that form a single chain of inference (one claim and a series of premises). Width is defined in terms of the number of ADUs that are located on the same level in the argument graph, i.e. which have the same distance to the argument’s claim. For instance, an argument claim < premise < premise, where < indicates support, has a depth of 3, and a width of 1. Adding another supporting premise to the claim would increase the width to 2.

42Conversation and thread constellations differ on these measures. Regarding maximal values, in conversations, both source and reply tweets have a maximum depth of three ADUs (mean: 1.60 vs. 1.65) and a maximum width of two ADUs (mean: 1.00 vs. 1.25). Threads, however, show somewhat more variance: Source tweets have a maximum depth of three ADUs, and for reply tweets, the maximum is four (mean: 1.59 vs. 1.81). With respect to width, both source and reply tweets differ in mean (1.32 vs. 1.54), while showing maximum widths of seven and five ADUs, respectively. However, given the substantial distance to the mean, these values can be seen as outliers.

43The actual proportions of tweets per depth/width values are given in Figure 4. We note, first, that conversations show a higher proportion of non-argumentative tweets (depth and width are zero), as do source tweets compared to reply tweets. Second, threads have a higher mode, i.e., a depth of 2. This is especially the case for reply tweets. They also include the only tweet with a depth of 4, while reply tweets in conversations often contain argumentation with a maximum depth of one – these are tweets where all claims remain unsupported/not attacked. Third, regarding argument width, both conversations and threads have modes at a width of 1. However, in conversations these more frequently include tweets with depth 1, i.e., incomplete arguments. Finally, threads also exhibit widths of >2. Reply tweets tend to have a higher argument width, which confirms the difference in means previously described.

Arg Subset

Tweet

Conversation

Thread

Mean proportion of incomplete arguments

no

Source

0.38

0.29

Mean proportion of incomplete arguments

yes

Source

0.44

0.32

Mean proportion of incomplete arguments

no

Reply

0.48

0.31

Mean proportion of incomplete arguments

yes

Reply

0.51

0.34

Proportion of Tweets containing only complete arguments

no

Source

0.55

0.67

Proportion of Tweets containing only complete arguments

yes

Source

0.47

0.63

Proportion of Tweets containing only complete arguments

no

Reply

0.45

0.55

Proportion of Tweets containing only complete arguments

yes

Reply

0.42

0.51

Table 3: (Mean) proportions of incomplete arguments per Tweet and of Tweets with only complete arguments. Arg Subset refers to a subset containing only argumentative tweets, i.e. tweets with at least one ADU.

44Table 3 contains mean proportions of incomplete arguments, i.e., without a premise, and proportions of tweets that contain only complete arguments. Notably, conversations show a higher mean proportion of incomplete arguments than threads (source: 0.38 vs 0.29; reply: 0.48 vs 0.31). Also, reply tweets in conversations tend to contain more incomplete arguments than source tweets (0.48 vs 0.38). In threads, however, the difference is less obvious (0.31 vs 0.29). When calculating the scores only for the argumentative tweets, the mean proportions in conversations increase especially in source tweets, while proportions only mildly change in threads.

45A similar pattern can be observed for the proportions of tweets that only contain complete arguments. Threads show a notably higher proportion than conversations in this regard, both for source tweets (0.67 vs 0.55) and reply tweets (0.55 vs 0.45). Analyzing only argumentative tweets again reduces the proportion primarily for source tweets in conversations (0.55 vs 0.47), while less notable changes can be observed for threads.

46To summarize, we find that threads tend to contain arguments of higher depth and width compared to conversations. Furthermore, conversational tweet pairs are more often non-argumentative or do contain more unsupported claims, i.e. incomplete arguments. Threads, on the other hand, more often contain only complete arguments.

4.4. Inter-Tweet Relations

47For inter-tweet relations in conversations (see Table 4) we find that a proportion of 0.45 tweet pairs exhibit a (partially) disagreeing relation. This is the largest class, followed by (partial) agreement with 0.20. Question/answer and continuation (elaboration) show proportions of 0.15 and 0.10, respectively. To summarize, a proportion of roughly 0.65 has been annotated with either (partial) agreement or (partial) disagreement. Only one tweet pair of different authors was annotated with more than one label [(partial) agreement/(partial) disagreement].

48Threads show a substantially different pattern. A vast majority of pairs (0.91) have been annotated with continuation (elaboration), while a small subset has been assignedother” (0.08), and an even smaller subset has been annotated with partial disagreement (0.01). No thread has been annotated with any of the other classes. Also, no pair has been assigned more than one label.

Fig.4: Proportions of argumentation depth and width by tweet type for conversations/threads.

The top four plots contain the depth data, the bottom four plots show the width data. Note that the maximum widths in threads are not shown in this plot.

  

Conversation

Thread

Continuation (Elaboration)

0.10

0.91

Question/Answer

0.15

0.00

(Partial)Agreement

0.20

0.00

(Partial)Disagreement

0.45

0.01

Other

0.15

0.08

Table 4: Inter-Tweet Relations. The sum of proportions by conversation/thread can amount to > 1 as tweet pairs can be assigned multiple labels.

5. Discussion

49We now discuss our results in relation to our three research questions and then offer some considerations of potential limitations of our study.

50RQ 1. To assess the usage of Twitter for complex argumentative exchanges by MPs, we investigated complexity mainly 1) from the perspective of ADU and relation distributions across tweet types and conversation/thread constellations, and 2) by utilizing the concepts of argumentation depth/width.

51First, we find a lot of argumentation overall, thereby showing that our proposed method of filtering via connectives with senses Cause and Contrast is suitable for identifying argumentative tweets. While this procedure may have produced a certain bias in our data, we consider these steps necessary to identify tweets with complex argumentation. We leave the investigation of tweets with less overtly marked argumentation for future research.

52Second, we found that reply tweets contain more argumentative material than source tweets—in terms of the number of individual tokens, ADUs, and relations. This finding seems plausible, as source tweets may be written without any conversational intention. Reply tweets, in contrast, are reactions, which makes them likely to contain argumentation toward the source tweet.

53Third, support-simple is the most common relation between claim and premise. Hence, when producing complete argumentation, authors are most likely to utilize the simplest form of support. Attacks are used less frequently, but with attack-rebuttal being more common than attack-undercut, and so authors prefer plain rejections over relevance dismissals. These two findings may be typical for the medium: Twitter is a channel for quick, direct, and short responses to political statements.

54Finally, our argumentation depth/width analysis confirmed the basic patterns of the ADU and relation distributions. Reply tweets in threads show more complex argumentation reflected by higher argument depth and width. Note, however, that these differences are relatively mild due to the limited tweet length. Most tweets show a maximum depth of 2, i.e., contain at least one full argument, and a maximum width of 1. However, while being an important measure, depth/width does not show the complete picture of argumentation complexity and variance. Recall the reply tweet in figure 2. This tweet has a depth of 2 and a width of 1 but is still quite complex due to its use of different types of attacks and the linking of two arguments via the same premise.

55Overall, we conclude that MPs indeed use tweets for complex argumentative exchange since our corpus contains high densities of different facets of argumentation complexity.

56RQ 2. Our annotation scheme for argumentation was derived from a scheme initially used for short and rather ”polished” monologue texts. We thus test to what extent the original annotation scheme needs to be changed or extended to be applicable to tweets. We adapted the annotation scheme in three ways: 1) We use claim/premise notation instead of proponent/opponent; 2) we allow for more than one central claim; 3) we allow premises to support/attack more than one claim or premise.

57All changes are necessary to consider the argumentative characteristics of Twitter. We used a scheme based on a claim/premise distinction instead of proponent/opponent for two reasons. First, the use of opponent arguments is limited by the short length of tweets, as reflected by the low counts of attack relations. Second, as conversations essentially represent dialogue, the roles of proponent and opponent may naturally fall into the source and reply tweet, respectively. We find that modeling cross-tweet relations can be more intuitively achieved via our inter-tweet relations, e.g. (dis)agreement, which function on the full tweet level, instead of relations on the argument level.

58We further show that a tweet can contain several unrelated arguments. A single argumentative tree structure, as defined in the original annotation scheme, could not sufficiently represent this. Also, we provide evidence that premises sometimes support more than one claim or premise. They can thus be used as a linking element between two arguments.

59RQ 3. Our last RQ concerns differences between conversations and threads in terms of the argumentation complexity. We find that the latter shows a higher number of ADUs, which is indicated by a higher proportion of argumentative tokens, thereby suggesting a higher complexity. This is also the case with respect to the higher average count of relations. We argue that this is due to the author’s particular argumentative intention in threads. As threading allows for the concatenation of tweets, authors can carefully construct their argumentation both within tweets and across tweet boundaries. The higher argumentation complexity in threads is further reflected by an overall more significant depth and width. While fewer tweets show a depth of <= 1, more tweets show a depth of >= 2. Similarly, only threads include tweets with a width of > 2.

60While threads tend to contain more complex argumentation structures than conversations, arguments in conversations’ reply tweets more frequently include attacks. This is in line with our inter-tweet relation annotation, which reveals that tweets in conversations are most likely to contain (partial) disagreement. In contrast, threads mostly show continuation, which correlates with a comparative lack of attacks. Again, this is likely due to the argumentative intent. In threads, authors are inclined to construct complex argumentation, while in conversations they argue in favor of or against another person’s arguments.

61Given that we treat argument completeness as a proxy to argumentation quality, we can interpret the utmost lack of complexity in terms of argument depth/width, i.e. arguments with both depth and width of 1, as low quality. However, a fine-grained quality analysis is beyond the scope of this study. Previous research has argued that argumentation in tweets tends to be somewhat incomplete (Schaefer and Stede 2022). So far, however, no quantitative analysis has revealed to what extent this assumption holds. One central finding of our study is the predominance of claims compared to premises. Thus, while complex argumentation exists in tweets, a substantial amount of claims remains unsupported (or not attacked), i.e. the argument remains incomplete. This is confirmed by our argument depth analysis, which shows that conversations contain more incomplete arguments than threads. Similarly, threads substantially more often contain only complete arguments. We argue that this pattern is due to argumentation in threads being more carefully constructed as a complex chain of arguments, which may result in higher argument quality.

62Remarks on Limited Scope. While representing an important first exploration of complex argumentation in tweets, we need to consider our study’s limited scope. As we only investigated tweet pairs, we ignored longer chains of tweets. While facilitating the annotation, this decision had consequences for the argumentation complexity that can be found in our data. Longer conversational chains, for instance, would likely include data from >2 participants in the discourse, which may increase the complexity of inter-tweet relations as a tweet could be related to n−1 tweets in the chain, where n equals the length of the chain. Furthermore, an author may produce more than one tweet in a chain, which could potentially render the annotation more complicated as well. With respect to threads, a chain of >2 tweets would allow for more complex argumentation. However, given that the overall number of authors remains the same (and continuation is the dominant inter-tweet relation in threads), we argue that the annotation remains comparable to thread annotation in our current study.

Conclusion

63In this paper, we investigated the complexity of argumentation structures in tweets, which has remained understudied in previous research. To this end, we annotated a small set of tweet pairs written by MP with a set of ADUs and argumentative relations. We modified an annotation scheme designed for monologue to account for characteristics of tweet data, e.g. the existence of several independent arguments or the usage of premises that support/attack more than one ADU. We show that our method for filtering tweets and annotating them using our modified annotation scheme is a good candidate for identifying complex argumentation in tweets.

64Notably, our analyses provide evidence for complex argumentation across tweet types, i.e., source and reply tweets, and tweet pair constellations, i.e., conversations and threads. We further show that threads tend to exhibit more complex argumentation structures than conversations, as shown by the respective argument depths/widths as well as ADU and relation statistics. This may be due to the opportunity for more sophisticated planning of the respective argumentation in threads.

65While our work provides an important exploration of argumentation complexity in tweets, future research may focus on 1) the automatic extraction of complex argumentation and 2) the investigation of longer tweet chains.

Haut de page

Bibliographie

Aakhus, Mark & Martin Lewinski. 2017. « Advancing Polylogical Analysis of Large-Scale Argumentation: Disagreement Management in the Fracking Controversy. » Argumentation 31-1, 179–207

Addawood, Aseel & Masooda Bashir. 2016. « “What is Your Evidence?” A Study of Controversial Topics on Social Media. », Chris Reed (ed.). Proceeding of the Third Workshop on Argument Mining [Online] DOI:10.18653/v1/W16-2801

Khalid Al-Khatib, Johannes Henning, Wachsmuth Kiesel, Matthias Hagen & Benno Stein. 2016. « A News Editorial Corpus for Mining Argumentation Strategies. » In Yuji Matsumoto and Rashmi Prasad (eds). Proceeding of the 26th International Conference on Computational Linguistics, 3433–3443

Tom Bosc, Elena Cabrio & Serena Villata. 2016. « DART: a Dataset of Arguments and their Relations on Twitter », Calzolari, Nicoletta, Khalid Choukri, Thierry Declerck, Sara Goggi, Marko Grobelnik, Bente Maegaard, Joseph Mariani, Helene Mazo, Asuncion Moreno, Jan Odijk, and Stelios Piperidis (eds). Proceeding of the Tenth International Conference on Language Resources and Evaluation, 1258–1263

Daxenberger, Johannes, Steffen Eger, Ivan Habernal, Christian Stab & Iryna Gurevych. 2017. « What is the essence of a claim? cross-domain claim identification ». Palmer, Martha, Rebecca Hwa & Sebastian Riedel (eds). Proceeding of the 2017 Conference on Empirical Methods in Natural Language Processing, 2055– 2066

Dusmanu, Mihai, Elena Cabrio & Serena Villata. 2017. « Argument Mining on Twitter: Arguments, Facts and Sources », Palmer, Martha, Rebecca Hwa & Sebastian Riedel (eds). Proceeding of the 2017 Conference on Empirical Methods in Natural Language Processing, 2317–2322

Freeman, James B. 1991. Dialectics and the Macrostructure of Argument (Berlin: Foris)

Greco, Sara. 2023. « Twitter Activists’ Argumentation through Subdiscussions: Theory, Method and Illustration of the Controversy Surrounding Sustainable Fashion », Argumentation 37, 1-23

Green, Nancy. 2018. « Towards mining scientific discourse using argumentation schemes », Argument & Computation 9-2, 121–135

Klie, Jan-Christoph, Michael Bugert, Beto Boullosa, Richard Eckart de Castilho & Iryna Gurevych. 2018. « The Inception Platform: Machine-Assisted and Knowledge-Oriented Interactive Annotation », Proceeding of the 27th International Conference on Computational Linguistics: System Demonstrations, 5–9

Landis, J.R. & G.G. Koch. 1977. « The measurement of observer agreement for categorical data », Biometrics 33-1, 159–74

Lasser, Jana, Segun Taofeek Aroyehun, Almog Simchon, Fabio Carrella, David Garcia & Stephan Lewandowsky. 2022. « Social Media Sharing of Low-Quality News Sources by Political Elites », PNAS Nexus 1-4 [Online] https://doi.org/10.1093/pnasnexus/pgac186

Lawrence, John & Chris Reed. 2020. « Argument Mining: A Survey », Computational Linguistics 45-4 [Online] https://doi.org/10.1162/coli_a_00364

Levy, Ran, Yonatan Bilu, Daniel Hershcovich, Ehud Aharoni & Noam Slonim. 2014. « Context Dependent Claim Detection », Tsujii, Junichi & Jan Hajic (eds). Proceeding of the 25th International Conference on Computational Linguistics, 1489–1500

Moeschler, Jacques. 1989. « Pragmatic Connectives, Argumentative Coherence and Relevance ». Argumentation 3, 321– 339

Mochales Palau, Raquel & Marie-Francine Moens. 2009. « Argumentation Mining: The Detection, Classification and Structuring of Arguments in Text », Proceedings of the 12th International Conference on Artificial Intelligence and Law (New York : Association for Computing Machinery), 98–107

Peldszus, Andreas & Manfred Stede. 2013. « From Argument Diagrams to Argumentation Mining in Texts: A Survey. », International Journal of Cognitive Informatics and Natural Intelligence 7-1, 1-31

Rinott, Ruty, Lena Dankin, Carlos Alzate Perez, Mitesh M. Khapra, Ehud Aharoni & Noam Slonim. 2015. « Show Me Your Evidence : an Automatic Method for Context Dependent Evidence Detection », Màrquez, Lluís, Chris Callison-Burch & Jian Su (eds). Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing, 440–450

Schaefer, Robin & Manfred Stede. 2020. « Annotation and Detection of Arguments in Tweets », Cabrio, Elena & Serena Villata (eds). Proceeding of the 7th Workshop on Argument Mining, 53–58

Schaefer, Robin & Manfred Stede. 2022. « GerCCT: An Annotated Corpus for Mining Arguments in German Tweets on Climate Change », Calzolari, Nicoletta, Frédéric Béchet, Philippe Blache, Khalid Choukri, Christopher Cieri, Thierry Declerck, Sara Goggi, Hitoshi Isahara, Bente Maegaard, Joseph Mariani, Hélène Mazo, Jan Odijk & Stelios Piperidis (eds). Proceeding of the Thirteenth Language Resources and Evaluation Conference, 6121– 6130

Schaefer, Robin, René Knaebel & Manfred Stede. 2023. « Towards Fine-Grained Argumentation Strategy Analysis in Persuasive Essays », Alshomary, Milad, Chung-Chi Chen, Smaranda Muresan, Joonsuk Park & Julia Romberg (eds). Proceeding of the 10th Workshop on Argument Mining, 76–88

Sňajder, Jan. 2016. Social Media Argumentation Mining: The Quest for Deliberateness in Raucousness [Online] DOI:10.48550/arXiv.1701.00168

Stab, Christian & Iryna Gurevych. 2014. « Identifying Argumentative Discourse Structures in Persuasive Essays », Moschitti, Alessandro, Bo Pang & Walter Daelemans (eds). Proceeding of the 2014 Conference on Empirical Methods in Natural Language Processing, 46–56

Stede, Manfred & Jodi Schneider. 2019. Argumentation mining (Cham: Springer) [Online] https://doi.org/10.1007/978-3-031-02169-5

Stede, Manfred & Carla Umbach. 1998. « DiMLex: A Lexicon of Discourse Markers for Text Generation and Understanding, 36th Annual Meeting of the Association for Computational Linguistics and 17th International Conference on Computational Linguistics 2, 1238–1242

Wachsmuth, Henning, Nona Naderi, Yufang Hou, Yonatan Bilu, Vinodkumar Prabhakaran, Tim Alberdingk Thijm, Graeme Hirst & Benno Stein. 2017. « Computational Argumentation Quality Assessment in Natural Language », Lapata, Mirella, Phil Blunsom & Alexander Koller (eds). Proceeding of the 15th Conference of the European Chapter of the Association for Computational Linguistics 1, 176–187

Webber, Bonnie, Rashmi Prasad, Alan Lee & Aravind Joshi. 2019. The Penn Discourse Treebank 3.0 Annotation Manual (Philadelphia: Univ. of Pennsylvania)

Wührl, Amelie & Roman Klinger. 2021. « Claim Detection in Biomedical Twitter Posts », Demner-Fushman, Dina, Kevin Bretonnel Cohen, Sophia Ananiadou & Junichi Tsujii (eds). Proceeding of the 20th Workshop on Biomedical Language Processing, 131–142

Haut de page

Notes

1 For comprehensive overviews of the field of argument mining, see Lawrence and Reed (2020) or Stede and Schneider (2019).

2 Note that threads can be published incrementally or as one batch. In this work, however, we do not distinguish between the two cases.

3 According to Landis and Koch (1977), a κ value of 0.21-0.40 represents “fair”, 0.41-0.60 “moderate”, and 0.61-0.80 “substantial” agreement.

4 For data privacy reasons only tweet IDs can be freely distributed.

5 http://connective-lex.info

6 https://github.com/heartexlabs/label-studio

7 « Paris » refers to the Paris Agreement, i.e., the climate protection agreement made at the UN Climate Change Conference (COP21), which was held in Paris in 2015.

Haut de page

Pour citer cet article

Référence électronique

Robin Schaefer, Sophia Rauh et Manfred Stede, « On Complex Argumentation Structures in Tweets »Argumentation et Analyse du Discours [En ligne], 34 | 2025, mis en ligne le 10 avril 2025, consulté le 21 avril 2025. URL : http://journals.openedition.org/aad/9361 ; DOI : https://doi.org/10.4000/13q0j

Haut de page

Auteurs

Robin Schaefer

University of Potsdam (Germany)

Sophia Rauh

University of Potsdam (Germany)

Manfred Stede

University of Potsdam (Germany)

Haut de page

Droits d’auteur

CC-BY-NC-ND-4.0

Le texte seul est utilisable sous licence CC BY-NC-ND 4.0. Les autres éléments (illustrations, fichiers annexes importés) sont « Tous droits réservés », sauf mention contraire.

Haut de page
Rechercher dans OpenEdition Search

Vous allez être redirigé vers OpenEdition Search