Skip to navigation – Site map

HomeIssues20VariaMaking metaphor studies less WEIR...


Making metaphor studies less WEIRD. The methodological benefits and challenges of a greater diversity of languages and cultures in the study of metaphor

Simon Devylder


European languages represent 3% of the world 7,000+ languages, and yet, a recent survey shows that they amount for 84% of the studies in metaphor publications. The disproportionate reliance on such an infinitely small sample of the world diversity of languages seriously undermines the representativeness of generalizations about a phenomenon that is claimed to be found across languages and cultures. This paper is a call for more diversity in metaphor studies and a discussion about the methodological benefits and challenges posed by such ambition. Particularly relevant to increasing diversity in metaphor studies is the role of culture and context in metaphor identification procedures. This methodological question is addressed with a specific case study that shows how providing annotators with cultural and contextual information are key to increasing inter-rater agreements.

Top of page

Full text

1. Introduction: diversity, culture, and context

  • 1 Western European Industrialized Rich and Democratic.

1If metaphor is a universal phenomenon that goes across languages and cultures of the world, universal generalizations about this phenomenon must be formed on a representative sample of the world languages and cultures. A brief survey of the studies published in one of the most impactful journals in metaphor studies (i.e. Metaphor and Symbol), and over a period of five years (2015-2020), reveals that on a total of 84 studies, 71 investigated metaphors in WEIRD1 languages (i.e. 84% of the surveyed sample), 66 of which being on English alone. In contrast, only 14 studies investigated non-European languages (i.e. 16% of the surveyed sample). European languages represent approximately 3% of the estimated 7,000+ languages of the world. This tremendously low degree of representativeness of linguistic and cultural diversity forces us to face the inconvenient fact that the field has a major methodological problem. Making generalizations about any phenomenon on such an unrepresentative sample seriously limits the scope of the findings. What makes the disproportionate reliance on WEIRD languages and cultures even more problematic is that “members of WEIRD societies are among the least representative populations one could find for generalizing about humans” (Henrich, Heine & Norenzayan [2010: 61]).

2Dramatically increasing the representativeness of languages and cultures is thus not a trivial matter and needs to be considered rather urgently. Addressing the ‘diversity problem’ is however not about pointing the finger at the many qualitative contributions performed on these few European languages. On the contrary, the aim is to give more weight to these contributions by testing their replicability across a greater variety of languages and cultures. There is obviously a number of challenges to overcome in the endeavor. It is important to note, however, that accessibility to non‑European languages is not one of them. In 2021 for example, the Endangered Language Archive consisted of 675 collections of languages from Africa, Asia, Australia, Europe, Oceania, and the Americas that are openly accessible for download. These languages are documented in actual use, and the vast majority of collections are video-recordings of native speakers in a variety of situations. The video materials are almost systematically transcribed, glossed, and translated. In sum, accessibility to the diversity of the world languages is readily accessible and can therefore not be an excuse to keep the scope of metaphor studies limited to European languages and cultures. Their accessibility does not obviously shield the diversity endeavor from subsequent methodological challenges. One of these issues concerns the reliability with which annotators from a given culture manage to consistently identify metaphors in a corpus of another language and culture. In other words, what is the role of knowledge about the sociocultural background of the linguistic group whose language is under metaphorical investigation, if any?

3Beyond distinct and sometimes diverging approaches, the first shared principle that brings metaphor research together is the now accepted principle that theoretical generalizations should be supported by actual data. Whether the methods employed are quantitative and/or qualitative, data-driven metaphor studies all start with annotators deciding whether yes or no an expression found in a corpus should count as a metaphor. The quality of a data-driven metaphor study and the power of its resulting generalizing claims are first and foremost measured against the quality of the data it is based upon. Meaningful results are replicable results. If the outcome of a study S gives result R, (close) similar results have to be achieved by another team of annotators whom is given the same dataset and the same metaphor identification procedure. Although replicability will not ensure that one’s paper is more cited (Serra-Garcia & Gneezy [2021]), it will ensure that the results of a study do not depend on the subjective intuitions of the metaphor annotator (or on other factors) but can be replicated by any other annotator, and therefore have a high intersubjective validity. Neglecting to control the quality of this very first step of any data-driven metaphor study will inevitably cascade into a muddy pool of unreplicable results that ultimately sabotages the claims of an otherwise potentially valuable contribution.

  • 2 See Stampoulidis [2021: 86-91] for an extensive review and summary of most metaphor identification (...)

4A wide range of procedures for identifying metaphors have been developed for language (e.g., Cameron [2003]; Charteris-Black [2004]; Nacey et al. [2019]; Pragglejaz Group [2007]; Steen et al. [2010]), depiction (e.g., Forceville [2009]; Müller & Kappelhoff [2018]; Phillips & McQuarrie [2004]; Šorm & Steen [2018]) or gesture (e.g., Cienki [2017]). Nacey, Dorst, Krennmayr & Reijnierse [2019] specifically addressed the complexity of applying the MIPVU procedure to a number of different languages and laudably included studies on three non-European languages, which, as shown above, is rare enough to be noted. While we cannot review these procedures here for lack of space2, we can agree that they all serve the purpose of keeping the subjective intuitions of individual annotators on a short leash. Whether they successfully do so is another question that will not be addressed here, but a way to precisely assess ‘how short that leash is’ is to measure indexes of inter-rater agreements. Systematically measuring inter-rater agreements is a necessary step to control the intersubjective validity of a coded dataset. It has increasingly become the norm in methods of metaphor research as it is now widely accepted that low intersubjective validity undermines the general claims that are based on results that are difficult to replicate. Controlling for high intersubjective validity should thus be an equally high methodological priority for metaphor analysts operating within any theoretical frameworks.

5Theories of metaphor can be grouped with regards to their focal point and whether metaphor analysts define metaphor as a phenomenon that is predominantly found in the mind (e.g., Fauconnier & Turner [2002]; Gibbs [1994], Grady [1997], Kövecses [2000], Lakoff & Johnson [1980, 1999]), in discourse (Cameron & Deignan [2006], Deignan [2005], Kövecses [2016], [2020], Musolff [2006, 2016], Semino [2008], Steen [2008, 2015, 2017], Steen et al. [2010], Zinken [2007]), or in social interaction (Kolter et al. [2012], Deignan [2010], Jensen [2017, 2018], Jensen & Cuffari [2014], Jensen & Greve [2019], Müller [2008, 2011, 2017, 2019], Müller & Kappelhoff [2018]) (see Stampoulidis [2021: 42]). These levels of experience may be given different roles, importance, and articulated differently across frameworks but there seems to be a growing consensus that metaphors are not exclusively found in the mind, in discourse, or in social interaction: the metaphorization process goes across levels of experience. The premise of the consensus implies that the sociocultural conventions shared by a group of speakers under investigations and the specific context of communication from which the expressions of a corpus were extracted play an important, yet distinct, role in the metaphorization process. This is a premise for eco-friendly approaches to metaphor, for which the (sociocultural) environment (i.e. the ecology) and the social context of language use is central (e.g., Jensen [2017, 2018]; Jensen & Cuffari [2014]; Müller [2008, 2019]), for approaches where sociocultural conventions are defined as essential motivating forces giving rise to metaphors (e.g., Discourse Metaphor Theory), for approaches where the social nature of language use is essential (Cameron [2007, 2018]; Deliberate Metaphor Theory), or for approaches considering that metaphors are found at the level of actual communication and co-motivated by cognitive processes and sedimented sociocultural norms (Motivation and Sedimentation Model, Devylder & Zlatev [2020]; Stampoulidis, Bolognesi & Zlatev [2019]; Zlatev, Jacobsson & Paju [2021]). This would also be a premise for some Conceptual Metaphor Theory theoretical developments that extend the theory to also integrate culture and discourse factors (e.g., Gibbs [2017a]; Kövecses [2020]).

6Combined with this theoretical implication, the methodological consensus that data‑driven metaphor studies should have intersubjective validity entails that these two theoretical constructs (i.e. culture & context) should be operationalized in the metaphor identification process. In other words, if sociocultural conventions and context-specific information are considered to participate to the emergence of metaphors by a theory, then they should accordingly be implemented in the corresponding metaphor identification procedure used within a given theoretical framework.

7Such translation validity principle is not systematically implemented when it comes to control for how much sociocultural conventions and context-specific information account for the identification of a metaphor. Identifying a token as a metaphor or not ultimately boils down to the annotator’s implementation of the adopted metaphor identification procedure. If the metaphor identification procedure does not carefully control for the annotators’ knowledge about the sociocultural conventions and context-specific information, then an annotator can harness their own subjective intuitions about what they think they know about the sociocultural norm and the nature of the communicative context.

8We generally take it for granted that similarly trained annotators from any WEIRD country could equally well identify metaphors in languages and other sign systems of any other WEIRD culture. But is that really the case? Stampoulidis & Bolognesi [2019] showed that such assumptions should be reconsidered and found that lack of shared cultural background of even closely related European cultures (i.e. Greek & Italian) has a significant negative impact on inter-rater agreement. Not controlling for the amount of shared cultural background knowledge of the group of speakers under investigation, or for inferences drawn by the annotator from the context of communication therefore appears to be quite problematic and potentially constitutes an important vector of low intersubjective validity. As developed above, low intersubjective validity, as measured by inter-rater indexes, seriously undermines the replicability of a study and hence the generalizing and explanatory potential of such results.

9In sum, taking into account sociocultural conventions and context-specific information appears to be theoretically relevant to most approaches in metaphor research. Controlling the quality of the operationalization of these two dimensions in metaphor identification procedures would also, in theory, have an impact on intersubjective validity. The purpose of this paper is to test if this hypothesis can be verified empirically and to discuss what it implies methodologically. We test this hypothesis with a specific case study and find that culture and context are indeed key to increasing inter-rater agreements. The methods employed are described in Section 2, and the results of the case study are reported in Section 3. The latter are discussed in Section 4, before summarizing the methodological implications and concluding in Section 5.

2. Methods

2.1. Research questions

10Does a team of annotators use their knowledge about the sociocultural conventions of the group of speakers under analysis as well as information about the communicative context to inform their coding decision? In theory, as developed in the introduction, the sociocultural conventions shared by a group of speakers under investigation and the specific context of communication from which the expressions of a corpus are extracted should play an important role in the emergence of metaphors. From the analyst’s perspective, and for the sake of translation validity, these theoretically essential dimensions of the metaphorization process should therefore be implemented in the metaphor identification process. If the same information about sociocultural conventions and context of communication is indeed used to determine whether an expression X is a metaphor, we could reasonably expect annotators who were provided with the same information to agree with each other on that decision. In other words, this information will constrain their subjective intuitions to flow into the same stream. If, on the other hand, annotators do not use such information, then a group who were provided with this type of information will not have higher inter-rater agreement than a group who has no access to this information. The source of disagreement would then come from other external factors, and integrating contextual and cultural background in the identification process should only remain a secondary concern to identification procedure designers. This is an empirical question that I propose to address with the following experiment.

2.2. Material

11A metaphor identification task was designed using a corpus of expressions that were produced by a group of speakers from a Melanesian culture (Paamese culture, Vanuatu, South-Pacific). A sample of ten short video clips was extracted from the Black Sand Story collection (Devylder [2019]), which itself consists of 15 hours of transcribed, glossed, translated, and gesture-coded audio-video recordings on Paamese language and culture. These ten video-clips were more specifically extracted from kinship interviews where Paamese speakers were asked about various aspects of their family structure, tradition, and history. The video‑clips were selected with regards to their metaphorical potential and were equally distributed as targets, e.g. (1), and controls, e.g. (2).

(1) namusil sise onen mama onak (Paamese)
‘I followed my mother’s path’

(2) hesan anien akh hetal min ato (Paamese)
‘she will give me food and chicken’

12The second type of material produced for the experiment was a four-minute video about Paamese cultural background. This short introduction to Paamese culture was balanced with information that was relevant to the list of expressions to analyze (e.g. Paamese rules of arranged marriage) and irrelevant information (e.g. the geographical situation of Paama island). It is worth noting that this video provided information about culturally specific conventions but not about linguistic conventions (e.g. that the word sise ‘path’ is polysemous in Paamese).

2.3. Procedure

13Twelve participants were informed that they were going to be presented with different linguistic expressions in an Oceanic language. They were shown the ten video clips in random order and in the form of an online task. They were asked to decide whether or not a specific segment was a metaphor. Participants had no further instructions about the task. Once the metaphor identification task was fully performed, each participant was asked for feedback. Since this study aimed to contribute to metaphor research across metaphor theories, participants were not given any definition of metaphor and were not trained in any specific paradigm. The procedure was designed to address two specific questions. First, would an annotator need to know the context of communication in which a given speech segment is produced to assess whether the speaker is ‘talking about something in terms of something else’ (which we can assume is a broad definition of metaphor by non-metaphor experts)? Second, would an annotator need some culturally specific information to determine whether some elements of a speech segment are particularly meaningful in the given culture or not? For instance, does ‘following one’s path’ as in (1) or ‘giving food and chicken’ as in (2) resonate with sedimented conventions found in Paamese culture? Could then (1) and/or (2) potentially describe something else than ‘actual motion’ or ‘food giving’, respectively? To answer these two questions participants were equally distributed across four groups corresponding to four conditions as summarized in Table 1.

Table 1. Participants performed the task in four different conditions

Condition 1

Condition 2

Condition 3

Condition 4

Communicative Context





Sociocultural Background





14The participants who performed the metaphor identification task in the first condition (i.e. Group 1) were only showed videos of the speaker who produced the linguistic expression to identify a transcription of the segment, a simplified gloss, and a ‘literal’ English translation (see Figure 1). In other words, they could try to guess what the context of communication and the cultural background from the video and their own intuitions were, but they were not given any specific information about the sociocultural conventions of this particular group of speakers or about the context of communication.

Figure 1. Stimuli type for Group 1 & 3

Figure 1. Stimuli type for Group 1 & 3

15The participants who performed the task in the second condition (Group 2) were provided with the same material to annotate as participants from Group 1, but they were also given three specific pieces of information about the context of communication. They were told that these videos were interviews about family relations, and were provided with the question that was asked by the interviewer before the production of this specific speech segment. They were also given the right and left co-texts of the speech segment to analyze (see Figure 2).

Figure 2. Stimuli type for Group 2 & 4

Figure 2. Stimuli type for Group 2 & 4

16The participants who performed the task in the third condition (Group 3) were provided with the same material to annotate as participants from Group 1, but in contrast to Group 2 (i.e. no information about the context of communication), however, Group 3 participants watched the four-minute video about Paamese cultural background described above.

17Finally, participants who were asked to identify metaphors in the fourth condition (Group 4) were provided with both information about the context of communication and the sociocultural background of the speakers. Translated into these four conditions, the research questions described in 2.1. are operationalized into the following predictions:

  • Group 4 will have a higher inter-rater agreement index than Group 1;

  • Group 4 will have a higher inter-rater agreement index than Group 2 & 3;

  • Group 2 & 3 will have a higher inter-rater agreement index than Group 1;

  • Group 2 & 3 will not have different inter-rater agreements.

2.4. Participants

18Twelve participants were recruited at random within the relatively large population of Lund University Centre for Languages and Literature (Sweden) and on the basis of four criteria: a) they should be from a Western country and residing in one, b) they should not be knowledgeable about Melanesian cultures, c) they should have received an education in semantics, but d) should not be trained in metaphor studies (e.g. having done a class project, defended an MA thesis or PhD thesis with a focus on metaphor would disqualify them). The recruited participants’ lack of expertise in metaphor theories was supported by the feedback that they provided after the task, as there was no mention of any theory-specific concepts such as ‘cross-domain mapping’ or ‘deliberateness’, etc. Recruiting participants of European background who were not knowledgeable about Melanesian cultures was a way to precisely control the type and amount of knowledge about the sociocultural conventions and context-specific information. This will allow us to assess whether ‘sociocultural conventions’ and ‘context-specific information’ in metaphor identification can have an impact on inter-rater agreements.

3. Analysis

19The participants’ answers to the question “is this segment a metaphor?” were extracted and coded with a value (Condition 1, 2, 3, 4) corresponding to the group of annotators, in which the participant was placed. The twelve participants were equally divided in each of the four groups, so there were three annotators per group. Agreement between participants within each group was then measured with the Fleiss´s Kappa coefficient (Fleiss & Cohen [1973]), which is an extension from the Cohen Kappa measure of inter-rater agreement that is used for three or more raters of a categorical outcome. The coefficient K is measured as followed where Po is the observed agreement and Pe is the expected agreement:

20K can range from 0 to 1 where 0 is no agreement, or agreement that one can expect to find by chance and where 1 is perfect agreement. A coefficient over .75 is considered to be excellent agreement beyond chance, values between .40 and .75 are taken to reflect good agreement beyond chance, and a coefficient that is below .40 represent poor agreement beyond chance. The coefficients calculated for the four groups of annotators of the metaphor identification task are reported in Figure 3.

Figure 3. Inter-rater agreement indexes across four conditions

Figure 3. Inter-rater agreement indexes across four conditions

21As detailed above in Section 2.2., the twelve participants had to decide whether or not a segment of speech could be identified as a metaphor. The follow-up feedback survey showed that participants shared a basic understanding of metaphor that could be summarized as ‘an expression that says something in terms of something else’.

22Reconstructing the flow of subjective intuitions that plausibly guided participants to reach their decision about a specific speech segment (3) can inform the interpretation of the inter‑rater agreement coefficients reported above in Figure 3.

(3) Sise kowa inao nimahris (Paamese)
‘the path on which I come back’

23According to the Paamese traditional marriage system (Devylder [2019]), Paamese women are supposed to marry one of their father’s maternal uncle’s sons (i.e. FMBS for Father’s Mother’s Brother’s Son). When a woman marries her Paamese husband she leaves her clan’s territory to permanently join her husband’s. This incurs some kind of debt from Clan A towards Clan B. But thanks to the Paamese marital exchange system (i.e. ‘patrilineal dual exogamy’), this debt is paid after three generations. Indeed, three generations ago from that of the speaker, Clan B ‘lost’ one of its female clan members when the speaker’s paternal great-grandmother left Clan B to marry the speaker’s paternal great-grandfather from Clan A. Thanks to the union of the speaker and her husband, the great-grandmother that Clan B had lost has now been replaced by her great-granddaughter, three generations after. Paamese family relations are conventionally described in terms of sise ‘paths’ and this exchange system as motion along these paths. ‘Coming back on her great-grandmother’s family line’ is what the speaker means in (3) when she explains that she ‘comes back on the path’ to be with her husband, which is accompanied by an iconic gesture enacting a return movement of a figure on a landmark. Although this Paamese tradition is quite complex and probably difficult to fully grasp in a short segment of a short video, sharing the culture specific knowledge that family relations were conceptualized as paths in Paamese culture seemed to have been useful to Group 3 participants.

24Table 2 reports the decision each participant (A1-A12) made and the feedback they gave to explain their decision about this specific stimulus (3). The table also shows under which condition (i.e. group) each participant performed the task.

Table 2. Follow-up participant feedback

Table 2. Follow-up participant feedback

25As summarized in Table 1, each group of participants performed the identification task in four different conditions. Figure 4 is a reproduction of what participants saw on their screen. However, Group 1 & 3 were only provided with the orange-colored cells, while Group 2 & 4 had access to the additional contextual information. Group 3 & 4 also watched an introduction to Paamese culture and traditions before the task as described in 2.3.

Figure 4. Example of a segment to annotate by participants

Figure 4. Example of a segment to annotate by participants

26Figure 5 is a visualization of the flow of intuitions that the four groups of participants possibly went through to identify (3) as a metaphor or not. This reconstruction is based on the design of the experiment itself (+/- contextual information; +/- cultural background) and on the feedback given by each participant after having completed the task (see Table 2). The dashed lines represent the flow of their intuitions that is unconstrained from cultural and contextual knowledge. The solid lines represent an intuition that has potentially been constrained by information about the cultural and/or communicative context. The different line colours are intended to visualize the possible stages that the flow of intuition of each group possibly goes through. So, for example, once the [blue] flow of intuition of Group 4 participants goes through the [yellow] cultural knowledge it turns out [green].

Figure 5. Flow of subjective intuitions +/- constrained by contextual and cultural knowledge

Figure 5. Flow of subjective intuitions +/- constrained by contextual and cultural knowledge

27When participants/annotators from Group 1 had to decide whether (3) ‘the path on which I come back’ described something else than an actual motion event, they only had their own cultural background and a vague idea of the context of communication to hang onto. In other words, their individual subjective intuitions were not constrained by any knowledge about the culture of the speaker or the nature of the actual situation of communication. Because their intuitions were allowed to run free, they were also freer to vary across participants, which the inter-rater agreement coefficient reflects (k=0.33). Indeed, participants A1 & A2 decided that (3) described actual motion, in contrast to A3 who decided that (3) was a metaphor, the meaning of which was that the speaker would eventually re-unite with Keyla’s father. Participant A3’s interpretation is that ‘a path on which someone will come back’ must be describing some sort of marital reconciliation after a separation while using terms that can also be used to describe actual motion. This is a reasoning along these lines that probably motivated A3’s decision to identify this expression as a metaphor. Due to the lack of contextual and cultural knowledge, A1, A2, A3 had to make wild guesses. A3 possibly anchored their reasoning to the knowledge of their own culture-specific conventions where relationships are often talked about in terms of actual motion pertaining to the famously CMT-coined cross-domain mapping of “life is a journey”, and hereby interpreted that the Paamese speaker meant she was going to reconcile with Keyla’s dad. According to the explanation about Paamese marital traditions provided above, this WEIRDocentric interpretation is not what the Paamese speaker meant.

28Group 2 participants had to make the same decision, but in contrast to Group 1 they were given information a), b), c), d) about the communicative context. They were told that a) this speech segment was part of a kinship interview where b) the speaker was asked how she had met her husband. Group 2 participants were also informed that the segment was immediately preceded by c) a description of some sort of interaction between her husband’s family and her side of the family (‘my husband’s dad put his name to the side of my dad Cyrus’). The right co-text (d) seems to be a structural description of the Paamese speaker’s side of the family (i.e. ‘on my dad side, my aunty is only one’). Information a) and b) were useful to understand that the segment to analyze was not embedded in a conversation about some kind of spatial description of an actual path that would lead to somebody’s house. According to Group 2 participants’ feedback they all identify some sort of incongruity between a description of actual motion ‘the path on which I come back’ and the communicative context about family relations. However, while (3) can easily be anchored into the contextual information a) and b), it is less obvious to a non-Paamese person how to relate c) and d) to the segment ‘the path on which I come back’. In sum, the contextual information provided to Group 2 participants allowed them to identify that (3) was ‘an expression that says something in terms of something else’, but they could not really make sense of it as participant A6 explains “[…] it’s unclear what’s the intended meaning in this case”. In a sense, it is as if deciphering the meaning of an expression acts as a ‘feedback loop’ that would confirm that the already identified incongruity between the meaning of an expression (e.g. actual motion) and its context (e.g. something about family relations) is not just a case of polysemy but of metaphor. Participants A5 and A6 seemed hesitant (“she seems to mean”; “it seems to refer”) in making their diverging decisions to identify (3) as a non‑metaphor, and as a metaphor, respectively. This specific lack of agreement lowered the Kappa of Group 2, and one could perhaps predict that it is the lack of a ‘feedback loop’, which cultural information provides, that has a systematic negative impact on inter-rater agreement coefficients.

29Group 3 participant performed the identification task in the third condition, which did not provide them with the contextual information a) – d) but with cultural background information instead. As described above, the short introductory video was a mix of relevant and irrelevant information about Paamese culture. A relevant piece of information was a simplified introduction to Paamese traditional rules of marriage. The explanation consisted of a voice‑over and a visualization of a family tree diagram, a screenshot of which is reproduced in the cultural background ellipse in Figure 5. Group 3 participants seem to have make use of their recently acquired knowledge of Paamese culture, according to the feedback they provided (see Table 2). Although Group 3 participants were not provided with contextual information, the segment contains a hint that the speaker is talking about some sort of family relations when she mentions ‘Keyla’s dad’. Combined with the cultural background knowledge, it seemed to have directed the participants’ flow of intuitions in the same direction: ‘the path on which I come back’ must be referring to something else than actual motion and according to the cultural background video that ‘something else’ is the Paamese intertribal marriage exchange system.

30The participants from Group 4, who were provided with both contextual and cultural background information, were equipped with, on the one hand, a way to reconstruct the two potential interpretations of ‘the path on which I come back’(“Genealogical path vs. physical path” as A9 describes) and the incongruity of one interpretation (i.e. actual path) in the context of a kinship interview, and on the other hand, with knowledge about the Paamese conventions which allowed them to reconstruct the meaning of the segment, and hereby complete the ‘feedback loop’ that Group 2 participants were missing. In sum, when participants from Group 4 were asked to identify this specific segment as a metaphor or as a non-metaphor, their individual subjective intuitions and reasoning paths were very much constrained by the information they were given about the contextual and cultural background. Importantly they were constrained in the same way, as the type and amount of shared knowledge was controlled. Taken together, the prediction that Group 4 would reach a higher inter-rater agreement than the other groups was well founded and indeed confirmed.

4. Discussion

31This experiment should of course be replicated multiple times before drawing definitive conclusions. But for the time being, the results of this case study give a clear YES to the question “does a team of annotators use their knowledge about the sociocultural conventions the group of speakers under analysis as well as information about the communicative context to inform their coding decision?”. More specifically these results tell us that prediction 1 was verified. The three annotators who were provided with information about the sociocultural background of Paamese speakers, and with information about the communicative context (Group 4) had higher inter-rater agreement (k=0.73) than the three annotators who were provided with no such information (Group 1; k=0.33). This variation from poor agreement beyond chance (k=0.33) to good – almost excellent – agreement beyond chance (k=0.73), suggests that the independent annotators from Group 4 used the information provided to them about the sociocultural conventions and context of communication to decide if a speech segment was a metaphor or not. Prediction 2 is also verified. The annotators who were provided with both types of information about the sociocultural background of Paamese speakers agreed more with each other than the annotators who were provided with only one type of information (Group 2 & 3). The latter annotators (i.e. from Group 2 & 3) managed to agree more with each other than the Group 1 whose subjective annotations were left unleashed, and who consequently poorly agreed with each other, hence confirming prediction 3. Prediction 4 was however not verified, and participants/annotators appeared to make better use of the cultural background information than the information about the communicative context. We nonetheless cannot conclude at this stage that information about the cultural background ‘is more important’ in ensuring higher intersubjective validity than information about the communicative context.

32An interesting phenomenon seemed to have been brought to light by the qualitative analysis of the participants’ answers and feedback to a specific segment they had to analyze (Figure 5), and that we could identify as a type of “feedback loop”. What seems to have helped participants from Group 4 to agree with each other more than the other groups is perhaps not only the maximum amount of information that they were given (contextual + cultural information) but also the complementarity of both types of information in the metaphor identification process. To go back to example (3) (‘the path on which I come back’), we could hypothesize that the first step of the annotator’s metaphor identification process is the identification of ‘some sort of incongruity’. This phenomenon is inherently relational and must be an incongruity of something. Therefore, if that ‘something’ is absent, there is no incongruity to begin with. I propose that the value of that ‘something’ is the communicative context: if the meaning of an utterance (e.g., here a motion event) is somewhat odd, unexpected, or incongruous to the communicative context (e.g., here the answer to a question about the speaker’s husband), then a red flag is raised. However, according to the results of the study and the comments provided by the participants of the various groups, this flag raising is only the first side of a loop that needs to be completed by a second type of information. The latter can be identified with knowledge about the sociocultural background. In the case of (3) participants who were given information about Paamese marital practices were able to confirm that ‘the path on which I come back’ was indeed referring to marriage rules in the terms of an actual motion event. To use a metaphor, the metaphor identification process could be described as a kind of system where the annotator receives a multitude of inputs from the material under analysis. When incongruity between the meaning of an utterance and the communicative context is detected a branch of the network is activated and asks for feedback: can the meaning of this segment have an analogical relationship with the communicative context and be incongruous with it at the same time? The answer to that question is the output, and similarly to the process of a feedback loop it becomes an input in the system of metaphor identification: if the feedback is positive, the annotator will be confident that this segment is a metaphor, less so or not at all if the feedback is negative or absent. The results of the study presented in this paper indicate that sociocultural knowledge provides this feedback to the loop that is initiated by contextual information in relation to the meaning of the segment under identification. Should this process have a psychological reality, a possible explanation why participants of Group 3 performed better than those of Group 2 could be that all participants were expecting incongruity as they knew metaphors were most likely to occur rather frequently (in reality half of the time) in the task. Therefore, even though Group 3 participants did not have contextual information, its “incongruity-flagging” role was perhaps at least partially already activated by the experimental setting. The feedback loop would then be completed thanks to the information about cultural knowledge that Group 3 participants were provided with, but which was kept from Group 2 participants.

5. Summary and conclusion

33To summarize and conclude, this paper has hopefully convinced metaphor researchers to take home three important pieces of information. First, it is in the best interest of metaphor studies as a field of investigation to actively include a much greater diversity of non-European languages and cultures. Since such data is openly accessible, it is both up to individual researchers and editors of metaphor journals to implement such a practice that would benefit the field as a whole: more replicable findings across languages and cultures make such findings more robust and give the field more scientific rigor. Secondly, implementing this call for diversity sheds light on a methodological issue that should be considered by studies on all languages and types of data. Since it may be less obvious that annotators from seemingly close cultural backgrounds may not share the same sociocultural conventions, WEIRD metaphor studies may arguably be even more at risk: controlling the quality of intersubjective validity by constraining the subjective intuitions of metaphor annotators should be systematic. This intersubjective quality control cannot be based on just any principles. For instance, asking a team of annotators to only count as metaphors “six letter words that starts with an A” would drastically improve inter-rater agreements, but would not tell us much about metaphors. This is why the translational validity of theoretical constructs into metaphor identification procedures matters (Zlatev & Moskaluk, in press). The importance of sociocultural conventions shared by a group of speakers under investigations and the specific context of communication is now recognized as key by most metaphor theories. The results of the presented case study demonstrate the benefits of operationalizing this theoretical consensus into metaphor identification procedures: providing annotators with a homogeneous cultural and contextual information increase inter-rater agreements, a common measurable proxy for intersubjective quality control. As a sidenote, it is almost surprising that annotators with so little theoretical background in metaphor research and an almost non-existent identification procedure performed relatively well in agreeing with each other. Whether this tells us something about implementing a parsimony principle in metaphor definitions and identification procedures is a discussion that will have to be postponed but perhaps deserves to be picked up by future research. Third, the presented case study has opened up to another question to be further investigated. Do the mechanisms of what I suggested to identify as a feedback loop, which is initiated by contextual information and completed by sociocultural information, have some sort of psychological reality? Such hypothesis perhaps deserves to be tested as it may have interesting methodological implications, particularly in metaphor identification procedures.

Top of page


Black Max 1979, “More about metaphor”, in Ortony A. (Ed.), Metaphor and thought, Cambridge University Press.

Cameron Lynne, 2003, “Metaphor in educational discourse, advances in applied linguistics”, Bloomsbury Academia, London: Continuum.

Cameron Lynne & Deignan Alice, 2006, “The emergence of metaphor in discourse”, Applied Linguistics 27(4), 671-690.

Charteris-Black Jonathan, 2004, Corpus approaches to critical metaphor analysis, London: Palgrave Macmillan.

Cienki Alan, 2008, “Why study metaphor and gesture”, in Cienki A. & Müller C. (Eds.), Metaphor and gesture, Amsterdam: John Benjamins, 5-25.

Cienki Alan, 2017, “Analysing metaphor in gesture: A set of metaphor identification guidelines for gesture (MIG-G)”, in Semino E. & Demjén Z. (Eds.), The Routledge handbook of metaphor and language, Abongdon: Routledge, 131-147.

Deignan Alice, 2005, Metaphor and Corpus Linguistics, Amsterdam: John Benjamins.

Deignan Alice, 2010, “The evaluative properties of metaphors”, in Low G., Todd Z., Deignan A. & Cameron L. (Eds.), Researching and applying metaphor in the real world, Amsterdam: John Benjamins, 357-374.

Devylder Simon, 2018, “Diagrammatic iconicity explains asymmetries in Paamese possessive constructions”, Cognitive Linguistics 29(2), 313-348.

Devylder Simon, 2019, “Black sand stories: a polysemiotic and multimodal documentation of Paamese sand stories, a critically endangered tradition of Vanuatu”, London: SOAS University of London, Endangered Languages Archive, Handle:

Devylder Simon & Zlatev Jordan, 2020, “Cutting and Breaking Metaphors of the Self and the Motivation and Sedimentation Model”, in Baicchi A. (Ed), Figurative Meaning Construction in Thought and Language, Amsterdam: Benjamins, 253-282.

Fauconnier Gilles & Turner Mark, 2002, The way we think: Conceptual blending and the mind’s hidden complexities, Basic Books.

Fleiss Joseph L. & Cohen Jacob, 1973, “The equivalence of weighted kappa and the intraclass correlation coefficient as measures of reliability”, Educational and Psychological Measurement 33, 613–619.

Forceville Charles, 2009, “Non-verbal and multimodal metaphor in a cognitivist framework: Agendas for research”, in Forceville C. & Uriós-Aparisi E. (Eds.), Multimodal metaphor, Berlin: De Gruyter, 19-42.

Gibbs Raymond, 1994, The poetics of mind: Figurative thought, language, and understanding, Cambridge University Press.

Gibbs Raymond, 2017a, “The embodied and discourse views of metaphor: Why these are not so different and how they can be brought closer together”, in Hampe B. (Ed.), Metaphor: Embodied cognition and discourse, Amsterdam: John Benjamins, 319-365.

Gibbs Raymond, 2017b, Metaphor wars: Conceptual metaphors in human life, Cambridge University Press.

Grady Joseph, 1997, Foundations of meaning: Primary metaphors and primary scenes, University of California, Berkeley.

Henrich Joseph, Heine Steven & Norenzayan Ara, 2010, “The weirdest people in the world?”, Behavioral and brain sciences 33(2-3), 61-83.

Itkonen Esa, 2005, Analogy as structure and process: approaches in linguistics, cognitive psychology and philosophy of science, John Benjamins.

Jensen Thomas & Cuffari Elena, 2014, “Doubleness in experience: Toward a distributed enactive approach to metaphoricity”, Metaphor and Symbol 29(4), 278-297.

Jensen Thomas & Greve Linda, 2019, “Ecological cognition and metaphor”, Metaphor and Symbol 34(1), 1-16.

Jensen Thomas, 2017, “Doing metaphor: An ecological perspective on metaphoricity in discourse”, in Hampe E. (Ed.), Metaphor: Embodied cognition to discourse, Cambridge University Press, 257-276.

Jensen Thomas, 2018, “The world between us: The social affordances of metaphor in face-to-face interaction”, RASK Internationalt tidsskrift for sprog og kommunikation 47, 45-76.

Kövecses Zoltan, 2000, “Metaphor and emotion”, Language, culture, and body in human feeling, Cambridge University Press.

Kövecses Zoltan, 2016, “Conceptual metaphor theory”, in Semino E. & Demjén Z. (Eds.), The Routledge handbook of metaphor and language, Taylor and Francis, 13-27.

Kövecses Zoltan, 2020, Extended conceptual metaphor theory, Cambridge University Press.

Lakoff Georges & Johnson Mark, 1980, Metaphors we live By, The University of Chicago Press.

Lakoff Georges & Johnson Mark, 1999, Philosophy in the flesh: The embodied mind and its challenge to western thought, Basic Books.

Müller Cornelia, 2008, Metaphors dead and alive, sleeping and waking: A dynamic view, University of Chicago Press.

Müller Cornelia, 2011, “Are ‘deliberate’ metaphors really special? Deliberateness in the light of metaphor activation”, Metaphor and the Social World 1(1), 61-66.

Müller Cornelia, 2017, “Why mixed metaphors make sense”, in Gibbs R. W. (Ed.), Mixing metaphor, Amsterdam: John Benjamins, 31-56.

Müller Cornelia, 2019, “Metaphorizing as embodied interactivity: What gesturing and film viewing can tell us about an ecological view on metaphor”, Metaphor and Symbol 34(1), 61-79.

Müller Cornelia & Kappelhoff Hermann, 2018, Cinematic metaphor: Experience-Affectivity-Temporality, Berlin: De Gruyter.

Musolff Andreas, 2006, “Metaphor scenarios in public discourse”, Metaphor and Symbol 21(1), 23-38.

Musolff Andreas, 2016, Political metaphor analysis: Discourse and scenarios, Bloomsbury Publishing.

Nacey Susan, Dorst Aletta, Krennmayr Tina & Reijnierse Gudrun, 2019, Metaphor Identification in Multiple Languages: MIPVU around the world, John Benjamins Publishing.

Phillips Barbara & McQuarrie Edward, 2004, “Beyond visual metaphor: A new typology of visual rhetoric in advertising”, Marketing Theory, 4(1–2), 113-136.

Pragglejaz Group, 2007, “A practical and flexible method for identifying metaphorically used words in discourse”, Metaphor and Symbol 23(1), 1-39.

Semino Elena, 2008, Metaphor in discourse, Cambridge: Cambridge University Press.

Serra-Garcia Marta & Gneezy Uri, 2021, “Nonreplicable publications are cited more than replicable ones”, Science Advances Vol. 7/21, eabd1705

Šorm Ester & Steen Gerard, 2018, “VISMIP: Towards a method for visual metaphor identification”, in Steen G. (Ed.), Visual metaphor: Structure and process, John Benjamins: 47-88.

Stampoulidis Georgios, 2021, Street Artivism on Athenian Walls: A cognitive semiotic analysis of metaphor and narrative in street art (Doctoral dissertation, Lund University).

Stampoulidis Georgios & Bolognesi Marianna, 2019, “Bringing metaphors back to the streets: A corpus-based study for the identification and interpretation of rhetorical figures in street art”, Visual Communication.

Stampoulidis Georgios, Bolognesi Marianna & Zlatev, Jordan, 2019, “A cognitive semiotic exploration of metaphors in Greek street art”, Cognitive Semiotics 12(1).

Steen Gerard, 2008, “The paradox of metaphor: Why we need a three-dimensional model of metaphor”, Metaphor and Symbol, 23(4), 213-241.

Steen Gerard, 2015, “Developing, testing and interpreting deliberate metaphor theory”, Journal of Pragmatics 90, 67-72.

Steen Gerard, 2017, “Deliberate Metaphor Theory: Basic assumptions, main tenets, urgent issues”, Intercultural Pragmatics 14(1), 1-24.

Steen Gerard, Dorst Aletta, Herrmann Berenike, Kaal Anna, Krennmayr Tina & Pasma Tryntje, 2010, A method for linguistic metaphor identification, Amsterdam: John Benjamins.

Zinken Jörg, 2007, “Discourse metaphors: The link between figurative language and habitual analogies”, Cognitive Linguistics 18(3), 445-466.

Zlatev Jordan, Jacobsson Göran & Paju Liina, 2021, “Desiderata for metaphor theory, the Motivation & Sedimentation Model and motion-emotion metaphoremes”, in Soares da Silva A. (Ed.), Figures: Intersubjectivity and usage, Amsterdam: John Benjamins, 41-74.

Zlatev Jordan & Moskaluk Kalina, 2022, “Translation validity in metaphor theories”, Figurativity and Human Ecology 17, 123-148.

Top of page


1 Western European Industrialized Rich and Democratic.

2 See Stampoulidis [2021: 86-91] for an extensive review and summary of most metaphor identification procedures.

Top of page

List of illustrations

Title Figure 1. Stimuli type for Group 1 & 3
File image/png, 498k
Title Figure 2. Stimuli type for Group 2 & 4
File image/png, 660k
File image/png, 2.7k
Title Figure 3. Inter-rater agreement indexes across four conditions
File image/jpeg, 28k
Title Table 2. Follow-up participant feedback
File image/jpeg, 66k
Title Figure 4. Example of a segment to annotate by participants
File image/png, 732k
Title Figure 5. Flow of subjective intuitions +/- constrained by contextual and cultural knowledge
File image/jpeg, 342k
Top of page


Electronic reference

Simon Devylder, Making metaphor studies less WEIRD. The methodological benefits and challenges of a greater diversity of languages and cultures in the study of metaphorLexis [Online], 20 | 2022, Online since 29 December 2022, connection on 10 December 2023. URL:

Top of page

About the author

Simon Devylder

UiT, the Arctic University of Norway

Top of page



The text only may be used under licence CC BY-SA 4.0. All other elements (illustrations, imported files) are “All rights reserved”, unless otherwise stated.

Top of page
Search OpenEdition Search

You will be redirected to OpenEdition Search