Navigation – Plan du site

AccueilNuméros31Anaphoric Distance in Oral and Wr...

Anaphoric Distance in Oral and Written Language: Experimental Evidence

La distance anaphorique dans la langue orale et écrite : preuve expérimentale
Berfin Aktaş et Manfred Stede

Résumé

We investigate the variation in oral and written language in terms of anaphoric distance (i.e., the textual distance between anaphors and their antecedents), expanding corpus-based research with experimental evidence. Contrastive corpus studies demonstrate that oral genres include longer average anaphoric distance than written genres, if the distance is measured in terms of clauses (Fox, 1987; Aktaş & Stede, 2020). We designed an experiment in order to examine the contrasts in oral and written mediums, using the same genre. We aim to gain more insight about the impact of the medium, in a situation where both mediums convey a similar level of spontaneity, informality and interactivity. We designed a story continuation study, where the participants are recruited via crowdsourcing. To our knowledge, this is the first study of its kind, where anaphoric distance is manipulated systematically in a language production experiment in order to examine medium distinctions. We observed that participants use more pronouns in oral medium than in written medium if the anaphoric distance is long. This result is in line with the implications of the earlier corpus-based research. In addition, our results indicate that anaphoric distance has a larger effect in referential choice for the written medium.

Haut de page

Texte intégral

We thank the anonymous reviewers for their insightful comments and suggestions, which have helped us significantly improve the paper. We would also like to thank Anna Laurinavichyute for her helpful advice on the statistical modeling and Claudia Felser for her insightful comments on the experiment design and earlier drafts of this paper.
This work is funded by the Deutsche Forschungsgemeinschaft (DFG, German Research Foundation) – Projektnummer 317633480 – SFB 1287, Project A03.

1. Introduction

  • 1 We use the terms “oral” and “written” to refer to the production medium. We preferred to (...)

1We are investigating the variation in oral1 and written English in terms of coreference-related features of language. One of the most studied aspects of coreference is the textual distance, either linear or hierarchic, between the referential expressions denoting the same entity (henceforth, anaphoric distance). In this paper, we aim to expand existing corpus-based research on anaphoric distance with experimental evidence.

2The distance between anaphors and their antecedents (the closest previous mention of the same referent) is considered one of the most important factors influencing referential choice (Ariel, 1990; Chafe, 1976, 1994; Garrod & Sanford, 1982; Givón, 1983). Experimental evidence indicates that anaphoric distance affects the comprehension effort for the resolution of referential expressions. For instance, extending the anaphoric distance in an experimental setup increases the comprehension time for the referential expressions (Clark & Sengul, 1979; Streb et al., 2004).

3A number of contrastive corpus-based studies investigate the quantitative properties of anaphoric distance across oral and written mediums by analysing various genres such as telephone conversations, broadcast conversations, and news texts (cf. e.g., Fox, 1987; Biber, 1992; Amoia et al., 2012; Kunz et al., 2016; Aktaş et. al., 2019; Aktaş & Stede, 2020). Corpus based research demonstrates that there are differences between oral and written corpora with respect to anaphoric distance. For instance, Fox (1987) computes the linear anaphoric distance in terms of clauses and finds that the average anaphoric distance between third person pronouns and their antecedents is longer in oral texts than in written texts.

4Oral-written medium comparison in terms of coreference variation shows greater contrasts for the intrinsically different genres. For instance, Aktaş and Stede (2020) compare different genres in terms of average clause-based anaphoric distance and observe that the difference is more prominent for telephone conversations and news texts (2.97 vs. 1.45) than for broadcast news and written news (1.66 vs. 1.45). Although telephone conversations and broadcast news are both produced verbally, written news differs significantly from telephone conversations regarding spontaneity and informality, but not from broadcast news. Furthermore, they observed that full noun phrases (e.g., proper names, definite and indefinite noun phrases) are more frequent than third person pronouns for both mediums, with third person pronouns being more common in oral than in written genres.

5In addition, oral and written texts diverge in terms of textual features. Certain discourse markers (e.g., mhm, oh yeah), for example, are widespread in oral conversations but uncommon in news texts and oral genres are more intense in terms of the number of clauses than written genres (Aktaş & Stede, 2020). In another comparative study, Schäpers (2009: 137) examines oral and written corpora that are composed of the same number of words. Schäpers also reports that oral texts have a higher number of clauses than do written texts. Consequently, clauses in written texts are longer. In addition, Schäpers notices discrepancies between mediums regarding how the clauses are linked to each other. For instance, coordination is more common in oral language, whereas in written texts, subordinate clauses are more common.

6Akinnaso (1982) argues that comparison of the mediums (spoken vs. written) should be conducted on the texts that are produced by the same individuals, exhibit same degree of planning, formality, interactivity (e.g., monologue or dialog) and belong to the same genre. In line with Akinnaso’s argument, in this paper, we have the motivation that comparing the oral and written mediums for the same genre, in a setting where the textual differences (e.g., clause length and clause types) are eliminated, can provide more insight about the impact of the medium, when both mediums convey a similar level of spontaneity, informality and interactivity2. Therefore, in order to verify the results of the earlier corpus studies (e.g., oral medium allows more pronoun usage in longer anaphoric distances in comparison to written medium) and get a more precise account of the distance question within a single genre, we designed a crowdsourcing story continuation study using a crowdsourcing platform3. In this study, participants are expected to produce texts, in either oral or written form, and use referring expressions denoting the main character in the story. We are examining production biases in response to oral and written stimuli, where clause-based anaphoric distance is manipulated systematically.

7Our first hypothesis is that for all noun phrases in responses, oral responses will include more frequent use of pronouns than written responses. Our second and main hypothesis is a specific version of the first hypothesis: oral responses will include more frequent pronoun usage as a first reference to the main character than written responses for long stimuli. (i.e., oral responses allow for longer anaphoric distance). In addition, we aim to provide quantitative results on various aspects of the oral-written medium contrast.

8This paper is structured as follows: in Section 2, we present relevant previous work in the literature. Section 3 describes the design settings and execution procedures for running and evaluating the experiment. We report the results of the experiment in Section 4, employing descriptive and inferential statistics. In Section 5, we discuss our findings. Section 6 concludes and explores future directions.

2. Related work

9Various contrastive corpus-based studies investigate the differences in terms of average anaphoric distance across oral and written mediums. Since the experimental factor in our study is the linear textual distance in terms of clauses (i.e., clause-based distance), we include only the studies investigating the contrasts on clause-based anaphoric distance.

10Fox (1987) examines the use of anaphoric third person singular human references (pronouns and noun phrases) in oral and written texts. The data is composed of conversations (face-to-face and telephone conversations) and expository prose (newspaper articles and psychoanalytic biography). Fox reports that conversations allow longer clausal distances between pronouns and their antecedents than written texts do (2.52 clauses in conversations vs. 1.21 in written texts). Aktaş and Stede (2020) undertake a corpus study involving various genres such as telephone conversations, news and Twitter conversations. Among other measures, they compare the oral and written genres in terms of clause-based anaphoric distance. Their findings are compatible with Fox (1987). In their analysis, the difference is more prominent for conversational texts and news texts (2.97 vs. 1.45) than for broadcast news and written news (1.66 vs. 1.45), although conversational texts and broadcast news are both produced in the oral medium.

11In addition to the anaphoric distance, Aktaş and Stede (2020) compare the oral and written genres also in terms of the relative distribution of nominal expressions according to their syntactic categories (i.e., pronouns vs. non-pronominal noun phrases [NPs]). Their findings indicate that third person pronouns are more frequent in oral conversations than in written news (33% vs. 11%, according to Table 2 in that paper). Fox (1987) compares the frequencies of pronouns and full NPs in settings with no interfering referent, with an interfering referent of a different gender, and an interfering referent of the same gender. Her findings show that in the absence of an intervening referent, pronouns are used more intensely for both environments (64% in written vs. 94% in spoken language) compared to full NPs. The use for pronoun decreases when there exist interfering referents. When a referent of the same gender interferes, for example, the usage of pronouns drops to 13% in written texts and 57% in spoken texts.

12In previous experimental research, the impact of anaphoric distance on pronoun processing has so far been studied within a single medium, where the anaphoric distance is modified to manipulate the working memory. Clark and Sengul (1979) demonstrated that text distance between antecedents and the anaphor affects reading time. In their experiments, participants took less time comprehending a sentence when the referents of such noun phrases were mentioned one sentence back than when they were mentioned two or three sentences back. Streb et al. (2004) conducted experiments in which they manipulated the number of intervening words between the anaphor and its antecedent, hence altering the anaphoric distance. In line with Clark and Sengul (1979), they found that comprehension time increases with the expanded distance. This effect held for both pronouns and proper names. They also reported that the cases of “medium” (antecedent in the previous sentence) and “far” (antecedent 2 sentences before) are not distinguishable.

13In contrast to such studies, we examine the variation between oral and written narrative discourse with a production experiment. To the extent of our knowledge, there is no previous work with the same goal and strategy.

3. Experiment

14The starting point of this experiment is the assumption that heavier expressions (e.g., proper names) are produced more often than lighter ones (e.g., pronouns) when a referent is less active or salient; this has been validated by multiple studies (e.g., Ariel, 1990; Arnold, 1998; Chafe, 1976). The anaphoric distance is highly correlated with accessibility or salience in these studies. We here investigate the question of whether there is a difference regarding the referential choice across oral and written mediums, related to the varying length of anaphoric distance. In our work, we build on the study of Fox (1987) and Aktaş and Stede (2020) and aim to provide empirical evidence on medium distinctions in language production regarding the anaphoric distance. We hypothesize that oral responses in our story continuation experiment will include more prominent pronoun usage than will written responses for long anaphoric distance.

  • 4 There exist different forms of crowdsourcing, such as microworking and game-wit (...)
  • 5 Despite its popularity, crowdsourcing platforms also get criticisms, which we acknowledge (...)

15The interface of this experiment is designed and implemented as a crowdsourcing study. Crowdsourcing is intensively used to generate language resources for NLP (Natural Language Processing) tasks, such as argument mining (Lavee et al., 2019) or coreference annotation (Poesio et al., 2019)4. These studies and many others (e.g., Enochson & Culbertson, 2015; Iskender et al., 2021) reported promising quality assurance results for the crowdsourced data, and hence validated the effective use of crowdsourcing even for complicated NLP tasks, such as argument annotation5.

  • 6 The authors of this study live in a country where English is not the native language of t (...)

16We preferred to use crowdsourcing instead of a laboratory setting because crowdsourcing platforms provide an efficient way for reaching the native speakers of English located in English speaking countries, from both financial and organisational perspectives6. We recruited native speakers of English for our experiment to ensure a comparable level of English competency for all participants. Crowd workers participating in this study were paid according to the minimum hourly wage of the country of their residence.

17Our task (i.e., story continuation) does not require expertise in any specific linguistic field. That being the case, in order to ensure the quality of the responses, we only needed to confirm the linguistic competence and the engagement of the participants in the study. Therefore, we recruited the participants via a qualification study (see Section 3.1.1). We created a pool of crowd workers after evaluation of the responses collected in the qualification phase and then used that pool to recruit the participants for the actual experiment.

18The experimental factor in the main experiment is the length of the short stories that are used as the stimuli for triggering the continuations. The test conditions of the experiment are constructed by manipulating the story lengths as 1, 2, 3, 5, and 8 clauses. The design principles applied in the creation of the stimuli are presented in Section 3.3.2. In order to validate the design of the experimental settings (e.g., stimuli design, instruction wording, etc.), we conducted a pilot study before the actual experiment (see Section 3.2). The pilot study provided practical hints for refining the experiment stimuli so that they can more effectively serve our purposes. We conducted the main experiment 8 weeks after the pilot study. The details of the main experiment are given in Section 3.3.

3.1. Participant qualification study

  • 7 Stimuli used in the qualification task are less restricted than the stimuli in the actu (...)

19The qualification study was implemented as one crowdsourcing task (i.e., a set of stimuli/questions which should be completed and submitted by a crowd worker in one attempt). It is a smaller version of the actual experiment with fewer stimuli7, yet containing additional questions for qualification purposes. Participants of this study were expected to continue the short stories in the stimuli, which were provided in either audio or written form. In addition to the story continuation sections, comprehension questions were also inserted into the workflow in order to check the engagement of the crowd workers. Only the workers who declared being a native speaker of English were allowed to complete the qualification study.

20The first page in the qualification study is the information page and contains the description in [1].

[1] Native English speakers: Qualify for “Story Continuation” tasks and earn up to 10 Euro today by continuing very short stories (up to 3 sentences). This is your qualification test. Duration is approx. 6-8 minutes.
Requirements: English native speakers only; Microphone and speaker or headset needed.

On the next page, we describe the experiment and get a confirmation from the crowd workers about whether they are native speakers of English. We used the instructions in [2] and [3] for this purpose.

[2] Dear participants, thanks for taking part in this qualification for our story continuation micro task series. Here, you will see 6 very short stories in between 1 to 8 sentences of length. You are to conceive a continuation of these stories, with at least 1 and at most 3 sentences. We will present the stories in different ways to you. You are to respond to written stories by writing, and to the audible stories by short speech recordings. For every story, please try to imagine what happens next to the main character and write or say a continuation that comes to mind quickly, without very elaborate planning!
  • 8 We qualified the participants in two different crowdsourcing tasks, which have the same (...)
[3] We are looking for up to 1008 English native speakers who show best results in this qualification in order to participate in up to 60 more stories. In order to participate, please confirm that you are an English native speaker or cancel this task.

21If a crowd worker confirms being a native speaker of English, we then presented the technical requirements for the experiment. For instance, they should allow the browser to access their microphone in order to complete the study. We inserted an audio message into the page to confirm that the users can hear our audios and understand the content. The message asks for the result of a simple mathematical operation (“What is the result of 3 plus 4?”). The crowd workers are expected to write the answer in the text box provided on the same page. If the answer of the arithmetical operation was correct, we then continued with the qualification task.

22The qualification study contains two comprehension and six story continuation questions, each shown on separate pages. The story continuation questions were displayed randomly, with no specific preference as to the order. Crowd workers should pass to the next page manually (by pressing the “Next” button) after completing the request on the current page.

23The story continuation questions contain three written and three audio stories. [4] represents an example for a written story and [5] is an audio story9.

[4] The door opened quietly, the dark silhouette of a man entered the room, and the door closed again. The man seemed to be looking for something.
[5] Jack is very happy because he got promoted at work. He is now organizing a big party to celebrate this.

The crowd workers were expected to continue the stories by imagining “what happened next to the main character” and generate responses using the same medium as the story. For this purpose, we provided a text box for responding to the written stories and a voice recording button for the audio stories. For the audio stories, we described what we expect from the participants with the statements in [6] and for the written stories in [7].

[6] Now go ahead, listen, try to imagine what happens next to the main character and record a continuation of up to 3 sentences that come to mind quickly, without very elaborate planning.
[7] Please try to imagine what happens next to the main character again. This time, write a continuation of up to 3 sentences that come to mind quickly! This is how the story begins [HERE THE STORY COMES].

The comprehension questions are about the stimuli in [4] and [5]. Therefore, they always followed these stories in the flow. [8] shows the comprehension question about [4], which asks for the best description of the situation among the given options, and [9] is a question about the situation in [5].

[8] Please select the answer that best matches, according to your understanding.
[9] What made Jack feel happy?

24The crowd workers were expected to choose the most suitable option (3 options were provided for each question) to respond to the comprehension questions. We continued the qualification task even if a crowd worker did not choose the correct option but used all the answers when evaluating the qualification responses in choosing the eligible participants.

  • 10 We did not incorporate participant information in the experiment analysis because we di (...)

25At the end of each qualification task, we asked the crowd workers to provide information about their age, gender (female, male, other), level of education (less than high school, high school, bachelor’s degree, postgraduate degree) and spoken English variety (American, British, other) for anonymous statistical analysis10.

26We went over all the responses of the qualification study manually and eliminated 19 crowd workers considered to be ineligible to participate in the main experiment. Some of these unqualified crowd workers seem to not interpret the instructions correctly, and for the others, the audio quality of their recordings was not good. We created a pool of 556 crowd workers at the end of qualification phase.

3.2. Pilot experiment

2740 participants among the qualified crowd workers were recruited for a pilot experiment. These participants later also joined the actual experiment, which was conducted 8 weeks after the pilot study.

28The structure of the pilot study is the same as the actual experiment. As in the qualification study, we showed short stories to the participants, both in oral and written mediums and asked them to continue the stories by imagining “what happened next to the main character”, using the same medium as the story. The participants saw only one stimulus on the active page. The instructions were similar to the qualification study, with slight changes in the wording as demonstrated in [10] for written and in [11] for oral story continuation collection.

[10] Here comes your next story beginning. Please always try to imagine what happens next to the main character! Write a continuation of up to 3 sentences that come to mind quickly. This is how the story begins [HERE COMES THE STORY]
[11] Please continue with listening to the next story and try to imagine what happens next to the main character! Please record a continuation of up to 3 sentences that come to mind quickly!

29In the pilot study, we observed substantial differences in referential choice regarding the story length. For instance, for the written stories in [12] and [13] which differ significantly in length, we observed a great difference in the participants’ responses in terms of pronoun usage.

[12] James went to IKEA yesterday.
[13] Betty wrote a new novel. It is a crime novel. The book is called Insomnia. The story passed in India, therefore it got attention from Indian media. After a review was published in newspapers, it became popular. Everybody knows it now.

83% of the continuations for [12] include a pronoun as the first reference to “James”, whereas a pronoun is used in only 18% of the continuations for [13] as the first reference to “Betty”. These observations were compatible with the previous studies in the literature (see Section 2). This outcome indicates that we replicated the previous results with our experimental design, which we consider as proper evidence for the validation of the design.

30In addition to the validation of the experimental design, the pilot experiment was also beneficial for finalizing the stimuli structure of the main experiment. We discovered practical hints about the stimuli design which were useful for getting more responses about, and hence more referential expressions to, the main character. The following are the conventions that emerged from the investigation of responses in the pilot experiment:

  1. We did not use direct speech in the stories because participants tend to continue the statements similar to [14] by extending the included speech text instead of mentioning “what happened next to the main character”. An example continuation for [14] received in the pilot experiment is given in [15], where a participant is expanding Clara’s speech instead of mentioning what happened to her next.

    [14] Clara whispered resentfully, “it is almost 3 o’clock”.
    [15] “Where were you?”
  2. We tried to avoid transfer verbs (e.g., “buy”, “sell”, “give”) in 1- and 2-clause-length stimuli. Since these verbs introduce a new object and can shift the focus to that object, the responses to such stories were usually about the object instead of the main character. For instance, continuation examples from the pilot experiment for the story [16] are given in [17] and [18] below:

    [16] Patricia bought a new bicycle.
    [17] The new bicycle was red and fast and had a really loud bell on the front.
    [18] It was bright read with strings coming up the handle.
    It was a bike that every girl wanted.

    In this example, in order to get more responses about “Patricia” rather than “the bicycle”, we modified the story as in [19].

    [19] Patricia rode all day yesterday.
    • 11 Change in the protagonist’s name is only in order to balance the female-male name rat (...)

    We do not introduce a human entity in the stories apart from the main character (see Section 3.3.2 for details). Therefore, it is inevitable to insert non-human entities into long (5- and 8-clause-length) stories. In those cases, we preferred to not to focus only on one object but mention different circumstances, to avoid responses about the objects. Therefore, instead of [20], for instance, we preferred to use [21] in the final stimuli set11.

    [20] Mark moved to a new apartment. It has three rooms. The rooms are spacious, the kitchen is renovated. It is located in a quiet neighbourhood. It has a balcony. The view is impressive when the sun goes down over the sea.
    [21] Gabriella moved to a new neighbourhood. The previous area was chaotic because it was very central. This new neighbourhood is quiet. There are parks nearby. The new house is great. It is quite peaceful. A whole new life can start here.

3.3. Main experiment

3.3.1. Participants

31Participants of the main experiment are recruited from the qualified crowd worker pool generated according to the outcome of the qualification study described in Section 3.1. Recruited participants are mostly based in US and UK. 245 English speakers, who declared to be native in English, participated in the main experiment. Mean age was 36.7 years (range 18-64) (based on the age information provided by 235 of the participants).

32Distribution of participants with respect to:

  1. gender is 58% female, 40% male, and 2% other (based on the gender information provided by 235 of the participants);

  2. English variety is 48% American English, 50% British English and 2% other (based on the language information provided by 216 of the participants);

  3. level of education is 2% with less than a high school diploma, 34% with a high school degree, 45% with a bachelor’s degree, and 19% with a postgraduate degree (based on the level of education information provided by 216 of the participants).

3.3.2. Materials

  • 12 In addition to the features we took into account in the experimental design, there ex (...)

33We considered the following features in interaction with anaphoric distance and restricted the structure of the stimuli in order to minimize the variation in terms of these features12.

    • 13 Excluding the connectives linking the clauses.

    Average clause length
    We tried to create constant length clauses in the stimuli (i.e., 5 words). However, there are cases where we needed to bend this rule, for the sake of naturalness of the stories. In those cases, we kept the average clause length (i.e., total number of words13 in the story divided by the number of clauses) constant at 5 words.

    • 14 Stimuli with the same content were shown to the mutually exclusive groups, so each (...)

    Token types
    The same stories14 are used for both audio and written forms to minimize the impact of diverging tokens and token types.

  1. Clause types

    1. Only finite subordinate clauses are used in the stories.

    2. For establishing explicit relations, only temporal and causal discourse connectives are used.

  2. Bridging anaphora
    We avoid establishing bridging anaphoric connections with the main character in the stimuli.

34In addition to the above constraints, we also considered the following principles in the design of the stimuli:

  1. The experimental factor in this study is the length of the stories. The test conditions of the experiment are constructed by manipulating the story lengths as 1, 2, 3, 5, and 8 clauses.

  2. The main character is referred only once in each story, always introduced in the first clause and referred to by a proper name.

  3. Only one human reference (main character) is introduced in the stimuli.

  4. Stories are always written in past tense.

  5. Stories always start with the main character in subject position in order to avoid possible complications due to the differences in syntactic roles.

  6. 8 stimuli for each clause length condition (4 with common male names + 4 with common female names) are created. As a result, we have 40 stimuli in total.

  7. 20 filler stories having no constraints are used in addition to the 40 actual stimuli. Responses to the filler stories are not included in the analysis.

35The above-mentioned constraints and principles regarding the stimuli design are determined at the beginning of the design phase. The stimuli set is finalized after evaluation of the pilot experiment described in the previous section. In Table 1, we present story texts from the final stimuli set for each test condition and filler stories.

Table 1 – Stimuli examples

Condition Example
1-clause length Robert finally solved the mystery.
2-clause length Barbara had a quick lunch before the doorbell rang once again.
3-clause length Lisa washed the dishes resentfully after the party ended. It was a long and gloomy night.
5-clause length Carol made a peppermint tea before the sun rose. Herbs can balance the emotions. Peppermint helps for anxiety so it was a perfect choice for this morning.
8-clause length Charlotte was tired. There are good days in life; there are also challenging days. Today was one of the latter. There was nobody around. The building was almost empty. It was getting dark. Even the computer turned to sleep mode.
Filler The door opened quietly, the dark silhouette of a man entered the room, and the door closed again. The man seemed to be looking for something.
Filler The evening passed quietly, unmarked by anything extraordinary. At night, Darcy opened his heart to Jane.

3.3.3. Procedures

36As previously stated, this experiment is designed to collect audio and written responses from the participants by describing what happened next to the main character of the demonstrated short stories. The aim is to collect oral and written narratives where both types are produced in a spontaneous and informal way by using the same medium as the stimulus.

37We created three crowdsourcing tasks, each containing 20 stimuli, where 10 of them are presented in audio and the other 10 in written format. Either 6 or 7 of the stimuli in each crowdsourcing task are filler stories, both in audio and written forms. All 3 crowdsourcing tasks were available to every participant. We asked them to complete all 3 tasks (i.e., 60 stimuli). Some participants completed all 3 tasks, but we also received responses for only 1 or 2 crowdsourcing tasks from some participants, which we also consider in the analysis.

38The user interface of the main experiment is similar to the pilot study described in Section 3.2. We used the same instructions as in [10] and [11] at the top of the pages where we present the written and audio stories, respectively.

39The experiment was executed in October-November 2020 on two mutually exclusive groups of participants (Group1 and Group2). Both groups were exposed to the same 60 stimuli yet in different mediums. For instance, if the story “Patricia rode all day yesterday” is shown to the participants of Group1 as a written stimulus, Group2 is exposed to its audio version. Consequently, we did not collect narratives for the same oral and written stories from the same individuals, which was a necessary constraint for getting spontaneous responses at each step. We presented the stimuli in a random order but distributed the filler stories uniformly between the stories, so that the participants are not exposed to the actual stimuli repeatedly.

  • 15 There exist empty responses in the written cases, this is why there is a difference b (...)

40Table 2 summarises the participation statistics for the experiment. Group1 is composed of 131 and Group2 is composed of 114 participants. 220 crowdsourcing tasks (i.e., 20 stimuli in each) were submitted by Group1, whereas 239 task submissions were received from Group2. We excluded the responses to filler stories and processed 3,054 responses for oral stories and 3,04915 for written stories, in the analysis.

41As mentioned earlier, we provided 3 crowdsourcing tasks (i.e., 60 stimuli) for all the participants. Although we asked them to complete all the tasks, we did not make it mandatory as the tasks can take a considerable amount of time. As a result, we received a varying number of task submissions from the participants, as presented in Table 2. For instance, 52 participants in Group1 and 34 participants in Group2 submitted only 1 crowdsourcing task (i.e., 20 stimuli). The number of participants who completed all 3 tasks is unfortunately very small in Group1 compared to Group2 (10 vs. 45). During the execution of the experiment for Group1, a technical issue occurred with showing the third task to the participants. Consequently, many of the participants in Group1 could not see nor complete the third task. However, we resolved the issue by increasing the number of participants in Group1 to keep at least 70 responses for each story in our stimuli set, which is demonstrated at row 10 in Table 2.

Table 2 – Main experiment participation summary

Group1 Group2 Total
1 Participants 131 114 245
2 Submitted crowdsourcing tasks 220 239 459
3 Number of oral responses (including responses to fillers) 2,200 2,390 4,590
4 Number of written responses (including responses to fillers) 2,200 2,390 4,590
5 Number of oral responses (excluding responses to fillers) 1,465 1,589 3,054
6 Number of written responses (excluding responses to fillers) 1,465 1,584 3,049
7 Participants completed 3 tasks 10 45 55
8 Participants completed 2 tasks 69 35 104
9 Participants completed 1 task 52 34 86
10 Minimum number of responses for a stimulus 70 73 70
11 Maximum number of responses for a stimulus 76 85 85

42At the end of each crowdsourcing task, we provided a free text area for the participants to express their feedback about the task. We received feedback from 28 participants. Two of the comments were about a temporary technical problem in the interface they encountered, one was complaining about the amount of payment. The rest of the comments were all very positive about the experiment (e.g., “I really enjoy doing these tasks!”, “Another good study which I enjoyed doing.”, “This was fun! :-)”). With this positive feedback and the observed coherent nature of the responses, we are confident that our participants experienced gratifying motivation and engagement for the experiment. The feedback also indicates that in addition to the payment they received, some of the participants were also motivated by finding entertainment in our experiment.

3.4. Transcription

43We followed a semi-automated procedure to transcribe the audio responses. First, the audio recordings were automatically transcribed via Google speech-to-text API16. We then went over all the transcriptions manually, listened to the recordings one by one and corrected the errors in the auto-transcriptions. We also added manual punctuation marks (sentence final markers and commas) when needed into the transcribed audio responses.

44In 18% of the cases, auto-transcription was totally correct; therefore, we did no modification on them. In 9% of the cases, we only added some punctuation and capitalized the sentence initial letters. In the remaining 73% of auto-transcriptions, we made corrections in the transcribed text. Some of the transcriptions included erroneous wording and some others were missing speech pieces. Non-verbal communication like laughter and pause, and prosodic features such as stress intonation, sound prolongation, emphasis, etc. are not marked in the transcriptions. We eliminated 5% of the responses after listening the audio responses. Most of these responses had provided silent or noisy audio recordings and the others had generated texts on topics different from the stimulus.

3.5. Analysis

45For the analysis, we processed the responses submitted by the participants as story continuations. Table 3 shows the statistics on the length of responses we received. The lengths of oral responses are computed after we inserted the final punctuation marks into the transcribed texts. As shown in rows 1 and 2, the responses to written stories are longer on average than responses to oral stories. Responses tend to be longer as the length of the stimuli increases. Especially the difference in response length for 1-clause stimuli and 5- and 8-clause stimuli is remarkable.

Table 3 – Average response length statistics

Oral Written Total
1 Average response length (chars) 120 133 126
2 Average response length (words) 23 25 24
3 Average response length for 1-clause length stimuli (words) 19 23 21
4 Average response length for 2-clause length stimuli (words) 23 24 23
5 Average response length for 3-clause length stimuli (words) 23 25 24
6 Average response length for 5-clause length stimuli (words) 24 25 25
7 Average response length for 8-clause length stimuli (words) 24 25 25

46In the rest of this section, we present more statistics to better describe the responses and the way we process this data.

3.5.1. Syntactic properties of responses

47We automatically segmented the responses into sentences using the nltk library (Bird et al., 2009) in Python. We then run the Berkeley Neural Parser (Kitaev & Klein, 2018) on our data and extracted clause and NP structures using the constituency parse trees generated by the parser. We did not do any manual correction on the parse trees although we observed considerable number of errors in them. Thus, the numbers presented in this section should be considered a rough approximation of data description rather than a robust analysis.

48Table 4 gives quantitative information on the syntactic structures in the responses. Subordinate clauses are more common for both mediums but the difference between the number of clause types is more prominent for the written medium. In addition, written responses contain less, and therefore longer, clauses than the oral medium, given that the written responses are longer than oral responses. As subordination is more common for both mediums, this result is not in line with what has been reported earlier, for instance, by Schäpers (2009). However subordinate clauses are much more dominant and longer in written responses. These observations are compatible with the results reported in Schäpers (2009).

49Third person pronouns and non-pronominal NPs have similar frequency distributions in both mediums, with third person pronouns being more common in the oral medium than in the oral medium with the slight difference of 1%. The length of the non-pronominal NPs (in terms of words) is computed as 2.89 for the oral and 2.86 for the written medium. This result does not confirm the findings in the literature where it is reported that NPs are longer in written medium (e.g., Aktaş & Stede, 2020). However, because the differences in these comparisons are not substantial enough to eliminate the inaccuracies caused by automatic parsing, we avoid making generalisations from them.

Table 4 – Syntactic properties

  • 17 With respect to all NPs.
Oral Written Total
1 Coordinate clauses 3,261 2,127 5,388
2 Subordinate clauses 3,787 3,652 7,439
3 Third person pronouns 7,266 7,828 15,094
4 Non-pronominal NPs 17,592 19,999 37,591
5 Percentage of third person pronouns17 29% 28% 29%
6 Percentage of non-pronominal NPs 71% 72% 71%
7 Non-pronominal NP length 2.89 2.86 2.87

3.5.2. Discourse features in responses

50In order to explore the type of responses and their discourse relation to the stimuli, we created a subset of the data containing approximately 9% of the responses (i.e., 500 in total) with a balanced number of oral and written responses. Also, the sample is balanced for continuation length. We marked the responses in this subset using a scheme inspired from the Temporal and Expansion discourse relations (e.g., in the Penn Discourse TreeBank framework, Prasad et al., 2019). We define three distinct groups marked by E1, E2 and T as described below (S and R denote the clauses in the stimuli and the responses, respectively):

  1. [S1 … Sn] Temporal [R1 … Rn] (= T):
    There is one event in S, and a temporally-later event in R. The stimulus event is usually stated in the first clause S1, while S2…Sn are additional elaborations. So, the relation holds between the complete stimulus and the response.

    [22] Stimulus: “Matthew went out for the first time after the lockdown was lifted. Restaurants were open and busy. The streets were empty because it was very cold outside.”
    Response: “He entered the warm restaurant to be met by the smell of good wine and strong garlic. He breathed in the scent, relieved as a feeling of normally settled over him.”
  2. S1 … [Sn] Expansion [R1 … Rn] (E1)
    R adds some elaboration to the final clause of S. Importantly, the response does not introduce a new event (i.e., the relation is not Temporal).

    [23] Stimulus: “Carol made a peppermint tea before the sun rose. Herbs can balance the emotions. Peppermint helps for anxiety so it was a perfect choice for this morning.”
    Response: “Peppermint also helps with bloating, and she suffers with this from IBS.”
  3. [S1] … Sn Expansion [R1 … Rn] (E2)
    R adds some elaboration to the first clause in S. So, the relation holds between the first stimulus clause and the response. Importantly, R does not introduce a new event (i.e., the relation is not Temporal).

    [24] Stimulus: “Theresa published an article in Lancet. Lancet is a leading medical magazine. It covers everything about health. It publishes up-to-date studies. There is a review process before the articles are issued. Lancet accepts only original papers. Therefore, this is an impressive success.”
    Response: “Her article was about covid-19. She was very excited to have this published in Lancet.”
  • 18 In Section 5, we provide a brief discussion on this difference by refer (...)

51As shown in Table 5, the most common relation in our subset is Temporal (T). In the written responses, we observe a higher percentage of temporal relations than in the oral responses18.

Table 5 – Discourse relations between the responses and the stimuli

Relation type Oral (%) Written (%) Total (%)
1 E1 12 10 11
2 E2 27 17 22
3 T 61 73 67

52Distribution of referential forms for the first reference to the main character in these responses is presented in Table 6. The statistics show that E1 and T type of relations trigger more frequent use of full NPs (names in this case) than the E2 relations.

Table 6 – Referential choice with respect to discourse relation type

E1 (%) E2 (%) T (%)
1 Name 36 14 27
2 Pronoun 64 86 73

53We fitted a generalised mixed-effects logistic regression (Bates et al., 2015) to this subset of data to predict the binary “referential choice” (pronoun vs. name) from the relation type, with random effects for the participants and the stimuli. For the relation type, sum coding is used, with E1 being coded as (1,0), E2 being coded as (0,1), and T being coded as (-1,-1). The model results are reported in Table 7. As the model indicates there is a correlation between the relation type and the referential form. For instance, pronouns in E2 relations are used more frequently than the average pronoun usage for all the relation types (p < 0.05) which is in line with the frequency values presented in Table 6.

Table 7 – Summary of the final model (significance thresholds: ***p < 0.001, **p < 0.01, *p < 0.05)

Estimate (log-odds) Odds-ratio S.E. z val. p
Intercept 1.27 3.53 0.22 5.76 ***
relation-type1 -0.48 0.62 0.29 -1.64 0.10
relation-type2 0.56 1.75 0.26 2.12 *

3.5.3. Processing of responses

54For the investigation of our main research question, we marked the first references to the main character in the responses. For instance, first references to the protagonist in stimulus [25] (i.e., Robert) are marked as in responses [26] and [27].

[25] Robert finally solved the mystery.
[26] ROBERT gasped as he realised the truth. All this time his boss had told him to stop investigating and now Robert knew why.
[27] It suddenly came to HIM out of nowhere as he looked out of the train window watching the fields, and forests go by.

55We do not count the pronouns used to refer to the main character but which do not match the gender. We received 36 responses with an unmatching gender reference. For some names, participants occasionally use short or casual forms. For instance, “Fred” or “Freddie” were used instead of “Frederick” and “Danny” instead of “Daniel”. We count these cases as instances of reference by the name.

56If there exist no reference to the main character in the response, as demonstrated in [29] which is a continuation to the story [28], we do not consider that response in the analysis. We received 443 responses without a reference to the main character.

[28] Tony started organic farming for health reasons. Organic food is healthy because no chemicals are used in the process. It yields more food with less expense when it is done systematically. The products are much tastier: Tomatoes smell wonderful, peppers are flavorful.
[29] Everything tastes fresh and natural. Things grow as it should be grown. Nature has a way of nurturing her harvest.

4. Results

4.1. Descriptive analysis

57As mentioned earlier, we only used proper names to introduce the main characters and did not provide additional details except for implying the person’s gender by using common male and female names. The participants used either a personal pronoun or the name for referring to the main character. We did not encounter any other referential forms denoting the main character in the responses, such as definite or indefinite noun phrases.

58We computed the distribution of the form of referential expression (pronoun vs. explicit name) of the first reference to the main character in the responses.

59The percentage of pronouns as the first reference (pronoun usage, henceforth) is 79.6% in oral and 77.1% in written medium. The average story length for which the participants used a pronoun in the continuations as the first reference to the main character is 3.56 clauses for the oral and 3.44 clauses for the written medium.

Figure 1 – Distribution of pronoun usage for each stimulus length

Figure 1 – Distribution of pronoun usage             for each stimulus length

60Distribution of pronoun usage with respect to the stimuli length (in terms of clauses) is shown in Figure 1. It demonstrates that pronoun usage decreases with the increasing length of the stimuli for both mediums. The chart also indicates that pronoun usage is higher in oral responses than in written responses for longer stories.

61In order to get more information about the internal patterns of responses, we further show the distribution of pronoun usage for each story length via box plots (Figure 2) and bar plots demonstrating pronoun usage for each stimulus (Figures 3-7).

Figure 2 – Box-plots of pronoun usage for each stimulus length

Figure 2 – Box-plots of pronoun usage for             each stimulus length

62Figure 2 indicates that the median values of pronoun usage are greater in the oral data for 2-, 3-, 5- and 8-clause-length stimuli, whereas the median is obviously higher in written data for 1-clause-length stories. The median of the written responses for 1-clause-length is also greater than all other values in the corresponding oral responses. For the 8-clause-length stories, the oral median is only slightly smaller than the upper quartile of the written data and the upper quartile of the oral data is greater than all the values in the distribution of written responses. As a result, we conclude from this box plot representation that pronoun usage is higher in written responses for very short 1-clause-length stories but higher in oral data for long 8-clause-length stories. We cannot make such clear inferences from the box plots demonstrating the distributions for 2-, 3-, and 5-clause-length stories.

Figure 3 – Pronoun usage in responses to 1-clause-length stimuli

Figure 3 – Pronoun usage in responses to             1-clause-length stimuli

Figure 4 – Pronoun usage in responses to 2-clause-length stimuli

Figure 4 – Pronoun usage in responses to             2-clause-length stimuli

Figure 5 – Pronoun usage in responses to 3-clause-length stimuli

Figure 5 – Pronoun usage in responses to             3-clause-length stimuli

Figure 6 – Pronoun usage in responses to 5-clause-length stimuli

Figure 6 – Pronoun usage in responses to             5-clause-length stimuli

Figure 7 – Pronoun usage in responses to 8-clause-length stimuli

Figure 7 – Pronoun usage in responses to             8-clause-length stimuli
  • 19 The same pattern also exists for stimulus S4 although the difference is too small to (...)

63Figures 3-7 support our interpretation of the box-plots in Figure 2. We observed that pronoun usage in written responses for all the stimuli19 is higher than in oral responses for 1-clause-length stories, whereas the responses to 8-clause-length stories exhibit the opposite behaviour; pronoun usage is higher in oral responses for all the stories except one (S34) for the 8-clause-length condition. We do not observe such clear patterns for 2-, 3-, and 5-clause-length stimuli, which are shown in Figures 4-6.

4.2. Generalised mixed-effects logistic regression

64We fitted a generalised mixed-effects logistic regression model to the responses in order to evaluate the factors influencing the referential choice, using the lme4 package (Bates et al., 2015) in R. The dependent variable (i.e., the first referential choice for the main character in each response) is a binary variable, with the possible values of Pronoun and Name. We considered the “stimulus length” as a numerical variable rather than a categorical one and centered it around its mean. We applied sum coding for the “medium”, with the “oral” medium being coded as 1 and the “written” medium being coded as -1.

65We increased the complexity of the model gradually. Table 8 shows the summary and comparison of the models via likelihood ratio tests (done by the anova method in R). We first fitted a fixed-effects model (m0). Medium, length of the stimuli and their interactions are considered as fixed effects in this model. Next, in order to address the differences between participants, we added a random intercept of participants to the model (m1). The likelihood ratio test indicates that m1 is a better model because it has a smaller AIC (Akaike Information Criterion) value, as shown in Table 8. In the next step, we added a second random effect for the stimuli (m2) which also improved the model significantly (p < 0.001). Adding a random slope over “stimuli” leads to the singularity of the model. A random slope for “medium” over the “participants” (m3) does not cause singularity and contributes to the model significantly (p < 0.05). Therefore, we include random slope for the medium over participants. We did not include the participants’ features in the model such as gender and age because we do not control these variables in our experiment.

Table 8 – Comparison of the generalised mixed-effect models fitted to the data (significance thresholds: ***p < 0.001, **p < 0.01, *p < 0.05)

Model Formula AIC R2(f) R2(t) Anova p
m0 ref_expr∼medium*length 5,554 0.05 - - -
m1 ref_expr∼medium*length +
(1|participant)
4,518 0.08 0.51 m1,m0 ***
m2 ref_expr∼medium*length +
(1|participant) + (1|stimulus)
4,384 0.09 0.56 m2,m1 ***
m3 ref_expr∼medium*length +
(1+medium|participant) +
(1|stimulus)
4,381 0.09 0.57 m3,m2 *

66The column R2(f) shows McFadden’s pseudo-R2 value for fixed effects, whereas the R2(t) column represents the conditional R2, which describes the proportion of variance explained by both the fixed and random effects. The R2 numbers in the table indicate that the model without random factors fits to our data very poorly (i.e., explanatory power is 5% for m0). In contrast, the total explanatory power of a mixed-effects model is substantial (conditional R2 is 57%) and the part related to the fixed effects alone (R2(f) = 9%) is also higher than the R2(f) of the fixed-effects logistic regression (m0).

67The summary for the fixed effects of the final model (m3) is demonstrated in Table 9.

Table 9 – Summary of the final model (significance thresholds: ***p < 0.001, **p < 0.01, *p < 0.05)

Estimate (log-odds) Odds-ratio S.E. z val. p
Intercept 1.93 6.87 0.16 12.05 ***
medium 0.09 1.10 0.092 1.97 *
length -0.32 0.73 0.04 -7.53 ***
medium:length 0.03 1.03 0.036 2.05 *

68The Intercept in Table 9 indicates that the mean value for pronoun usage across the dataset is substantially greater than name usage in the fitted model for the average length of stimuli (odds-ratio = 6.87). Participants seem to use fewer pronouns in written medium than in oral medium for the average length of stimuli, and the difference between mediums is statistically significant. The model coefficients indicate that an increase in the length of the stimuli decreases the pronoun usage significantly (p < 0.001) for both mediums. The decrease in pronoun usage, as demonstrated in Figure 8, is more prominent for the written medium than the oral medium for the increasing length of stimuli (p < 0.05). This inference is compatible with the data demonstrated in Figure 2, which indicates that pronoun usage in the written medium is affected more than the oral medium by the increasing length of the stimuli.

Figure 8 – Predicted probabilities of pronoun usage with increasing length (centered values) across mediums

Figure 8 – Predicted probabilities of             pronoun usage with increasing length (centered values) across             mediums

5. Discussion

69Our first observation on our analysis concerns the type of texts produced in response to stimuli. We presented an analysis of discourse relations on a subset of the experiment data in Section 3.5.2. According to the findings, 67% of all responses introduce a Temporal relationship to the stimuli by introducing a temporarily-later event. According to the framework introduced by Labov and Waletzky (1967), these responses form a temporal juncture with the event introduced in the stimulus, and therefore these stimulus-response pairs can be considered as narratives (i.e., Labov and Waletzky define narrative as a sequence of clauses which contains at least one temporal juncture). Our analysis also shows that written responses have more Temporal discourse relations between stimuli and responses than oral responses. The interplay between the realms of narrative structure and reference was studied by Klein and Stutterheim (1992), who used the term “referential movement” in stories consisting of a “main structure” that is at various points interrupted by portions of “minor structure” (descriptive, elaborative content where the temporal references do not move forward in time). We see our distinction between coherence relation types as making the same point: In the narrations, the “T” cases represent a switch from a preceding “minor structure” back to the “main structure”. Klein and Stutterheim to our knowledge did not investigate the role of pronominalization, but we assume that the use of a full NP as opposed to a pronoun serves as a signpost to the audience for making the switch back to the main structure. Constructing this signpost, we furthermore assume, requires planning work that is undertaken more often in writing than in speech, which is compatible with the observation that T discourse relations are established more often in written responses than oral responses in our experiment.

70The second observation we made based on the results shown in Section 4 is regarding the dominant use of pronouns for both mediums as a first reference to the main character in responses. The statistics reported in Section 3.5.1 suggests a pattern similar to what is reported by Aktaş and Stede (2020), who found that non-pronominal NPs are more common than third person pronouns for both mediums. However, the expression used for the first reference to the main character in our responses does not reveal a similar pattern because at least 60% of the first references are pronouns for both mediums. This result is not directly comparable to the results reported by Aktaş and Stede (2020) (i.e., this result only applies to the first referential form referring to the main character in the responses, not to all referring expressions), but its significant departure from the statistics in Section 3.5.1 is remarkable. Our findings are comparable to and consistent with the findings of Fox (1987), who reveals that in the absence of interfering referents between the antecedent and the anaphoric expression, pronoun usage is significantly prevalent in both mediums. Arnold and Griffin (2007: 521) state that “even when a pronoun would not be ambiguous, the presence of another character in the discourse decreased pronoun use”. We believe the high number of pronouns referring to the main character for both mediums (i.e., no difference between oral and written) might result from the fact that we introduce only one human (or animate) entity in our stimuli, so there is no competition with another entity.

  • 20 For a more thorough study of the data, we believe using the manually corrected parse tree (...)

71The third observation concerns the relative distribution of all third person pronouns and non-pronominal noun phrases across mediums. Frequency distributions regarding all NPs in the dataset presented in Section 3.5.1 indicate that the use of third person pronouns and non-pronominal NPs are similar for both mediums, with pronouns being slightly more frequent in oral than in written responses (i.e., 29% vs. 28%). The frequency distribution of the same elements when used as a first reference to the main character shows a similar trend in that pronouns are more common in oral than in written responses (i.e., 79.6% vs. 77.1%). The statistical model we fitted to our data confirms that oral medium contains more frequent use of pronouns, and this difference is significant (p < 0.05) for the average length of the stimuli. As a result, our findings on the pronoun usage as a first reference to the main character indicates that third person pronouns are more frequent in the oral medium than in the written medium on average. For the result concerning all NPs, the numbers are very similar (29% vs. 28%) and we obtained these results through automated parsing, which contains errors by nature; we therefore interpreted the NP distribution difference between mediums as inconclusive20.

72The fourth observation is regarding the distribution of pronoun usage for varying lengths of the stimuli. Figures 3-7 show that the patterns for 1- and 8-clause length stimuli are regular, with written responses containing more pronoun usage than oral responses for 1-clause stimuli and the opposite tendency for 8-clause stimuli. A closer look at Figures 3-7 reveals irregularities in the seemingly regular patterns. For instance, written responses to stimulus S34 in Figure 7 (given in [30]) include more pronoun usage (51%) than the oral responses (47%). Consequently, S34 does not fit the general pattern of 8-clause-length stories. In addition, this stimulus triggers less pronouns than the other stimuli with the same length (see Figure 7). Another stimulus (S36) in Figure 7, which is given in [31] represents a regular pattern with much higher pronoun usage.

[30] Danny moved to an island. It was rather small. Vehicles were not permitted there. It was a mysterious place, too. There was a volcano in the South. One night, the volcano became active, then the sky brightened. That was really scary.
[31] Gabriela moved to a new neighborhood. The previous area was chaotic because it was very central. This new neighborhood is quiet. There are parks nearby. The new house is great. It is quite peaceful. A whole new life can start here.

73In our experimental procedure, the event brought about by the main character is introduced in the first clause for the stimuli longer than one clause, which is usually followed by elaborative clause(s) establishing Expansion relations (see Section 3.5) with the first clause. Different from the general pattern, the stimulus in [30] contains a second event temporarily ordered (a “complicating action” clause according to Labov & Waletzky [1967]) with respect to the main character’s action introduced in the first clause. The more regular stimulus in [31] does not contain a complicating action, and therefore cannot be considered as a narrative according to Labov and Waletzky (1967). The subset we used for the discourse relation evaluation in Section 3.5.2 contains 18 responses to stimulus in [30] and 14 responses to stimulus in [31]. Responses to [30] contain only E1 (11%) and T (89%) type of relations, whereas responses to [31] contain all three relation types (14%, 14% and 72% for E1, E2 and T, respectively). This indicates that responses to [30] does not elaborate on the main character’s action introduced in the first clause, but introduces new events or expand the second action in the stimulus, whereas responses to [31] relate to the stimulus with Expansion relation (E2) elaborating the main action (“orientation” according to the framework of Labov & Waletzky [1967]). This observation indicates that the type of the stimuli (e.g., narrative or not) has an impact on the type of relation established by the response. Therefore, considering the correlation between the relation type and referential choice discussed in Section 3.5.2, we can hypothesize that the type of the stimulus affects the form of the first reference to the main character. However, this hypothesis needs further investigation, with all the stimuli being analysed for their qualitative features and the relation types with their responses being labelled for a larger number of instances.

74The fifth observation on the data presented in Section 4 is that pronoun usage decreases with increasing anaphoric distance for both mediums, which is consistent with previous comprehension research (Clark & Sengul, 1979; Streb et al., 2004), with the assumption that longer pronoun comprehension time means less pronoun usage in production, for a setting with long anaphoric distance. The difference in pronoun usage for the written medium between 1-clause-length and 8-clause-length stories (in Figure 1) is 32%, whereas it is 20% for the oral medium. This finding is consistent with prior corpus-based research (Fox, 1987; Aktaş & Stede, 2020), which indicated that oral texts allow for greater anaphoric distance than written texts. In our interpretation, the difference in the decrease of pronoun usage between oral and written mediums indicates that anaphoric distance has a larger effect in referential choice for the written medium, which is also confirmed with the generalised mixed-effects analysis in Section 4.2.

6. Summary and conclusion

75The aim of this study is to investigate the differences across oral and written mediums, in terms of the impact of anaphoric distance on the referential choice in language production. Relevant contrastive corpus-based studies (Fox, 1987; Aktaş & Stede, 2020) indicate that oral texts include longer average anaphoric distance than written texts between third person pronouns and their antecedents, where the distance is measured in terms of clauses. In relation to these findings, we hypothesized that oral texts will contain more pronouns than written texts, in the case of long anaphoric distances. We designed an experimental setup to test this hypothesis, where we manipulated the anaphoric clause-based distance systematically while keeping constant the features of the antecedent, such as animacy, plurality, grammatical role and the form of referential expression (i.e., name or pronoun). Previous studies (e.g., Schäpers, 2009) demonstrate that clause lengths, types and tokens differ between oral and written mediums. Therefore, we manipulated the distance in terms of number of clauses, but controlled the length of clauses, type of tokens and the coherence relations linking the clauses, in order to obtain substantial information about the impact of the medium.

76To the extent of our knowledge, this is the first experimental study investigating the medium distinctions with respect to the impact of anaphoric distance in a production task.

77We used short stories varying in length as the stimuli and expected the participants to continue these short stories by imagining “what happened next to the main character?”. This experiment was conducted via crowdsourcing. Only 5% of the responses had to be eliminated due to, for instance, bad audio quality or texts on topics different from the stimulus. Therefore, this study indicates that crowdsourcing can provide high quality data for such kind of tasks where participants generate language data for a given context, in both oral and written forms.

78Our first hypothesis concerned the distribution of third-person pronouns in oral and written responses in relation to non-pronominal NPs. Based on prior findings in the literature (e.g., Fox, 1987), we expected to detect more third person pronouns in the oral than in the written medium. As summarised in the previous sections, we did find a confirming statistical effect concerning the syntactic types of the first references to the main characters in the responses. However, frequency distribution of all NPs, which is obtained by automated parsing, does not show a strong indication for this direction. As a result, based on our data, we believe the first hypothesis is only partially confirmed. For extracting significant conclusions about all of the NPs across mediums, further analysis based on robust parsing is required.

79In the analysis, we observed that participants use fewer pronouns when the story length (and hence, the anaphoric distance) was increased, in both mediums.

80However, the gap for the average pronoun usage between 1-clause-length stories and 8-clause length stories is greater in written responses (32%) than in oral responses (20%). We fitted a generalised mixed-effects logistic regression model to this data. The model coefficients indicate that pronoun usage is smaller in the written medium than the oral medium when the stimuli length is increased, as shown in Figure 8, which clearly illustrates that pronoun usage decreases more for the written than for the oral medium for the long stimuli. Therefore, we infer that the anaphoric distance has a larger effect on the written medium. Exploration of pronoun usage patterns for each stimulus length reveals that pronoun usage is more common in oral responses than in written responses for long stories, which confirms our second hypothesis and the findings of the above-mentioned comparative corpus studies.

81We observed variation in response to the stories having the same length. In our experiment, we did not control potentially explanatory variables for this variation, such as text type of the stimulus (e.g., narrative or not), number of referring expressions in the stimulus and length of the name of the main character in the stimulus. Continuing the same line of research and conducting experiments with different control setups can help to better understand the variation for the same length of the stimuli.

82In addition to the analysis presented in this paper, we are also planning to extend the scope of this work by performing computational experiments on the same data. Same and Van Deemter (2020) made an extensive evaluation of the feature sets used in a variety of computational systems for generating referring expressions. Their evaluation reveals the importance of four feature sets, three of which related to the antecedent (animacy and plurality, grammatical role, and form), which we controlled and kept constant in our stimulus design. The fourth important feature set that Same and Van Deemter (2020) use is recency, which is the experimental factor in our experiment. We believe that computational classification experiments on our data can give interesting insights into the impact of recency, as the other potentially important features are kept constant.

Haut de page

Bibliographie

Ahn, L. von 2006. Games with a Purpose. Computer 39 (6): 92-94.

Akinnaso, F.N. 1982. On the Differences between Spoken and Written Language. Language and Speech 25 (2): 97-125.

Aktaş, B., Scheffler, T. & Stede, M. 2019. Coreference in English OntoNotes: Properties and Genre Differences. In K. Ekštein (ed.), Text, Speech, and Dialogue. Cham: Springer: 171-184.

Aktaş, B. & Stede, M. 2020. Variation in Coreference Strategies across Genres and Production Media. In D. Scott, N. Bel & C. Zong (eds.), Proceedings of the 28th International Conference on Computational Linguistics (December 8-13, 2020, Barcelona, Spain). Stroudsburg: Association for Computational Linguistics (ACL): 5774-5785. Available online: https://aclanthology.org/2020.coling-main.508.pdf.

Amoia, M., Kunz, K. & Lapshinova-Koltunski, E. 2012. Coreference in Spoken vs. Written Texts: A Corpus-Based Analysis. In N. Calzolari, K. Choukri, T. Declerck, M. Uğur Doğan, B. Maegaard, J. Mariani, A. Moreno, J. Odijk & S. Piperidis (eds.), Proceedings of the 8th International Conference on Language Resources and Evaluation – LREC 2012. Paris: European Language Resources Association (ELRA): 158-164. Available online: http://www.lrec-conf.org/proceedings/lrec2012/pdf/629_Paper.pdf.

Ariel, M. 1990. Accessing Noun-Phrase Antecedents. London – New York: Routledge.

Arnold, J.E. 1998. Reference Form and Discourse Patterns. PhD thesis. Stanford University, Department of Linguistics. Stanford.

Arnold, J.E. & Griffin, Z.M. 2007. The Effect of Additional Characters on Choice of Referring Expression: Everyone Counts. Journal of Memory and Language 56 (4): 521-536.

Bates, D., Mächler, M., Bolker, B.M. & Walker, S.C. 2015. Fitting Linear Mixed-Effects Models Using lme4. Journal of Statistical Software 67 (1): 1-48. Available online: https://www.jstatsoft.org/index.php/jss/article/view/v067i01/946.

Biber, D. 1992. Using Computer-Based Text Corpora to Analyze the Referential Strategies of Spoken and Written Texts. In J. Svartvik (ed.), Directions in Corpus Linguistics: Proceedings of Nobel Symposium 82 (Stockholm, 4-8 August 1991). Berlin – New York: De Gruyter Mouton: 213-252.

Bird, S., Klein, E. & Loper, E. 2009. Natural Language Processing with Python: Analyzing Text with the Natural Language Toolkit. Beijing – Farnham: O’Reilly.

Chafe, W.L. 1976. Givenness, Contrastiveness, Definiteness, Subjects, Topics, and Point of View. In C.N.  (ed.), Subject and Topic. New York – San Francisco – London: Academic Press: 25-55.

Chafe, W.L. 1994. Discourse, Consciousness, and Time: The Flow and Displacement of Conscious Experience in Speaking and Writing. Chicago – London: The University of Chicago Press.

Clark, H.H. & Sengul, C.J. 1979. In Search of Referents for Nouns and Pronouns. Memory and Cognition 7 (1): 35-41.

Enochson, K. & Culbertson, J. 2015. Collecting Psycholinguistic Response Time Data Using Amazon Mechanical Turk. PLoS ONE 10 (3): e0116946. Available online: https://journals.plos.org/plosone/article?id=10.1371/journal.pone.0116946.

Fort, K., Adda, G. & Cohen, K.B. 2011. Amazon Mechanical Turk: Gold Mine or Coal Mine? Computational Linguistics 37 (2): 413-420. Available online: https://direct.mit.edu/coli/article-pdf/37/2/413/1798871/coli_a_00057.pdf.

Fox, B.A. 1987. Discourse Structure and Anaphora: Written and Conversational English. Cambridge – New York – Melbourne: Cambridge University Press.

Garrod, S.C. & Sanford, A.J. 1982. The Mental Representation of Discourse in a Focussed Memory System: Implications for the Interpretation of Anaphoric Noun Phrases. Journal of Semantics 1 (1): 21-41.

Givón, T. 1983. Topic Continuity in Discourse: A Quantitative Cross-Language Study. Amsterdam – Philadelphia: J. Benjamins.

Iskender, N., Schaefer, R., Polzehl, T. & Möller, S. 2021. Argument Mining in Tweets: Comparing Crowd and Expert Annotations for Automated Claim and Evidence Detection. In E. Métais, F. Meziane, H. Horacek & E. Kapetanios (eds.), Natural Language Processing and Information Systems: 26th International Conference on Applications of Natural Language to Information Systems, NLDB 2021, Saarbrücken, Germany, June 23-25, 2021, Proceedings. Cham: Springer: 275-288.

Kitaev, N. & Klein, D. 2018. Constituency Parsing with a Self-Attentive Encoder. In I. Gurevych & Y. Miyao (eds.), Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). Stroudsburg: Association for Computational Linguistics (ACL): 2676-2686. Available online: https://aclanthology.org/P18-1249.pdf.

Klein, W. & Stutterheim, C. von 1992. Textstruktur und referentielle Bewegung. Zeitschrift für Literaturwissenschaft und Linguistik 22 (86): 67-92.

Kunz, K., Lapshinova-Koltunski, E. & Martínez, J.M. 2016. Beyond Identity Coreference: Contrasting Indicators of Textual Coherence in English and German. In M. Ogrodniczuk & V. Ng (eds.), Proceedings of the Workshop on Coreference Resolution Beyond OntoNotes (CORBON 2016). Stroudsburg: Association for Computational Linguistics (ACL): 23-31. Available online: https://aclanthology.org/W16-0704.pdf.

Labov, W. & Waletzky, J. 1967. Narrative Analysis: Oral Versions of Personal Experience. In J. Helm (ed.), Essays on the Verbal and the Visual Arts. Seattle – London: University of Washington Press: 3-38.

Lavee, T., Kotlerman, L., Orbach, M., Bilu, Y., Jacovi, M., Aharonov, R. & Slonim, N. 2019. Crowd-Sourcing Annotation of Complex NLU Tasks: A Case Study of Argumentative Content Annotation. In S. Paun & D. Hovy (eds.), Proceedings of the First Workshop on Aggregating and Analysing Crowdsourced Annotations for NLP. Stroudsburg: Association for Computational Linguistics (ACL): 29-38. Available online: https://aclanthology.org/D19-5905.pdf.

Panteli, N., Rapti, A., Scholarios, D. 2020. “If He Just Knew Who We Were”: Microworkers’ Emerging Bonds of Attachment in a Fragmented Employment Relationship. Work, Employment and Society 34 (3): 476-494.

Poesio, M., Chamberlain, J., Paun, S., Yu, J., Uma, A. & Kruschwitz, U. 2019. A Crowdsourced Corpus of Multiple Judgments and Disagreement on Anaphoric Interpretation. In J. Burstein, C. Doran & T. Solorio (eds.), Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers). Stroudsburg: Association for Computational Linguistics (ACL): 1778-1789. Available online: https://aclanthology.org/N19-1176.pdf.

Prasad, R., Webber, B., Lee, A. & Joshi, A. 2019. Penn Discourse Treebank Version 3.0, LDC2019T05. Web Download. Philadelphia: Linguistic Data Consortium. URL: https://catalog.ldc.upenn.edu/LDC2019T05.

Same, F. & Van Deemter, K. 2020. A Linguistic Perspective on Reference: Choosing a Feature Set for Generating Referring Expressions in Context. In D. Scott, N. Bel & C. Zong (eds.), Proceedings of the 28th International Conference on Computational Linguistics (December 8-13, 2020, Barcelona, Spain). Stroudsburg: Association for Computational Linguistics (ACL): 4575-4586. Available online: https://aclanthology.org/2020.coling-main.403.pdf.

Schäpers, U.K.E. 2009. Nominal versus Clausal Complexity in Spoken and Written English: Theory and Description. Frankfurt am Main – Oxford: P. Lang.

Streb, J., Hennighausen, E. & Rösler, F. 2004. Different Anaphoric Expressions Are Investigated by Event-Related Brain Potentials. Journal of Psycholinguistic Research 33 (3): 175-201.

Webster, J. 2016. Microworkers of the Gig Economy: Separate and Precarious. New Labor Forum 25 (3): 56-64.

Haut de page

Notes

1 We use the terms “oral” and “written” to refer to the production medium. We preferred to use “oral” instead of “spoken”, as “spoken” often indicates immediate speaker contact, which is not the case in our setting.

2 Both oral and written texts are produced by the same individuals. For details about the experimental procedures, see Section 3.3.3.

3 See: https://www.crowdee.com/.

4 There exist different forms of crowdsourcing, such as microworking and game-with-a-purpose (Ahn, 2006). In contrast to our study and the previous crowdsourcing studies we listed above, Poesio et al. (2019) uses the game-with-a-purpose approach, which due to the general similarity, is a form of applying crowdsourcing to coreference research.

5 Despite its popularity, crowdsourcing platforms also get criticisms, which we acknowledge as important, concerning the data quality they provide (Fort et al., 2011) and work conditions of the crowdworkers (Webster, 2016; Panteli et al., 2020).

6 The authors of this study live in a country where English is not the native language of the majority.

7 Stimuli used in the qualification task are less restricted than the stimuli in the actual experiment as this task serves for participant qualification rather than hypothesis testing.

8 We qualified the participants in two different crowdsourcing tasks, which have the same instructions. That’s why the number stated in this instruction is smaller than the target number of participants.

9 The text-to-speech interface at https://ttsmp3.com/ was used to create audio versions of the stories in all the phases of this experimental study.

10 We did not incorporate participant information in the experiment analysis because we did not control these variables throughout the data collection. However, we intend to examine the effects of these features in the computational experiments we plan as future work, as noted in Section 6.

11 Change in the protagonist’s name is only in order to balance the female-male name ratio in the final set.

12 In addition to the features we took into account in the experimental design, there exist other potentially important features, such as the number of referring expressions in the stimulus and the length of the name of the main character in the stimulus. We leave the exploration of those features in future studies.

13 Excluding the connectives linking the clauses.

14 Stimuli with the same content were shown to the mutually exclusive groups, so each participant is exposed to either audio or the written version of the same story.

15 There exist empty responses in the written cases, this is why there is a difference between the number of responses for oral and written mediums.

16 See: https://cloud.google.com/speech-to-text.

17 With respect to all NPs.

18 In Section 5, we provide a brief discussion on this difference by referring the work of Klein and Stutterheim (1992).

19 The same pattern also exists for stimulus S4 although the difference is too small to recognise in the chart.

20 For a more thorough study of the data, we believe using the manually corrected parse trees in the analysis could be beneficial, but we’ll leave that for a later endeavor.

Haut de page

Table des illustrations

Titre Figure 1 – Distribution of pronoun usage for each stimulus length
URL http://journals.openedition.org/discours/docannexe/image/12383/img-1.jpg
Fichier image/jpeg, 98k
Titre Figure 2 – Box-plots of pronoun usage for each stimulus length
URL http://journals.openedition.org/discours/docannexe/image/12383/img-2.jpg
Fichier image/jpeg, 99k
Titre Figure 3 – Pronoun usage in responses to 1-clause-length stimuli
URL http://journals.openedition.org/discours/docannexe/image/12383/img-3.jpg
Fichier image/jpeg, 127k
Titre Figure 4 – Pronoun usage in responses to 2-clause-length stimuli
URL http://journals.openedition.org/discours/docannexe/image/12383/img-4.jpg
Fichier image/jpeg, 126k
Titre Figure 5 – Pronoun usage in responses to 3-clause-length stimuli
URL http://journals.openedition.org/discours/docannexe/image/12383/img-5.jpg
Fichier image/jpeg, 118k
Titre Figure 6 – Pronoun usage in responses to 5-clause-length stimuli
URL http://journals.openedition.org/discours/docannexe/image/12383/img-6.jpg
Fichier image/jpeg, 118k
Titre Figure 7 – Pronoun usage in responses to 8-clause-length stimuli
URL http://journals.openedition.org/discours/docannexe/image/12383/img-7.jpg
Fichier image/jpeg, 110k
Titre Figure 8 – Predicted probabilities of pronoun usage with increasing length (centered values) across mediums
URL http://journals.openedition.org/discours/docannexe/image/12383/img-8.jpg
Fichier image/jpeg, 58k
Haut de page

Pour citer cet article

Référence électronique

Berfin Aktaş et Manfred Stede, « Anaphoric Distance in Oral and Written Language: Experimental Evidence »Discours [En ligne], 31 | 2022, mis en ligne le 22 décembre 2022, consulté le 29 février 2024. URL : http://journals.openedition.org/discours/12383 ; DOI : https://doi.org/10.4000/discours.12383

Haut de page

Auteurs

Berfin Aktaş

University of Potsdam

Manfred Stede

University of Potsdam

Articles du même auteur

Haut de page

Droits d’auteur

CC-BY-4.0

Le texte seul est utilisable sous licence CC BY 4.0. Les autres éléments (illustrations, fichiers annexes importés) sont « Tous droits réservés », sauf mention contraire.

Haut de page
Rechercher dans OpenEdition Search

Vous allez être redirigé vers OpenEdition Search