We thank the anonymous reviewers for their insightful comments and suggestions, which have helped us significantly improve the paper. We would also like to thank Anna Laurinavichyute for her helpful advice on the statistical modeling and Claudia Felser for her insightful comments on the experiment design and earlier drafts of this paper.
This work is funded by the Deutsche Forschungsgemeinschaft (DFG, German Research Foundation) – Projektnummer 317633480 – SFB 1287, Project A03.

1. Introduction

2The distance between anaphors and their antecedents (the closest previous mention of the same referent) is considered one of the most important factors influencing referential choice (Ariel, 1990; Chafe, 1976, 1994; Garrod & Sanford, 1982; Givón, 1983). Experimental evidence indicates that anaphoric distance affects the comprehension effort for the resolution of referential expressions. For instance, extending the anaphoric distance in an experimental setup increases the comprehension time for the referential expressions (Clark & Sengul, 1979; Streb et al., 2004).

3A number of contrastive corpus-based studies investigate the quantitative properties of anaphoric distance across oral and written mediums by analysing various genres such as telephone conversations, broadcast conversations, and news texts (cf. e.g., Fox, 1987; Biber, 1992; Amoia et al., 2012; Kunz et al., 2016; Aktaş et. al., 2019; Aktaş & Stede, 2020). Corpus based research demonstrates that there are differences between oral and written corpora with respect to anaphoric distance. For instance, Fox (1987) computes the linear anaphoric distance in terms of clauses and finds that the average anaphoric distance between third person pronouns and their antecedents is longer in oral texts than in written texts.

4Oral-written medium comparison in terms of coreference variation shows greater contrasts for the intrinsically different genres. For instance, Aktaş and Stede (2020) compare different genres in terms of average clause-based anaphoric distance and observe that the difference is more prominent for telephone conversations and news texts (2.97 vs. 1.45) than for broadcast news and written news (1.66 vs. 1.45). Although telephone conversations and broadcast news are both produced verbally, written news differs significantly from telephone conversations regarding spontaneity and informality, but not from broadcast news. Furthermore, they observed that full noun phrases (e.g., proper names, definite and indefinite noun phrases) are more frequent than third person pronouns for both mediums, with third person pronouns being more common in oral than in written genres.

5In addition, oral and written texts diverge in terms of textual features. Certain discourse markers (e.g., mhm, oh yeah), for example, are widespread in oral conversations but uncommon in news texts and oral genres are more intense in terms of the number of clauses than written genres (Aktaş & Stede, 2020). In another comparative study, Schäpers (2009: 137) examines oral and written corpora that are composed of the same number of words. Schäpers also reports that oral texts have a higher number of clauses than do written texts. Consequently, clauses in written texts are longer. In addition, Schäpers notices discrepancies between mediums regarding how the clauses are linked to each other. For instance, coordination is more common in oral language, whereas in written texts, subordinate clauses are more common.

7Our first hypothesis is that for all noun phrases in responses, oral responses will include more frequent use of pronouns than written responses. Our second and main hypothesis is a specific version of the first hypothesis: oral responses will include more frequent pronoun usage as a first reference to the main character than written responses for long stimuli. (i.e., oral responses allow for longer anaphoric distance). In addition, we aim to provide quantitative results on various aspects of the oral-written medium contrast.

8This paper is structured as follows: in Section 2, we present relevant previous work in the literature. Section 3 describes the design settings and execution procedures for running and evaluating the experiment. We report the results of the experiment in Section 4, employing descriptive and inferential statistics. In Section 5, we discuss our findings. Section 6 concludes and explores future directions.

2. Related work

9Various contrastive corpus-based studies investigate the differences in terms of average anaphoric distance across oral and written mediums. Since the experimental factor in our study is the linear textual distance in terms of clauses (i.e., clause-based distance), we include only the studies investigating the contrasts on clause-based anaphoric distance.

10Fox (1987) examines the use of anaphoric third person singular human references (pronouns and noun phrases) in oral and written texts. The data is composed of conversations (face-to-face and telephone conversations) and expository prose (newspaper articles and psychoanalytic biography). Fox reports that conversations allow longer clausal distances between pronouns and their antecedents than written texts do (2.52 clauses in conversations vs. 1.21 in written texts). Aktaş and Stede (2020) undertake a corpus study involving various genres such as telephone conversations, news and Twitter conversations. Among other measures, they compare the oral and written genres in terms of clause-based anaphoric distance. Their findings are compatible with Fox (1987). In their analysis, the difference is more prominent for conversational texts and news texts (2.97 vs. 1.45) than for broadcast news and written news (1.66 vs. 1.45), although conversational texts and broadcast news are both produced in the oral medium.

11In addition to the anaphoric distance, Aktaş and Stede (2020) compare the oral and written genres also in terms of the relative distribution of nominal expressions according to their syntactic categories (i.e., pronouns vs. non-pronominal noun phrases [NPs]). Their findings indicate that third person pronouns are more frequent in oral conversations than in written news (33% vs. 11%, according to Table 2 in that paper). Fox (1987) compares the frequencies of pronouns and full NPs in settings with no interfering referent, with an interfering referent of a different gender, and an interfering referent of the same gender. Her findings show that in the absence of an intervening referent, pronouns are used more intensely for both environments (64% in written vs. 94% in spoken language) compared to full NPs. The use for pronoun decreases when there exist interfering referents. When a referent of the same gender interferes, for example, the usage of pronouns drops to 13% in written texts and 57% in spoken texts.

12In previous experimental research, the impact of anaphoric distance on pronoun processing has so far been studied within a single medium, where the anaphoric distance is modified to manipulate the working memory. Clark and Sengul (1979) demonstrated that text distance between antecedents and the anaphor affects reading time. In their experiments, participants took less time comprehending a sentence when the referents of such noun phrases were mentioned one sentence back than when they were mentioned two or three sentences back. Streb et al. (2004) conducted experiments in which they manipulated the number of intervening words between the anaphor and its antecedent, hence altering the anaphoric distance. In line with Clark and Sengul (1979), they found that comprehension time increases with the expanded distance. This effect held for both pronouns and proper names. They also reported that the cases of “medium” (antecedent in the previous sentence) and “far” (antecedent 2 sentences before) are not distinguishable.

13In contrast to such studies, we examine the variation between oral and written narrative discourse with a production experiment. To the extent of our knowledge, there is no previous work with the same goal and strategy.

3. Experiment

14The starting point of this experiment is the assumption that heavier expressions (e.g., proper names) are produced more often than lighter ones (e.g., pronouns) when a referent is less active or salient; this has been validated by multiple studies (e.g., Ariel, 1990; Arnold, 1998; Chafe, 1976). The anaphoric distance is highly correlated with accessibility or salience in these studies. We here investigate the question of whether there is a difference regarding the referential choice across oral and written mediums, related to the varying length of anaphoric distance. In our work, we build on the study of Fox (1987) and Aktaş and Stede (2020) and aim to provide empirical evidence on medium distinctions in language production regarding the anaphoric distance. We hypothesize that oral responses in our story continuation experiment will include more prominent pronoun usage than will written responses for long anaphoric distance.

17Our task (i.e., story continuation) does not require expertise in any specific linguistic field. That being the case, in order to ensure the quality of the responses, we only needed to confirm the linguistic competence and the engagement of the participants in the study. Therefore, we recruited the participants via a qualification study (see Section 3.1.1). We created a pool of crowd workers after evaluation of the responses collected in the qualification phase and then used that pool to recruit the participants for the actual experiment.

18The experimental factor in the main experiment is the length of the short stories that are used as the stimuli for triggering the continuations. The test conditions of the experiment are constructed by manipulating the story lengths as 1, 2, 3, 5, and 8 clauses. The design principles applied in the creation of the stimuli are presented in Section 3.3.2. In order to validate the design of the experimental settings (e.g., stimuli design, instruction wording, etc.), we conducted a pilot study before the actual experiment (see Section 3.2). The pilot study provided practical hints for refining the experiment stimuli so that they can more effectively serve our purposes. We conducted the main experiment 8 weeks after the pilot study. The details of the main experiment are given in Section 3.3.

3.1. Participant qualification study

20The first page in the qualification study is the information page and contains the description in [1].

[1]	Native English speakers: Qualify for “Story Continuation” tasks and earn up to 10 Euro today by continuing very short stories (up to 3 sentences). This is your qualification test. Duration is approx. 6-8 minutes. Requirements: English native speakers only; Microphone and speaker or headset needed.

On the next page, we describe the experiment and get a confirmation from the crowd workers about whether they are native speakers of English. We used the instructions in [2] and [3] for this purpose.

[2]

Dear participants, thanks for taking part in this qualification for our story continuation micro task series. Here, you will see 6 very short stories in between 1 to 8 sentences of length. You are to conceive a continuation of these stories, with at least 1 and at most 3 sentences. We will present the stories in different ways to you. You are to respond to written stories by writing, and to the audible stories by short speech recordings. For every story, please try to imagine what happens next to the main character and write or say a continuation that comes to mind quickly, without very elaborate planning!

21If a crowd worker confirms being a native speaker of English, we then presented the technical requirements for the experiment. For instance, they should allow the browser to access their microphone in order to complete the study. We inserted an audio message into the page to confirm that the users can hear our audios and understand the content. The message asks for the result of a simple mathematical operation (“What is the result of 3 plus 4?”). The crowd workers are expected to write the answer in the text box provided on the same page. If the answer of the arithmetical operation was correct, we then continued with the qualification task.

22The qualification study contains two comprehension and six story continuation questions, each shown on separate pages. The story continuation questions were displayed randomly, with no specific preference as to the order. Crowd workers should pass to the next page manually (by pressing the “Next” button) after completing the request on the current page.

[4]	The door opened quietly, the dark silhouette of a man entered the room, and the door closed again. The man seemed to be looking for something.

[5]	Jack is very happy because he got promoted at work. He is now organizing a big party to celebrate this.

The crowd workers were expected to continue the stories by imagining “what happened next to the main character” and generate responses using the same medium as the story. For this purpose, we provided a text box for responding to the written stories and a voice recording button for the audio stories. For the audio stories, we described what we expect from the participants with the statements in [6] and for the written stories in [7].

[6]	Now go ahead, listen, try to imagine what happens next to the main character and record a continuation of up to 3 sentences that come to mind quickly, without very elaborate planning.

[7]	Please try to imagine what happens next to the main character again. This time, write a continuation of up to 3 sentences that come to mind quickly! This is how the story begins [HERE THE STORY COMES].

The comprehension questions are about the stimuli in [4] and [5]. Therefore, they always followed these stories in the flow. [8] shows the comprehension question about [4], which asks for the best description of the situation among the given options, and [9] is a question about the situation in [5].

[8]	Please select the answer that best matches, according to your understanding.

[9]	What made Jack feel happy?

24The crowd workers were expected to choose the most suitable option (3 options were provided for each question) to respond to the comprehension questions. We continued the qualification task even if a crowd worker did not choose the correct option but used all the answers when evaluating the qualification responses in choosing the eligible participants.

26We went over all the responses of the qualification study manually and eliminated 19 crowd workers considered to be ineligible to participate in the main experiment. Some of these unqualified crowd workers seem to not interpret the instructions correctly, and for the others, the audio quality of their recordings was not good. We created a pool of 556 crowd workers at the end of qualification phase.

3.2. Pilot experiment

2740 participants among the qualified crowd workers were recruited for a pilot experiment. These participants later also joined the actual experiment, which was conducted 8 weeks after the pilot study.

28The structure of the pilot study is the same as the actual experiment. As in the qualification study, we showed short stories to the participants, both in oral and written mediums and asked them to continue the stories by imagining “what happened next to the main character”, using the same medium as the story. The participants saw only one stimulus on the active page. The instructions were similar to the qualification study, with slight changes in the wording as demonstrated in [10] for written and in [11] for oral story continuation collection.

[10]	Here comes your next story beginning. Please always try to imagine what happens next to the main character! Write a continuation of up to 3 sentences that come to mind quickly. This is how the story begins [HERE COMES THE STORY]

[11]	Please continue with listening to the next story and try to imagine what happens next to the main character! Please record a continuation of up to 3 sentences that come to mind quickly!

29In the pilot study, we observed substantial differences in referential choice regarding the story length. For instance, for the written stories in [12] and [13] which differ significantly in length, we observed a great difference in the participants’ responses in terms of pronoun usage.

[12]	James went to IKEA yesterday.

[13]	Betty wrote a new novel. It is a crime novel. The book is called Insomnia. The story passed in India, therefore it got attention from Indian media. After a review was published in newspapers, it became popular. Everybody knows it now.

83% of the continuations for [12] include a pronoun as the first reference to “James”, whereas a pronoun is used in only 18% of the continuations for [13] as the first reference to “Betty”. These observations were compatible with the previous studies in the literature (see Section 2). This outcome indicates that we replicated the previous results with our experimental design, which we consider as proper evidence for the validation of the design.

30In addition to the validation of the experimental design, the pilot experiment was also beneficial for finalizing the stimuli structure of the main experiment. We discovered practical hints about the stimuli design which were useful for getting more responses about, and hence more referential expressions to, the main character. The following are the conventions that emerged from the investigation of responses in the pilot experiment:

1. We did not use direct speech in the stories because participants tend to continue the statements similar to [14] by extending the included speech text instead of mentioning “what happened next to the main character”. An example continuation for [14] received in the pilot experiment is given in [15], where a participant is expanding Clara’s speech instead of mentioning what happened to her next.

   [14]	Clara whispered resentfully, “it is almost 3 o’clock”.

   [15]	“Where were you?”

2. We tried to avoid transfer verbs (e.g., “buy”, “sell”, “give”) in 1- and 2-clause-length stimuli. Since these verbs introduce a new object and can shift the focus to that object, the responses to such stories were usually about the object instead of the main character. For instance, continuation examples from the pilot experiment for the story [16] are given in [17] and [18] below:

   [16]	Patricia bought a new bicycle.

   [17]	The new bicycle was red and fast and had a really loud bell on the front.

   [18]	It was bright read with strings coming up the handle. It was a bike that every girl wanted.

   In this example, in order to get more responses about “Patricia” rather than “the bicycle”, we modified the story as in [19].

   [19]	Patricia rode all day yesterday.

3. • 11 Change in the protagonist’s name is only in order to balance the female-male name rat (...)

   We do not introduce a human entity in the stories apart from the main character (see Section 3.3.2 for details). Therefore, it is inevitable to insert non-human entities into long (5- and 8-clause-length) stories. In those cases, we preferred to not to focus only on one object but mention different circumstances, to avoid responses about the objects. Therefore, instead of [20], for instance, we preferred to use [21] in the final stimuli set11.

   [20]	Mark moved to a new apartment. It has three rooms. The rooms are spacious, the kitchen is renovated. It is located in a quiet neighbourhood. It has a balcony. The view is impressive when the sun goes down over the sea.

   [21]	Gabriella moved to a new neighbourhood. The previous area was chaotic because it was very central. This new neighbourhood is quiet. There are parks nearby. The new house is great. It is quite peaceful. A whole new life can start here.

3.3. Main experiment

3.3.1. Participants

31Participants of the main experiment are recruited from the qualified crowd worker pool generated according to the outcome of the qualification study described in Section 3.1. Recruited participants are mostly based in US and UK. 245 English speakers, who declared to be native in English, participated in the main experiment. Mean age was 36.7 years (range 18-64) (based on the age information provided by 235 of the participants).

32Distribution of participants with respect to:

1. gender is 58% female, 40% male, and 2% other (based on the gender information provided by 235 of the participants);

2. English variety is 48% American English, 50% British English and 2% other (based on the language information provided by 216 of the participants);

3. level of education is 2% with less than a high school diploma, 34% with a high school degree, 45% with a bachelor’s degree, and 19% with a postgraduate degree (based on the level of education information provided by 216 of the participants).

3.3.2. Materials

1. • 13 Excluding the connectives linking the clauses.

   Average clause length
   We tried to create constant length clauses in the stimuli (i.e., 5 words). However, there are cases where we needed to bend this rule, for the sake of naturalness of the stories. In those cases, we kept the average clause length (i.e., total number of words13 in the story divided by the number of clauses) constant at 5 words.

2. • 14 Stimuli with the same content were shown to the mutually exclusive groups, so each (...)

   Token types
   The same stories14 are used for both audio and written forms to minimize the impact of diverging tokens and token types.

3. Clause types

   1. Only finite subordinate clauses are used in the stories.

   2. For establishing explicit relations, only temporal and causal discourse connectives are used.

4. Bridging anaphora
   We avoid establishing bridging anaphoric connections with the main character in the stimuli.

34In addition to the above constraints, we also considered the following principles in the design of the stimuli:

1. The experimental factor in this study is the length of the stories. The test conditions of the experiment are constructed by manipulating the story lengths as 1, 2, 3, 5, and 8 clauses.

2. The main character is referred only once in each story, always introduced in the first clause and referred to by a proper name.

3. Only one human reference (main character) is introduced in the stimuli.

4. Stories are always written in past tense.

5. Stories always start with the main character in subject position in order to avoid possible complications due to the differences in syntactic roles.

6. 8 stimuli for each clause length condition (4 with common male names + 4 with common female names) are created. As a result, we have 40 stimuli in total.

7. 20 filler stories having no constraints are used in addition to the 40 actual stimuli. Responses to the filler stories are not included in the analysis.

35The above-mentioned constraints and principles regarding the stimuli design are determined at the beginning of the design phase. The stimuli set is finalized after evaluation of the pilot experiment described in the previous section. In Table 1, we present story texts from the final stimuli set for each test condition and filler stories.

Table 1 – Stimuli examples

Condition	Example
1-clause length	Robert finally solved the mystery.
2-clause length	Barbara had a quick lunch before the doorbell rang once again.
3-clause length	Lisa washed the dishes resentfully after the party ended. It was a long and gloomy night.
5-clause length	Carol made a peppermint tea before the sun rose. Herbs can balance the emotions. Peppermint helps for anxiety so it was a perfect choice for this morning.
8-clause length	Charlotte was tired. There are good days in life; there are also challenging days. Today was one of the latter. There was nobody around. The building was almost empty. It was getting dark. Even the computer turned to sleep mode.
Filler	The door opened quietly, the dark silhouette of a man entered the room, and the door closed again. The man seemed to be looking for something.
Filler	The evening passed quietly, unmarked by anything extraordinary. At night, Darcy opened his heart to Jane.

3.3.3. Procedures

36As previously stated, this experiment is designed to collect audio and written responses from the participants by describing what happened next to the main character of the demonstrated short stories. The aim is to collect oral and written narratives where both types are produced in a spontaneous and informal way by using the same medium as the stimulus.

37We created three crowdsourcing tasks, each containing 20 stimuli, where 10 of them are presented in audio and the other 10 in written format. Either 6 or 7 of the stimuli in each crowdsourcing task are filler stories, both in audio and written forms. All 3 crowdsourcing tasks were available to every participant. We asked them to complete all 3 tasks (i.e., 60 stimuli). Some participants completed all 3 tasks, but we also received responses for only 1 or 2 crowdsourcing tasks from some participants, which we also consider in the analysis.

38The user interface of the main experiment is similar to the pilot study described in Section 3.2. We used the same instructions as in [10] and [11] at the top of the pages where we present the written and audio stories, respectively.

39The experiment was executed in October-November 2020 on two mutually exclusive groups of participants (Group1 and Group2). Both groups were exposed to the same 60 stimuli yet in different mediums. For instance, if the story “Patricia rode all day yesterday” is shown to the participants of Group1 as a written stimulus, Group2 is exposed to its audio version. Consequently, we did not collect narratives for the same oral and written stories from the same individuals, which was a necessary constraint for getting spontaneous responses at each step. We presented the stimuli in a random order but distributed the filler stories uniformly between the stories, so that the participants are not exposed to the actual stimuli repeatedly.

41As mentioned earlier, we provided 3 crowdsourcing tasks (i.e., 60 stimuli) for all the participants. Although we asked them to complete all the tasks, we did not make it mandatory as the tasks can take a considerable amount of time. As a result, we received a varying number of task submissions from the participants, as presented in Table 2. For instance, 52 participants in Group1 and 34 participants in Group2 submitted only 1 crowdsourcing task (i.e., 20 stimuli). The number of participants who completed all 3 tasks is unfortunately very small in Group1 compared to Group2 (10 vs. 45). During the execution of the experiment for Group1, a technical issue occurred with showing the third task to the participants. Consequently, many of the participants in Group1 could not see nor complete the third task. However, we resolved the issue by increasing the number of participants in Group1 to keep at least 70 responses for each story in our stimuli set, which is demonstrated at row 10 in Table 2.

Table 2 – Main experiment participation summary

		Group1	Group2	Total
1	Participants	131	114	245
2	Submitted crowdsourcing tasks	220	239	459
3	Number of oral responses (including responses to fillers)	2,200	2,390	4,590
4	Number of written responses (including responses to fillers)	2,200	2,390	4,590
5	Number of oral responses (excluding responses to fillers)	1,465	1,589	3,054
6	Number of written responses (excluding responses to fillers)	1,465	1,584	3,049
7	Participants completed 3 tasks	10	45	55
8	Participants completed 2 tasks	69	35	104
9	Participants completed 1 task	52	34	86
10	Minimum number of responses for a stimulus	70	73	70
11	Maximum number of responses for a stimulus	76	85	85

42At the end of each crowdsourcing task, we provided a free text area for the participants to express their feedback about the task. We received feedback from 28 participants. Two of the comments were about a temporary technical problem in the interface they encountered, one was complaining about the amount of payment. The rest of the comments were all very positive about the experiment (e.g., “I really enjoy doing these tasks!”, “Another good study which I enjoyed doing.”, “This was fun! :-)”). With this positive feedback and the observed coherent nature of the responses, we are confident that our participants experienced gratifying motivation and engagement for the experiment. The feedback also indicates that in addition to the payment they received, some of the participants were also motivated by finding entertainment in our experiment.

3.4. Transcription

44In 18% of the cases, auto-transcription was totally correct; therefore, we did no modification on them. In 9% of the cases, we only added some punctuation and capitalized the sentence initial letters. In the remaining 73% of auto-transcriptions, we made corrections in the transcribed text. Some of the transcriptions included erroneous wording and some others were missing speech pieces. Non-verbal communication like laughter and pause, and prosodic features such as stress intonation, sound prolongation, emphasis, etc. are not marked in the transcriptions. We eliminated 5% of the responses after listening the audio responses. Most of these responses had provided silent or noisy audio recordings and the others had generated texts on topics different from the stimulus.

3.5. Analysis

45For the analysis, we processed the responses submitted by the participants as story continuations. Table 3 shows the statistics on the length of responses we received. The lengths of oral responses are computed after we inserted the final punctuation marks into the transcribed texts. As shown in rows 1 and 2, the responses to written stories are longer on average than responses to oral stories. Responses tend to be longer as the length of the stimuli increases. Especially the difference in response length for 1-clause stimuli and 5- and 8-clause stimuli is remarkable.

Table 3 – Average response length statistics

		Oral	Written	Total
1	Average response length (chars)	120	133	126
2	Average response length (words)	23	25	24
3	Average response length for 1-clause length stimuli (words)	19	23	21
4	Average response length for 2-clause length stimuli (words)	23	24	23
5	Average response length for 3-clause length stimuli (words)	23	25	24
6	Average response length for 5-clause length stimuli (words)	24	25	25
7	Average response length for 8-clause length stimuli (words)	24	25	25

46In the rest of this section, we present more statistics to better describe the responses and the way we process this data.

3.5.1. Syntactic properties of responses

47We automatically segmented the responses into sentences using the nltk library (Bird et al., 2009) in Python. We then run the Berkeley Neural Parser (Kitaev & Klein, 2018) on our data and extracted clause and NP structures using the constituency parse trees generated by the parser. We did not do any manual correction on the parse trees although we observed considerable number of errors in them. Thus, the numbers presented in this section should be considered a rough approximation of data description rather than a robust analysis.

48Table 4 gives quantitative information on the syntactic structures in the responses. Subordinate clauses are more common for both mediums but the difference between the number of clause types is more prominent for the written medium. In addition, written responses contain less, and therefore longer, clauses than the oral medium, given that the written responses are longer than oral responses. As subordination is more common for both mediums, this result is not in line with what has been reported earlier, for instance, by Schäpers (2009). However subordinate clauses are much more dominant and longer in written responses. These observations are compatible with the results reported in Schäpers (2009).

49Third person pronouns and non-pronominal NPs have similar frequency distributions in both mediums, with third person pronouns being more common in the oral medium than in the oral medium with the slight difference of 1%. The length of the non-pronominal NPs (in terms of words) is computed as 2.89 for the oral and 2.86 for the written medium. This result does not confirm the findings in the literature where it is reported that NPs are longer in written medium (e.g., Aktaş & Stede, 2020). However, because the differences in these comparisons are not substantial enough to eliminate the inaccuracies caused by automatic parsing, we avoid making generalisations from them.

Table 4 – Syntactic properties

3.5.2. Discourse features in responses

50In order to explore the type of responses and their discourse relation to the stimuli, we created a subset of the data containing approximately 9% of the responses (i.e., 500 in total) with a balanced number of oral and written responses. Also, the sample is balanced for continuation length. We marked the responses in this subset using a scheme inspired from the Temporal and Expansion discourse relations (e.g., in the Penn Discourse TreeBank framework, Prasad et al., 2019). We define three distinct groups marked by E1, E2 and T as described below (S and R denote the clauses in the stimuli and the responses, respectively):

1. [S1 … Sn] Temporal [R1 … Rn] (= T):
   There is one event in S, and a temporally-later event in R. The stimulus event is usually stated in the first clause S1, while S2…Sn are additional elaborations. So, the relation holds between the complete stimulus and the response.

   [22]

   Stimulus: “Matthew went out for the first time after the lockdown was lifted. Restaurants were open and busy. The streets were empty because it was very cold outside.” Response: “He entered the warm restaurant to be met by the smell of good wine and strong garlic. He breathed in the scent, relieved as a feeling of normally settled over him.”

2. S1 … [Sn] Expansion [R1 … Rn] (E1)
   R adds some elaboration to the final clause of S. Importantly, the response does not introduce a new event (i.e., the relation is not Temporal).

   [23]	Stimulus: “Carol made a peppermint tea before the sun rose. Herbs can balance the emotions. Peppermint helps for anxiety so it was a perfect choice for this morning.” Response: “Peppermint also helps with bloating, and she suffers with this from IBS.”

3. [S1] … Sn Expansion [R1 … Rn] (E2)
   R adds some elaboration to the first clause in S. So, the relation holds between the first stimulus clause and the response. Importantly, R does not introduce a new event (i.e., the relation is not Temporal).

   [24]

   Stimulus: “Theresa published an article in Lancet. Lancet is a leading medical magazine. It covers everything about health. It publishes up-to-date studies. There is a review process before the articles are issued. Lancet accepts only original papers. Therefore, this is an impressive success.” Response: “Her article was about covid-19. She was very excited to have this published in Lancet.”

Table 5 – Discourse relations between the responses and the stimuli

	Relation type	Oral (%)	Written (%)	Total (%)
1	E1	12	10	11
2	E2	27	17	22
3	T	61	73	67

52Distribution of referential forms for the first reference to the main character in these responses is presented in Table 6. The statistics show that E1 and T type of relations trigger more frequent use of full NPs (names in this case) than the E2 relations.

Table 6 – Referential choice with respect to discourse relation type

		E1 (%)	E2 (%)	T (%)
1	Name	36	14	27
2	Pronoun	64	86	73

53We fitted a generalised mixed-effects logistic regression (Bates et al., 2015) to this subset of data to predict the binary “referential choice” (pronoun vs. name) from the relation type, with random effects for the participants and the stimuli. For the relation type, sum coding is used, with E1 being coded as (1,0), E2 being coded as (0,1), and T being coded as (-1,-1). The model results are reported in Table 7. As the model indicates there is a correlation between the relation type and the referential form. For instance, pronouns in E2 relations are used more frequently than the average pronoun usage for all the relation types (p < 0.05) which is in line with the frequency values presented in Table 6.

Table 7 – Summary of the final model (significance thresholds: ***p < 0.001, **p < 0.01, *p < 0.05)

	Estimate (log-odds)	Odds-ratio	S.E.	z val.	p
Intercept	1.27	3.53	0.22	5.76	***
relation-type1	-0.48	0.62	0.29	-1.64	0.10
relation-type2	0.56	1.75	0.26	2.12	*

3.5.3. Processing of responses

54For the investigation of our main research question, we marked the first references to the main character in the responses. For instance, first references to the protagonist in stimulus [25] (i.e., Robert) are marked as in responses [26] and [27].

[25]	Robert finally solved the mystery.

[26]	ROBERT gasped as he realised the truth. All this time his boss had told him to stop investigating and now Robert knew why.

[27]	It suddenly came to HIM out of nowhere as he looked out of the train window watching the fields, and forests go by.

55We do not count the pronouns used to refer to the main character but which do not match the gender. We received 36 responses with an unmatching gender reference. For some names, participants occasionally use short or casual forms. For instance, “Fred” or “Freddie” were used instead of “Frederick” and “Danny” instead of “Daniel”. We count these cases as instances of reference by the name.

56If there exist no reference to the main character in the response, as demonstrated in [29] which is a continuation to the story [28], we do not consider that response in the analysis. We received 443 responses without a reference to the main character.

[28]	Tony started organic farming for health reasons. Organic food is healthy because no chemicals are used in the process. It yields more food with less expense when it is done systematically. The products are much tastier: Tomatoes smell wonderful, peppers are flavorful.

[29]	Everything tastes fresh and natural. Things grow as it should be grown. Nature has a way of nurturing her harvest.

4. Results

4.1. Descriptive analysis

57As mentioned earlier, we only used proper names to introduce the main characters and did not provide additional details except for implying the person’s gender by using common male and female names. The participants used either a personal pronoun or the name for referring to the main character. We did not encounter any other referential forms denoting the main character in the responses, such as definite or indefinite noun phrases.

58We computed the distribution of the form of referential expression (pronoun vs. explicit name) of the first reference to the main character in the responses.

59The percentage of pronouns as the first reference (pronoun usage, henceforth) is 79.6% in oral and 77.1% in written medium. The average story length for which the participants used a pronoun in the continuations as the first reference to the main character is 3.56 clauses for the oral and 3.44 clauses for the written medium.

60Distribution of pronoun usage with respect to the stimuli length (in terms of clauses) is shown in Figure 1. It demonstrates that pronoun usage decreases with the increasing length of the stimuli for both mediums. The chart also indicates that pronoun usage is higher in oral responses than in written responses for longer stories.

61In order to get more information about the internal patterns of responses, we further show the distribution of pronoun usage for each story length via box plots (Figure 2) and bar plots demonstrating pronoun usage for each stimulus (Figures 3-7).

62Figure 2 indicates that the median values of pronoun usage are greater in the oral data for 2-, 3-, 5- and 8-clause-length stimuli, whereas the median is obviously higher in written data for 1-clause-length stories. The median of the written responses for 1-clause-length is also greater than all other values in the corresponding oral responses. For the 8-clause-length stories, the oral median is only slightly smaller than the upper quartile of the written data and the upper quartile of the oral data is greater than all the values in the distribution of written responses. As a result, we conclude from this box plot representation that pronoun usage is higher in written responses for very short 1-clause-length stories but higher in oral data for long 8-clause-length stories. We cannot make such clear inferences from the box plots demonstrating the distributions for 2-, 3-, and 5-clause-length stories.

4.2. Generalised mixed-effects logistic regression

64We fitted a generalised mixed-effects logistic regression model to the responses in order to evaluate the factors influencing the referential choice, using the lme4 package (Bates et al., 2015) in R. The dependent variable (i.e., the first referential choice for the main character in each response) is a binary variable, with the possible values of Pronoun and Name. We considered the “stimulus length” as a numerical variable rather than a categorical one and centered it around its mean. We applied sum coding for the “medium”, with the “oral” medium being coded as 1 and the “written” medium being coded as -1.

65We increased the complexity of the model gradually. Table 8 shows the summary and comparison of the models via likelihood ratio tests (done by the anova method in R). We first fitted a fixed-effects model (m0). Medium, length of the stimuli and their interactions are considered as fixed effects in this model. Next, in order to address the differences between participants, we added a random intercept of participants to the model (m1). The likelihood ratio test indicates that m1 is a better model because it has a smaller AIC (Akaike Information Criterion) value, as shown in Table 8. In the next step, we added a second random effect for the stimuli (m2) which also improved the model significantly (p < 0.001). Adding a random slope over “stimuli” leads to the singularity of the model. A random slope for “medium” over the “participants” (m3) does not cause singularity and contributes to the model significantly (p < 0.05). Therefore, we include random slope for the medium over participants. We did not include the participants’ features in the model such as gender and age because we do not control these variables in our experiment.

Table 8 – Comparison of the generalised mixed-effect models fitted to the data (significance thresholds: ***p < 0.001, **p < 0.01, *p < 0.05)

Model	Formula	AIC	R2(f)	R2(t)	Anova	p
m0	ref_expr∼medium*length	5,554	0.05	-	-	-
m1	ref_expr∼medium*length + (1|participant)	4,518	0.08	0.51	m1,m0	***
m2	ref_expr∼medium*length + (1|participant) + (1|stimulus)	4,384	0.09	0.56	m2,m1	***
m3	ref_expr∼medium*length + (1+medium|participant) + (1|stimulus)	4,381	0.09	0.57	m3,m2	*

66The column R²(f) shows McFadden’s pseudo-R² value for fixed effects, whereas the R²(t) column represents the conditional R², which describes the proportion of variance explained by both the fixed and random effects. The R² numbers in the table indicate that the model without random factors fits to our data very poorly (i.e., explanatory power is 5% for m0). In contrast, the total explanatory power of a mixed-effects model is substantial (conditional R² is 57%) and the part related to the fixed effects alone (R²(f) = 9%) is also higher than the R²(f) of the fixed-effects logistic regression (m0).

67The summary for the fixed effects of the final model (m3) is demonstrated in Table 9.

Table 9 – Summary of the final model (significance thresholds: ***p < 0.001, **p < 0.01, *p < 0.05)

	Estimate (log-odds)	Odds-ratio	S.E.	z val.	p
Intercept	1.93	6.87	0.16	12.05	***
medium	0.09	1.10	0.092	1.97	*
length	-0.32	0.73	0.04	-7.53	***
medium:length	0.03	1.03	0.036	2.05	*

68The Intercept in Table 9 indicates that the mean value for pronoun usage across the dataset is substantially greater than name usage in the fitted model for the average length of stimuli (odds-ratio = 6.87). Participants seem to use fewer pronouns in written medium than in oral medium for the average length of stimuli, and the difference between mediums is statistically significant. The model coefficients indicate that an increase in the length of the stimuli decreases the pronoun usage significantly (p < 0.001) for both mediums. The decrease in pronoun usage, as demonstrated in Figure 8, is more prominent for the written medium than the oral medium for the increasing length of stimuli (p < 0.05). This inference is compatible with the data demonstrated in Figure 2, which indicates that pronoun usage in the written medium is affected more than the oral medium by the increasing length of the stimuli.

5. Discussion

69Our first observation on our analysis concerns the type of texts produced in response to stimuli. We presented an analysis of discourse relations on a subset of the experiment data in Section 3.5.2. According to the findings, 67% of all responses introduce a Temporal relationship to the stimuli by introducing a temporarily-later event. According to the framework introduced by Labov and Waletzky (1967), these responses form a temporal juncture with the event introduced in the stimulus, and therefore these stimulus-response pairs can be considered as narratives (i.e., Labov and Waletzky define narrative as a sequence of clauses which contains at least one temporal juncture). Our analysis also shows that written responses have more Temporal discourse relations between stimuli and responses than oral responses. The interplay between the realms of narrative structure and reference was studied by Klein and Stutterheim (1992), who used the term “referential movement” in stories consisting of a “main structure” that is at various points interrupted by portions of “minor structure” (descriptive, elaborative content where the temporal references do not move forward in time). We see our distinction between coherence relation types as making the same point: In the narrations, the “T” cases represent a switch from a preceding “minor structure” back to the “main structure”. Klein and Stutterheim to our knowledge did not investigate the role of pronominalization, but we assume that the use of a full NP as opposed to a pronoun serves as a signpost to the audience for making the switch back to the main structure. Constructing this signpost, we furthermore assume, requires planning work that is undertaken more often in writing than in speech, which is compatible with the observation that T discourse relations are established more often in written responses than oral responses in our experiment.

70The second observation we made based on the results shown in Section 4 is regarding the dominant use of pronouns for both mediums as a first reference to the main character in responses. The statistics reported in Section 3.5.1 suggests a pattern similar to what is reported by Aktaş and Stede (2020), who found that non-pronominal NPs are more common than third person pronouns for both mediums. However, the expression used for the first reference to the main character in our responses does not reveal a similar pattern because at least 60% of the first references are pronouns for both mediums. This result is not directly comparable to the results reported by Aktaş and Stede (2020) (i.e., this result only applies to the first referential form referring to the main character in the responses, not to all referring expressions), but its significant departure from the statistics in Section 3.5.1 is remarkable. Our findings are comparable to and consistent with the findings of Fox (1987), who reveals that in the absence of interfering referents between the antecedent and the anaphoric expression, pronoun usage is significantly prevalent in both mediums. Arnold and Griffin (2007: 521) state that “even when a pronoun would not be ambiguous, the presence of another character in the discourse decreased pronoun use”. We believe the high number of pronouns referring to the main character for both mediums (i.e., no difference between oral and written) might result from the fact that we introduce only one human (or animate) entity in our stimuli, so there is no competition with another entity.

72The fourth observation is regarding the distribution of pronoun usage for varying lengths of the stimuli. Figures 3-7 show that the patterns for 1- and 8-clause length stimuli are regular, with written responses containing more pronoun usage than oral responses for 1-clause stimuli and the opposite tendency for 8-clause stimuli. A closer look at Figures 3-7 reveals irregularities in the seemingly regular patterns. For instance, written responses to stimulus S34 in Figure 7 (given in [30]) include more pronoun usage (51%) than the oral responses (47%). Consequently, S34 does not fit the general pattern of 8-clause-length stories. In addition, this stimulus triggers less pronouns than the other stimuli with the same length (see Figure 7). Another stimulus (S36) in Figure 7, which is given in [31] represents a regular pattern with much higher pronoun usage.

[30]	Danny moved to an island. It was rather small. Vehicles were not permitted there. It was a mysterious place, too. There was a volcano in the South. One night, the volcano became active, then the sky brightened. That was really scary.

[31]	Gabriela moved to a new neighborhood. The previous area was chaotic because it was very central. This new neighborhood is quiet. There are parks nearby. The new house is great. It is quite peaceful. A whole new life can start here.

73In our experimental procedure, the event brought about by the main character is introduced in the first clause for the stimuli longer than one clause, which is usually followed by elaborative clause(s) establishing Expansion relations (see Section 3.5) with the first clause. Different from the general pattern, the stimulus in [30] contains a second event temporarily ordered (a “complicating action” clause according to Labov & Waletzky [1967]) with respect to the main character’s action introduced in the first clause. The more regular stimulus in [31] does not contain a complicating action, and therefore cannot be considered as a narrative according to Labov and Waletzky (1967). The subset we used for the discourse relation evaluation in Section 3.5.2 contains 18 responses to stimulus in [30] and 14 responses to stimulus in [31]. Responses to [30] contain only E1 (11%) and T (89%) type of relations, whereas responses to [31] contain all three relation types (14%, 14% and 72% for E1, E2 and T, respectively). This indicates that responses to [30] does not elaborate on the main character’s action introduced in the first clause, but introduces new events or expand the second action in the stimulus, whereas responses to [31] relate to the stimulus with Expansion relation (E2) elaborating the main action (“orientation” according to the framework of Labov & Waletzky [1967]). This observation indicates that the type of the stimuli (e.g., narrative or not) has an impact on the type of relation established by the response. Therefore, considering the correlation between the relation type and referential choice discussed in Section 3.5.2, we can hypothesize that the type of the stimulus affects the form of the first reference to the main character. However, this hypothesis needs further investigation, with all the stimuli being analysed for their qualitative features and the relation types with their responses being labelled for a larger number of instances.

74The fifth observation on the data presented in Section 4 is that pronoun usage decreases with increasing anaphoric distance for both mediums, which is consistent with previous comprehension research (Clark & Sengul, 1979; Streb et al., 2004), with the assumption that longer pronoun comprehension time means less pronoun usage in production, for a setting with long anaphoric distance. The difference in pronoun usage for the written medium between 1-clause-length and 8-clause-length stories (in Figure 1) is 32%, whereas it is 20% for the oral medium. This finding is consistent with prior corpus-based research (Fox, 1987; Aktaş & Stede, 2020), which indicated that oral texts allow for greater anaphoric distance than written texts. In our interpretation, the difference in the decrease of pronoun usage between oral and written mediums indicates that anaphoric distance has a larger effect in referential choice for the written medium, which is also confirmed with the generalised mixed-effects analysis in Section 4.2.

6. Summary and conclusion

75The aim of this study is to investigate the differences across oral and written mediums, in terms of the impact of anaphoric distance on the referential choice in language production. Relevant contrastive corpus-based studies (Fox, 1987; Aktaş & Stede, 2020) indicate that oral texts include longer average anaphoric distance than written texts between third person pronouns and their antecedents, where the distance is measured in terms of clauses. In relation to these findings, we hypothesized that oral texts will contain more pronouns than written texts, in the case of long anaphoric distances. We designed an experimental setup to test this hypothesis, where we manipulated the anaphoric clause-based distance systematically while keeping constant the features of the antecedent, such as animacy, plurality, grammatical role and the form of referential expression (i.e., name or pronoun). Previous studies (e.g., Schäpers, 2009) demonstrate that clause lengths, types and tokens differ between oral and written mediums. Therefore, we manipulated the distance in terms of number of clauses, but controlled the length of clauses, type of tokens and the coherence relations linking the clauses, in order to obtain substantial information about the impact of the medium.

76To the extent of our knowledge, this is the first experimental study investigating the medium distinctions with respect to the impact of anaphoric distance in a production task.

77We used short stories varying in length as the stimuli and expected the participants to continue these short stories by imagining “what happened next to the main character?”. This experiment was conducted via crowdsourcing. Only 5% of the responses had to be eliminated due to, for instance, bad audio quality or texts on topics different from the stimulus. Therefore, this study indicates that crowdsourcing can provide high quality data for such kind of tasks where participants generate language data for a given context, in both oral and written forms.

78Our first hypothesis concerned the distribution of third-person pronouns in oral and written responses in relation to non-pronominal NPs. Based on prior findings in the literature (e.g., Fox, 1987), we expected to detect more third person pronouns in the oral than in the written medium. As summarised in the previous sections, we did find a confirming statistical effect concerning the syntactic types of the first references to the main characters in the responses. However, frequency distribution of all NPs, which is obtained by automated parsing, does not show a strong indication for this direction. As a result, based on our data, we believe the first hypothesis is only partially confirmed. For extracting significant conclusions about all of the NPs across mediums, further analysis based on robust parsing is required.

79In the analysis, we observed that participants use fewer pronouns when the story length (and hence, the anaphoric distance) was increased, in both mediums.

80However, the gap for the average pronoun usage between 1-clause-length stories and 8-clause length stories is greater in written responses (32%) than in oral responses (20%). We fitted a generalised mixed-effects logistic regression model to this data. The model coefficients indicate that pronoun usage is smaller in the written medium than the oral medium when the stimuli length is increased, as shown in Figure 8, which clearly illustrates that pronoun usage decreases more for the written than for the oral medium for the long stimuli. Therefore, we infer that the anaphoric distance has a larger effect on the written medium. Exploration of pronoun usage patterns for each stimulus length reveals that pronoun usage is more common in oral responses than in written responses for long stories, which confirms our second hypothesis and the findings of the above-mentioned comparative corpus studies.

81We observed variation in response to the stories having the same length. In our experiment, we did not control potentially explanatory variables for this variation, such as text type of the stimulus (e.g., narrative or not), number of referring expressions in the stimulus and length of the name of the main character in the stimulus. Continuing the same line of research and conducting experiments with different control setups can help to better understand the variation for the same length of the stimuli.

82In addition to the analysis presented in this paper, we are also planning to extend the scope of this work by performing computational experiments on the same data. Same and Van Deemter (2020) made an extensive evaluation of the feature sets used in a variety of computational systems for generating referring expressions. Their evaluation reveals the importance of four feature sets, three of which related to the antecedent (animacy and plurality, grammatical role, and form), which we controlled and kept constant in our stimulus design. The fourth important feature set that Same and Van Deemter (2020) use is recency, which is the experimental factor in our experiment. We believe that computational classification experiments on our data can give interesting insights into the impact of recency, as the other potentially important features are kept constant.