1This paper presents how learners of a second language use machine translation (henceforth MT), what are their attitudes towards it and how MT impacts their written production. It focuses on the Slovene language as a case study for an official European Union language with a relatively small number of speakers—there are approximately 2.5 million speakers of Slovene as L1—and with quickly developing, yet relatively fragmentary, language technology support (Krek, 2022, p. 18). The research questions which I seek to answer in this paper began arising during the compilation of the Slovene learner corpus KOST (Stritar Kučuk, 2024). KOST1 comprises mainly essays written by university students from the South Slavic-speaking area who study at the University of Ljubljana, the largest Slovene university. Due to the historic relations in the region and also due to administrative reasons, they are mostly speakers of languages closely related to Slovene: central South-Slavic (Bosnian, Croatian, Montenegrin and Serbian) and Macedonian. To facilitate their integration in the study process and boost their language learning, the university offers them the Year Plus2 programme. Its main features are two language courses, which the students can attend free of charge in the first year of their studies while simultaneously studying their main subject.
2During the first years of the Year Plus programme, students submitted weekly written homework assignments in L2 Slovene. However, since the start of the KOST compilation in 2019, its compilers have been facing issues due to an increasing use of MT by students. At first, it could be identified by back-translating unusual, unexpected, mostly lexical errors, which were likely the result of students’ incorrect use or poor performance of MT. By 2020, the performance of MT had significantly improved (Ducar & Schocket, 2018, pp. 780–781). Since then, Slovene language teachers have identified essays written using MT through their low number of errors. The language proficiency of these texts is much higher than that of the texts that the students produce when speaking or writing in the classroom. The teachers, who are also some of the main KOST compilers, subsequently began to question whether these texts accurately reflect their students’ language proficiency. And, assuming that learner corpus data “represents language as produced by foreign or second language (L2) learners” (Gilquin, 2015, p. 9) and “seeks to be representative of this language variety” (ibid.), the question also arises whether texts written using MT support truly represent such language.
3According to teachers’ observations, the use of MT has increased significantly among students after the aforementioned quality improvement of MT and the shift to digital teaching during the Covid‑19 pandemic in 2020. The key question in this paper is therefore: do texts written by students with the assistance of MT provide insight into their language proficiency and learner language characteristics?
4To answer this research question, I also set some additional research questions:
-
How do the participants in my study use MT—what tools do they use, how frequently and in what contexts? What languages do they include in the MT process? Do they post‑edit their machine translations and what errors do they correct?
-
To what extent are they satisfied with MT?
-
Do they use MT when writing homework in their Slovene L2 class?
-
How time-consuming is text production for them in general?
-
Do students who are more proficient in Slovene use MT less?
-
Are there significant differences regarding the use of MT between speakers of languages that are more closely related to Slovene (such as the Central South Slavic languages) and speakers of, for instance, Macedonian?
-
Are students who initially have a better knowledge of Slovene more independent and do they rely less on MT?
5Finally, this paper attempts to consider how learner corpora should address the growing use of MT. Should texts that heavily rely on MT be included in KOST (and other learner corpora)?
6Much has been written about corpora in MT, learner translation corpora (cf. Granger & Lefer, 2023; Castagnoli, 2022), the use of MT in language teaching (Polakova & Klimova, 2023; Carré et al., 2022; Lee, 2022; Case, 2015; Ducar & Schocket, 2018), and the acceptability of texts produced by MT (Giamperi & Harper, 2022). Several studies have been conducted on the language learners’ use of MT as well as on their attitudes towards MT (Nassau et al., 2022; Ata & Debreli, 2021; Yang et al., 2023; Bin Dahmash, 2020; Clifford et al., 2013; Agustine & Permatasari, 2021). However, to my knowledge, little has been said about the impact of MT on learner production and only a few experiments have been conducted on this topic so far. For example, Abimbola (2023) used this method to analyse vocabulary acquisition and reading comprehension of 60 undergraduate Nigerian students of English. The results showed that the students who studied using MT had better outcomes for vocabulary acquisition and reading comprehension, and most students found MT helpful.
7Capturing the impact of MT on language acquisition scientifically is a challenging task. To find answers to the questions raised, I undertook several research steps.
8Initially I conducted two surveys among university students who studied Slovene as L2 within the Year Plus programme to obtain information on their use of MT and their attitudes toward it. 99 students responded to the survey in 2020 and 104 students responded in 2021. The surveys were extensively analysed in Stritar Kučuk (2021) and Stritar Kučuk (2022b) and showed that the students’ attitudes towards the use of MT are largely positive—though not always so—, and that they use it frequently and in various contexts, from academic to everyday situations. This corresponds to findings from other researchers (cf. Ata & Debreli, 2021; Yang et al., 2023; Bin Dahmash, 2020; Clifford et al., 2013; Agustine & Permatasari, 2021).
- 3 All the teachers working in the Year Plus programme took part in the experiment: Meta Klinar, Nataš (...)
9However, surveys may not accurately reflect the informants’ practices and can be subjectively biased, so their results may be far from the truth about the actual use of MT. Additionally, surveys do not provide insight into the impact of MT on language acquisition, so I decided to use the experimental approach. The experiment was conducted by the Year Plus teachers,3 who compared two texts written by Year Plus students on two separate occasions, one written using on‑line tools and the other written without any help.
10All the students who were learning Slovene as L2 in the Year Plus programme during the academic year 2021/22 were included in the experiment during the spring of 2022. They were asked to write two texts: an experimental one and a control one.
11During the first writing session, students composed an experimental text which intended to reflect their usual approach to writing in Slovene. The essay was written on a computer approximately one month before the end of the second term of the Slovene language course, during the final stages of each student’s language learning within our programme. The writing took place in a computer lab, and the students were permitted to use either their own personal computers or those provided by the university. The activity was announced to the students beforehand, but they were unfamiliar with the topic of writing, “When I was a child”. None of the groups had covered this topic in the course. However, it was general enough, and based on the material we had worked through with the students, we expected all of them to be able to write an essay about it. To assist them in composing their text, we provided some additional questions. The essay was required to have a minimum of 150 words and the students were given 45 minutes to write it.
12Prior to commencing the writing process, the students were divided into two groups:
-
- 4 Some students write their homework with the help of a person who speaks Slovene. This option was no (...)
71 students were instructed to produce the text in their usual Slovene homework style. They were given the option to write in Slovene without any aids, to use MT in any language of their choice, or to use other language tools they were familiar with.4 This group will be referred to as group IC (individual choice of writing) throughout.
-
60 students were instructed to write the text in any language of their choice except Slovene and translate it into Slovene using any available MT tool. This group of students will be referred to as group MT (writing using MT).
13Students in both groups were allowed to use any software to compose their essay, but they were required to submit it on the Year Plus e‑learning platform, which they were all familiar with. After writing, they were asked to complete an online questionnaire survey. It was designed in two variants, depending on the group, but the questions in the two versions largely overlapped, covering the following information about how they wrote the experimental text and how they usually used MT:
-
the writing style of the experimental text and their usual Slovene homework writing style;
-
post-editing of the experimental text;
-
time needed to write the experimental text and the usual Slovene homework;
-
the use of MT (various softwares, languages included in the MT process, frequency of use in various contexts, size of language units usually translated);
-
general satisfaction with MT.
14The questions were mostly closed-ended, although a few permitted open-ended responses. The survey was not anonymous so that we could link the students’ answers to their texts.
15The second writing session took place a month after the writing of the experimental text, during the students’ final written exam at the end of the second semester of their course. Students produced a control essay reflecting their language proficiency in Slovene as L2. The essay prompt for all students was “My Year Plus” and required a 150‑word response. The writing instruction provided clear guidance on the main points to be covered: students’ language course, main studies, and general student life. These texts were written on paper under strictly controlled writing conditions without the use of aids.
16All the essays from both writing sessions were rated based on the criteria used for the assessment of Slovene exams at different levels at the Examination Centre of the Centre for Slovene as a Second and Foreign Language.5 The students enrolled in our programme are predominantly speakers of languages closely related to Slovene (cf. chapter 4.4) so they typically start to learn Slovene at an approximate level of A2 and, depending on various factors, reach the level B1 or B2 by the end of their participation in Year Plus. Knowing our population and the range of their Slovene writing skills, we concluded that the most appropriate criteria for assessment are at the B2 level. Still, we made slight adjustments to these criteria (Table 1) so the maximum grade was 10 points. Year Plus teachers who rated the essays calibrated their assessments at a coordination meeting.
- 6 The criteria and their descriptors were defined in Slovene; the English version was translated by t (...)
Table 1. – Criteria for assessing texts.6
|
Grade
|
Content and coherence
|
Vocabulary
|
Grammar and orthography
|
|
4 points
|
/
|
Vocabulary is varied, not repetitive and used appropriately. There are only a few minimal errors.
|
Language structures are used correctly (syntax, morphology, orthography). Minor errors are tolerated.
|
|
3 points
|
/
|
Vocabulary with a few minor errors that do not significantly affect comprehension; or limited vocabulary is used.
|
Simple language structures (syntax, morphology) are mostly used correctly. Orthography (spelling, punctuation) is correct enough to follow the message. There may be several errors in inflections, word gender, word order.
|
|
2 points
|
The content is relevant, and the text is coherent. Some more complex structures are also used.
|
Vocabulary is very limited or there are many errors.
|
Errors are systematically repeated (in syntax, morphology, orthography), e.g. frequent mistakes in inflections, word order.
|
|
1 point
|
The text is deficient in content, the task is not fully completed. The text is poorly linked/coherent, or has only a limited range connectors (in ‘and’, potem ‘then’).
|
Many vocabulary errors make the text hard to understand.
|
Due to many inaccuracies in the use of language structures, comprehension is difficult.
|
|
0 point
|
Irrelevant content, text too short, not coherent.
|
The vocabulary is inappropriate and/or not Slovene, making the text unintelligible.
|
Comprehension is impossible due to many inaccuracies in the use of language structures.
|
17The texts were distributed among all participating assessors and anonymised to ensure objectivity. No teacher evaluated his/her students’ texts, and each text was independently assessed by two assessors. We ran a simple consistency test of the evaluation and the assessors and found that in 70% of cases, the two assessors either agreed or differed by one mark. For the purpose of this research, the inter‑rater reliability was thus satisfactory (cf. Ferbežar, 2019). The final grade was determined averaging the two grades. After assessing, I calculated the average scores for various groups of students and compared the results. This provided both quantitative and qualitative data.
18All the data was collected using simple, commonly used software. Students uploaded their experimental text to an online classroom in the Moodle environment,7 from which it was transferred to a text document. The control text was handwritten by the students and digitised by the teachers, after which it was also saved in a text document. For students, the online questionnaire was accessible on a common online platform,8 and the answers were transferred to Microsoft Excel. The analysis and evaluation of the texts was conducted manually.9 Subsequently, all metadata pertaining to the students and the text grades were attached to questionnaire results in Microsoft Excel. This programme was then used to process all the quantitative data.
19131 students participated in the study. In this report, Bosnian, Croatian, Montenegrin, and Serbian speaking students will be considered as a homogenous subgroup (referred to as BCMS) to simplify the analysis, although many regard these as four standard languages developing from a common Serbo-Croatian standard base (Požgaj Hadži & Bulc, 2022, p. 9). However, speakers of these languages face similar challenges when acquiring Slovene, and their linguistic production in Slovene is also comparable (Klinar et al., 2022, p. 189). The other significant subgroup based on their L1 are speakers of Macedonian, which is a language less closely related to Slovene than BCMS. Traditionally, its speakers have had more difficulties learning Slovene (Nikolovski, 2022; Arizankovska, 2009). To avoid skewed statistical results due to individual specifics, non-South-Slavic speaking students were excluded from the analysis: 3 Spanish speakers, 2 Polish speakers and 1 Russian speaker.
- 10 Given that the group of more advanced learners is relatively small and significantly smaller than t (...)
20A total of 125 students who provided all the necessary data for the research, including two texts and one questionnaire, were finally included in the analysis. Of these, 67 were in the IC group and 58 in the MT group. The distribution of their L1 is shown in Figure 1. 69% of the students were female and 31% male. The majority (82%) were speakers of a South Slavic language who had not learned Slovene before they started the Year Plus programme. The remaining 18% were placed into more advanced groups.10
Figure 1. – First languages of the students.
21This chapter presents the results of the questionnaire analysis and the text grading experiment. To facilitate comprehension, the analyses are discussed after the presentation of each set of results.
22Let us start the review of my findings with crucial questions for language teachers: In which language and with what aids do students produce homework essays in the language course? Do they write in the language they are learning, do they opt for a language they are more proficient in and then translate it, possibly using MT or other language tools, or do they write in another way? This study obtained information from two sources. For students from group IC, the experimental text they wrote supposedly reflected their usual homework writing style. Additionally, data from MT students who answered a survey question about this was included. Figure 2 shows the combined data and indicates that the majority of students choose to write in Slovene. Most of them use MT, although the extent to which they use it remains unknown. Fewer students use other language tools, and only a small number write their assignments without external help.
Figure 2. – Slovene homework writing style.
- 11 All quotes from the assessors were written in Slovene and translated into English by the author of (...)
- 12 All persons in the article are referred to using the masculine gender as grammatically neutral.
23Some of the choices may have been influenced by the experimental set‑up. For instance, one student who wrote in Slovene without any aids produced an experimental text that the assessor deemed as “the worst written text, most of the text towards the end written in BCMS, at the very end it was written in English”.11 The student’s choice of writing style could be due to a misunderstanding of the instructions or his12 desire to impress the assessors.
24One could expect that students who are more proficient in Slovene would be more independent and could rely less on MT. However, in my small sample, this was not found to be the case. Out of the students who learned Slovene in the advanced group, only one wrote his homework without assistance and two students used different language tools, while the rest wrote in Slovene with the help of MT.
25Let me comment briefly on the time that students needed to write the experimental text. This information was submitted in the survey. Students from the IC group mostly needed 15 to 30 minutes, while those from the MT group needed even less, mostly less than 15 minutes (Figure 3). Writing short essays is therefore not particularly time-consuming for students. However, according to Figure 4, they usually need a little more time when writing their actual homework. Most students attempt to write in Slovene, taking mostly between 15 and 30 minutes but also up to 45 minutes. It is important, however, to consider the potential for subjective bias in their responses.
Figure 3. – Time needed to write the experimental text.
Figure 4. – Time needed to write the usual Slovene homework.
26The results for the MT softwares students use are shown in Figure 5. Only four students do not use Google Translate.13 15 students also use Pons14 and 9 students use DeepL15 while other tools such as Yandex Translate16 and Presis17 are much less common. Beyond the MT software itemised for possible answers in the survey, 6 students mentioned tools that are not necessarily MT software, including Glosbe,18 the grammar checker Language Tool,19 and the Slovene dictionary portals Termania20 and Fran.21 In short, as only two students did not choose Google Translate as one of the MT softwares they use, it can be concluded that for a vast majority of students, MT means Google Translate.
Figure 5. – MT softwares used by students.
- 22 As the questionnaire was completed by students who are accustomed to this type of activity, the pre (...)
27To obtain information on the contexts in which students use MT, students assessed the frequency with which they use MT in their basic studies, Slovene language course, and everyday life. For each context, they had to select a frequency option on a scale ranging from “never” to “very frequently”.22 The results indicate that students use MT in all contexts (Figure 6). None of the students selected “never” in all three contexts, indicating that they use MT at least occasionally. MT is used the least in the Slovene course itself, which is of course desirable for teachers, but the question is to what extent these answers accurately reflect the actual practice. In the Slovene language course, only five students out of 125 never use MT.
Figure 6. – Frequency of using MT in different contexts.
28The results suggest that students use MT more frequently in formal than in informal contexts. These findings are, to some extent, expected for speakers of closely related languages whose mutual understanding is so high that the use of language tools in everyday situations is less necessary (Golubović & Gooskens, 2015). It should also be noted that—according to informally obtained data from teachers—a significant proportion of students reside with roommates who are also not native Slovene speakers. In such circumstances, the use of Slovene may not be indispensable for them in everyday life. However, further research could investigate how the use of MT varies according to the length of stay in Slovenia. It would also be interesting to delve deeper into the changes that happen to the use of MT in different contexts over time.
29The selection of languages used by students in the MT process is an important consideration, as their first language and Slovene are often supplemented by English as their first foreign language. The first source of information on this is the language in which the students wrote the experimental text. In the MT group, which was required to write the text in a language other than Slovene, 47 students chose to write in their L1 and 11 in English. In the IC group, only 8 students opted to write in a language other than Slovene, with 5 writing in their L1 and 3 in English. All but one of these students were Macedonian speakers and all had studied Slovene in beginner groups.
30BCMS students apparently prefer to translate from their L1 rather than from English, but with Macedonian speakers from both groups, about the same number of students chose to translate from Macedonian and English. This could be due to the level of development of language technologies for Macedonian, which was still quite low at the time of this experiment (Ljubešić, 2023). In my survey, students explained that they chose English rather than their L1 because it is more comfortable or easier for them (4 responses from BCMS and 4 from Macedonian students) and because MT translates better from English (4 responses for BCMS and 2 for Macedonian students).
31These results align with the students’ questionnaire responses regarding the language they usually translate from (source language) and the language they translate into (target language). Slovene is more frequently used as the target language (Figure 7). When translating, they primarily translate from their first language, when it is BCMS, or from English. Two students stated German as their target language and one student stated Russian, Italian and German as their source language. The reasons to include these languages into the MT process were not pursued further.
Figure 7. – Source and target language of MT according to students’ survey answers.
32Post-editing of machine-translated texts is frequently used in language classrooms (Ducar & Schocket, 2018). Although Year Plus teachers do not generally encourage students to use MT, we do remind them to post‑edit their MT texts.
33In the survey, 106 students who had translated texts at least partially using MT were asked if they checked the automatic translation. 87 of these participants confirmed this and 75 also confirmed that they had made corrections to their text afterwards. Figure 8 displays the number of corrections made by the students. It is unclear whether students also check machine translations of texts when writing homework or similar productions.
Figure 8. – Number of corrections students made while post‑editing their experimental text.
34According to students’ responses, the most common correction made was to the gender of verbal forms and adjectives (34 responses). This is a well‑known issue with MT software for Slovene (Popović & Arčan, 2015, p. 102), as it cannot determine the speaker’s gender. Some students provided more general feedback, such as “I corrected something that didn’t sound right”, while some gave specific examples, such as changing the word “oddaj” (‘shows’) to “filmi” (‘movies’).
35The students’ post-editing of the texts they copied from MT was confirmed by the assessors, who identified an important number of errors that were probably not produced by the MT system but were typical for BCMS (Balažic Bulc, 2004) or Macedonian (Nikolovski, 2022) speakers. These errors, such as erroneous word endings that are typical of the students’ L1 and non‑existent in Slovene, might have been introduced by the students themselves. However, the assessors also corrected numerous instances of the aforementioned incorrect gender usage in verbal forms. If the students had actually reviewed the texts, they should have identified and corrected these errors. It is possible that they simply overlooked them, but as one assessor noted, incorrect gender usage is a fundamental error that warrants the lowest grade.
36The survey asked students to rate their satisfaction with MT. The exact MT software was not specified, but the responses shown in Figure 5 suggest that it probably meant Google Translate for most students. As will be evident further in this paper, some of them also referred to Google Translate explicitly in their additional comments.
- 23 All the quotes in this section are from students’ surveys, translated into English by the author of (...)
37The majority of responses were positive, with most students expressing satisfaction (Figure 9). A third of students provided additional comments, with a few only highlighting their satisfaction: “My vocabulary is not great in Slovenian, and Google Translate helps me because it gives me new words and the whole sentence gets better form.”23 Fifteen students expressed dissatisfaction with the use of masculine and feminine gender in the translations, citing mistakes. However, some noted that these errors could be corrected: “We can see it every time and we can correct it.” Another fifteen students pointed out issues with inappropriate, nonsensical, and incomplete translations, particularly with technical terms (“It is not very useful for translating geodesy terms”), phrases, or more specific words. For example, Google Translate translated the English phrase “good job” to “dobro opravljeno” (‘well done’), while the student actually wanted to get “dobro delo” in the sense of work or profession. Additionally, some students commented on the performance of MT when translating into different languages: “Sometimes when I translate from English, Google Translate does not return the word I need — then I try to translate from Montenegrin, and most of the time it works.” It was also noted that translations should always be checked for accuracy.
Figure 9. – Students’ satisfaction with MT.
38In this context, students also responded on the size of language units they typically translate using MT. Previous surveys have shown that users primarily use MT to translate individual words or phrases, less frequently paragraphs, and rarely entire texts (Clifford et al., 2013, p. 111; Ata & Debreli, 2021, p. 110). My survey results (Figure 10) confirm this trend, which is supported by several instances of errors in texts translated using MT. For example, a BCMS student wrote: “sem bil pritrjen nanj” (‘I was attached to it’). In this case, the adjective “pritrjen” is used for physical attachment and should be replaced with “navezan” to convey the student’s emotional attachment to his computer. It appears that the student didn’t follow the instruction to translate the entire text, or at least sentence, and instead only looked up the single word. Based on experience with MT for Slovene (cf. Škerl, 2016), the Year Plus teachers often advise their students that MT works better with longer units. It is concerning, therefore, that students still tend to translate single words the most (cf. Cotelli Kureth et al., 2023). However, it is understandable that whole texts are not translated as frequently, as my respondents are speakers of closely related languages and therefore have a good understanding of Slovene, making translation of larger units unnecessary.
Figure 10. – Frequency of using MT with language units of different sizes.
- 24 The Pearson correlation coefficient for the IC group is 0.175 and for the MT group it is 0.386.
39Finally, let us include the control texts into the analysis and compare grades of both sets of texts. When focusing solely on the experimental texts, it is evident that students from the MT group received significantly higher grades (average 8.52, standard deviation 1.13) compared to the IC group, who received a lower average score (7.38, standard deviation 1.81). By adding the ratings of the control text (Figure 11), the difference between the average scores for students in the IC group is only slightly in favour of the experimental text (the average score for the control text is 7.26, standard deviation 1.74) while for students in the MT group, the average score of the control text (7.46, standard deviation 1.42) is lower than that of the experimental text. This could suggest that MT produces Slovene texts written at a higher level of language proficiency than L2 students produce on their own, although the Pearson correlation coefficient for both groups indicates only a weak positive linear relationship.24
Figure 11. – Average scores of students from both groups.
40When examining sub‑groups, it becomes apparent that students in group IC (Figure 12), who typically write their homework in Slovene (with or without assistance), achieved higher scores for the control text compared to their peers who write their homework in L1 or English. The control text of students who write in Slovene received a higher score than their experimental text. This could be due to the fact that they acquired more knowledge of Slovene during the period between the experimental and control text. It is worth noting that there are exceptions to this trend, such as a student from Macedonia who received a score of 10 for his experimental text. One assessor commented: “Written in a most beautiful and nice way.” The control text of the same student received only 5 points, with one assessor commenting that it was “very non‑Slovene”.
41For most students who write their homework in Slovene with the help of MT, the grades for both texts are generally similar. However, for students who typically write in Slovene without any assistance, the control texts scored even slightly higher than the experimental texts. It can be assumed that students who rely less on MT or at least attempt to produce their homework in Slovene, put more effort into learning the language, so they are more successful in language acquisition. On the other hand, it is also possible that these students have better second language performance or attention capacity in general (cf. Skehan, 2023).
Figure 12. – Average scores of students from group IC.
42Students who write their homework in a language other than Slovene, generally receive significantly lower grades for their control text compared to their experimental text. For example, a Macedonian student writing in English received a grade of 9 points for his experimental text, losing a point mainly due to the incorrect use of masculine gender in verbal forms. The same student received a grade of 6 points for his control text. Once more, a plausible explanation for this is that students who write their Slovene homework in their L1 or in a foreign language which they are more proficient in than in Slovene make less effort in learning Slovene and therefore make less progress. However, as the number of such students is relatively low, it is also possible that this is simply a consequence of their individual characteristics.
43As expected, students from both the IC and MT groups who wrote the experimental text in their first language scored better than those who wrote it in English. This may be due to their individual writing skills, as the teachers assessing the texts frequently pointed out non‑logical or non‑coherent writing, jumping between subjects, and lack of understanding of the text’s chronological order. Also, the sentence “I’ve always loved martial arts, which is why I used to train snowboarding every day as a kid” (in Slovene: “Od nekdaj sem oboževal borilne veščine, zato sem kot otrok vsak dan treniral deskanje na snegu”) does not make sense, regardless of the language it is written in. But we can assume that most students are more skilled at writing in their first language than in a foreign language such as English.
44Finally, let us compare the average grades of the control text alone with the way students typically write their homework (Figure 13). It is evident that two categories stand out: students who usually write their homework in English and translate it with MT have the lowest average score. This can be explained by the fact that these students may be the least motivated, investing the least energy in learning a new language and therefore making the least progress. On the other hand, students who write the assignments independently, without external assistance, achieve the highest average scores. This suggests that these students are highly motivated and dedicated. They put more effort into preparing the texts for the course and consequently into learning Slovene, and therefore have a better result. Of course, alternative explanations for these differences may exist; after all, the sample of students analysed is relatively small.
Figure 13. – Average grade according to the usual homework writing style.
45This research indicates that texts written using MT may not necessarily provide sufficient information about their author’s L2 proficiency. In my study, most students who wrote their experimental text in a language other than Slovene and then translated it into Slovene using MT got a lower grade for their control text, which they produced without any aids, than they did for their experimental text. Based on the analysis presented in this paper, it can be concluded that MT is a convenient shortcut for relatively well‑written texts, but it does not guarantee better knowledge of L2. My respondents who rely less on MT seem to be more successful at language acquisition. The question remains open as to whether students who frequently use MT have an unconscious uptake from using these tools and transfer this knowledge to their Slovene L2 production in a more controlled setting. The lower grades that the students in my analysis who relied more on MT received for their control text do not confirm this, but any generalisations on this matter would require separate, more focused research. The research presented here was conducted before the significant rise of artificial intelligence (AI) tools in 2022, which is assumed to produce even better written texts. However, in Year Plus classrooms, we have not observed significant changes that would necessitate alternative pedagogical or didactic approaches beyond those already employed to address the use of MT. Among these are the following:
-
Classroom tasks have been revised to include more guided writing which encourages autonomous production (for instance, the completion of sentences).
-
More attention has been given to the difficulty of tasks (tasks that exceed learners’ current proficiency level too much encourage the use of MT).
-
The amount of written homework has been reduced, and more controlled writing in the classroom environment has been introduced.
-
Students have been reminded that they should not take MT as a fully reliable tool and that they should post‑edit their translations.
46As over 95% of students use MT in some form or other during Slovene language courses, and since that obviously makes their language proficiency appear higher than it really is, their homework cannot be considered a reliable source for assessing their language proficiency. This confirms the pragmatically adopted decision of KOST corpus compilers to exclude any text that shows apparent signs of the use of MT, as it is impossible to determine the extent to which it was written using MT. This is always to some extent a subjective decision on the part of the corpus compilers, but all the professionals involved in the process have extensive experience with Slovene as a second language. They can therefore assume with a high level of certainty that a South Slavic beginner after the first month of learning Slovene is unlikely to produce an essay in Slovene which is free of linguistic errors and includes the correct use of some of the declensions that students traditionally have problems with, even at higher levels of learning—for example, the plural form of the instrumental case (Stritar Kučuk, 2022a).
47In recent years, as the KOST corpus itself has grown and the need to include more data become less urgent, we have systematically avoided all texts written in uncontrolled circumstances, and focused on texts written under teacher supervision—in the classroom or during an exam—thus ensuring that MT was not used. The analysis presented here confirms that our decision was sensible. If the KOST compilers decide in the future that machine-translated texts also present a relevant variety of learner language, they may include such texts, with appropriate meta‑tags. However, retrospective addition of such tags is not possible due to missing relevant data.
48As MT has become an important factor in the SLA process, its influence on learner language cannot be ignored by learner corpus compilers or language evaluators. On a final note, it is worth noting that of all the essays compiled during this research, only the control texts, written under controlled conditions, were included in the Slovene learner corpus KOST.