- 1 Our translation: « réduire les savoirs en langues à des connaissances linguistiques est un pas en a (...)
1Since the public roll-out of Generative Artificial Intelligence (GenAI) tools in late 2022, the field of education has experienced major transformation and triggered widespread concerns (Giannini, 2023). Yet this technology harks back to the 1960s when it already engendered considerable discussion in the field of education in general. It was then argued that AI had the potential to revolutionise personalised learning, intelligent tutoring systems, and automated assessment which can result in a more efficient and effective education. At the same time, there were worries that AI would lead to a dehumanised educational experience by negatively affecting creativity, critical thinking, and emotional intelligence of both teachers and learners (Djelti & Kouninef, 2022). At the time, Demaizière (1987) raised the many possibilities offered by AI tools but called for caution with their use in language teaching. Similarly, Boucher (1992) questioned the relationship between AI and language learning, echoed more recently by Huot (2022, p. 88) who spoke of a “disruptive revolution.” GenAI tools and the large language models (LLMs) on which they are based have evolved to an unprecedented level of performance, and are continuously improving (Tamkin et al., 2021). Today automatic language processing, which includes text translation, indexing, summarising, synthesising, mining and semantic content analysis, has never been so efficient. Students around the world are no doubt using these tools in both personal and institutional contexts (Babarti et al., 2024). In France, the French government, through its then-minister of Education, Gabriel Attal, press-released a 40-page working document in December 2023 titled Choc des savoirs : élever le niveau de notre école (Ministère de l’Éducation Nationale et de la Jeunesse, 2023). By implementing programs that rely on IA tools to enhance learners’ exposure to second languages, personalise their learning and prevent school dropout rates, France aims to become “the first country in the world to use AI tools to help learners progress at school” (p. 26). However, some researchers have contested the adequacy of the program because “reducing language skills to linguistic knowledge alone is a step backwards”1 (Miras et al., 2024, para. 11) and GenAI-based curricula need to be thoughtfully and cautiously included in a general educational system (Javaid et al., 2023).
- 2 Our translation: « Rédiger un devoir avec ChatGPT tout en le faisant passer pour le vôtre est en gé (...)
2In relation to restrictions by academic institutions, a teacher training department (MEEF) at the Université de Lorraine for instance has internally implemented a working paper aimed at English majors (Schmidt & Sweeney-Geslin, 2024). This 15-page guide explains GenAI/ChatGPT and provides advice on how to use it (with the right prompts), when (required by teachers), how to cite passages from ChatGPT (reference norms), and where to find resources about its use (open classroom, tutorials). On ethical issues, it clearly warns that “Writing homework with ChatGPT and pretending it is your own work is generally easy to detect and leaves you open to disciplinary proceedings, as in any case of fraud”2 (slide 11). Although some of the recommendations are debatable (e.g. the preference for printed resources and reliable websites, teachers knowing students’ exact language levels), it constitutes a typical example of how language teachers and departments view AI tools today. That said, many faculties do not have rules and regulations concerning the use of GenAI/ChatGPT by students when it comes to assignments, exams, and certifications. In the following section, we present some publications concerning the impact of GenAI/ChatGPT in the field of additional language (Lx) learning and teaching.
3Since its sudden appearance in the public arena in November 2022, ChatGPT has drawn the attention of scholars and non-scholars and led to countless publications on the issue, in scientific journals and platforms as well as in general public magazines, websites and social media. For instance, as of today’s date (13th August 2024), the basic search “ChatGPT” + “language learning” in Google Scholar yields 9,020 results just since 2023.3 Also, on The Conversation platform, a similar request returned hundreds of results dating from early 2023.4
4Within the field of language learning and teaching, much interest has been focused these last two years (since GPT-4) on the impact, implication, and future directions of GenAI tools. As a result, research has flourished, investigating various aspects of Lx learning including vocabulary (Ahmed et al., 2023), listening (Xing, 2023), and pronunciation (Fatima, 2024; Senowarsito & Ardini, 2023). Others have investigated the overall benefits and efficiency of AI tools for language learning in terms of perceptions (Guhan & Chandramohan, 2024), usability (Shaikh et al., 2023) and ethics (Auger et al., 2021). However, while theoretical and epistemological papers mostly focus on the applications of GenAI and ChatGPT in particular in the educational field (Kostka & Toncelli, 2023; Hatmanto & Sari, 2023; Huang & Li, 2023; Rüdian & Pinkwart, 2023), few explore empirical data, such as Lx learners’ productions.
5With the aim of investigating the impact of ChatGPT on ESL (English as a second language) learners’ linguistic proficiency, Guhan and Chandramohan (2024) collected quantitative and qualitative data through a questionnaire. The 359 respondents were asked to rate ChatGPT’s influence on their vocabulary acquisition, writing, speaking, motivation, and engagement. The results indicate that the majority thought ChatGPT had positively impacted their language learning by: expanding their knowledge of idioms (144), making learning more enjoyable (223), improving their pronunciation (183), increasing their confidence level (274) and motivation (194) among others. Although there was no direct measurement of impact on the participants’ actual performance (only perceptions), the study concluded that there is “a strong correlation between language acquisition and usage of ChatGPT in language classrooms.” (p. 81)
6Similarly, Hong (2023) discussed the opportunities and risks of using ChatGPT for teachers and learners alike, and pointed to the need for institutions to provide guidelines that are informed by research. The question of language assessment becomes crucial in the face of massive use of ChatGPT for academic assignments including homework and exams. Moqbel and Al-Kadi (2023) argue that traditional language assessment is no longer useful or valid to measure Lx learners’ output. The authors call for rethinking and readapting the way learners are assessed, orienting teaching “toward more advanced modes that correspond well to the new pedagogies of projects implementing informal and personalized learning.” (p. 81) Because ChatGPT produces human-like texts, Lx learners might end up becoming dependent, so the authors suggest giving “separate scores for each aspect or dimension of a task, such as grammar, vocabulary, etc.” (p. 75)
7In an international survey conducted just 10 weeks after the release of ChatGPT, Alm and Ohashi (2024) analysed knowledge and perceptions of it from 367 higher education language teachers in 48 countries/regions. It was found, among other things, that teachers were relatively aware of the issues, and were open to use ChatGPT mainly for course design and creating teaching materials. However, they expressed fears concerning students misusing the tool for classroom assignments, and stressed their own need for training. Kostka and Toncelli (2023) conducted qualitative research involving their students’ perceptions about using ChatGPT for classroom activities. Although the number of participants and details about the data collection are sketchy, the authors report positive results from students’ engagement and willingness to communicate.
8In a paper discussing epistemological issues, Ahmed et al. (2023) listed some potential ChatGPT applications for developing ESP (English for specific purposes) students’ languaging skills. When considered as a discussion forum or a personalised conversational companion, it can provide technical resources including answers, comments, explanations, and examples (e.g., the difference between a transistor and a diode, p. 46). However, the authors focus on the positive affordances of ChatGPT and did not consider giving any warning as to the veracity and sources of the answers provided: “The model can be trained to generate precise, accurate, and useful corrections and feedback for ESP students, providing immediate and individualised feedback on their language proficiency.” (p. 47) However, ChatGPT is now known to ‘hallucinate’, i.e. to generate and invent information that does not actually exist. That is why, supported by the Taiwanese government, Galmar (2022) offers training for Lx learners and teachers on the integration of GenAI tools in their classrooms (para. 12).
9On a similar note, Javaid et al. (2023) propose 27 ChatGPT applications that can be integrated in Lx education. Much enthusiasm has been generated about the benefit of the tool for teachers and students alike. For the former, it can save time that can be injected in other useful tasks and reduce their workload; for the latter, it can enhance their skills development, improve accuracy, reduce cognitive load, and provide feedback and comments tailored to their specific questions and needs. For instance, the authors claim that: “In order to assist students and improve their writing abilities, ChatGPT may check for grammatical and structural problems in their work and provide valuable comments.” (p. 2) But unlike Ahmed et al. (2023), this study has several reservations concerning the use of ChatGPT for educational purposes, such as ethical issues related to assignments (homework, essays, theses) and plagiarism (p. 10). The ethical problems raised are also discussed in many other papers relating to ChatGPT and Lx learning (see Auger et al., 2021; Mhlanga, 2023). Some authors think it is necessary to take ethical and legal issues into account (Collin & Marceau, 2021, para. 6), questioning “unfair, sexist, racist, and other objectionable comments” (Tighanimine, 2023, p. 35), and suggest that institutions define and regulate the use of GenAI tools within their boundaries (p. 37).
10This review highlights the possible influence of GenAI tools on different aspects of Lx learning and teaching, from the way they are perceived to their potential influence on language skills, their applications in the classroom and ethical perspectives. However, further research is required to explore learners’ actual uses of AI, especially on their language output. The current study adds to this body of knowledge by examining EFL (English as a foreign language) language learners’ written work in a formal assignment in a French university. After describing its context, the main objectives of this study are presented along with the research questions in the following section.
- 5 “UFR” is the label given to teaching departments or faculties (unite de formation et de recherche, (...)
11The UFR Lansad5 (Langues pour spécialistes d’autres disciplines) (languages for specialists of other disciplines) at the Université de Lorraine is tasked with providing language courses to students majoring in other disciplines. The department’s overall organisation for English is by semester, designed to allow students to progress in the target language, including a strong component of autonomous learning whereby students identify their own strengths and weaknesses, their preferred learning styles and strategies, and the resources available. The first semester of the undergraduate degree (S1) focuses on actively teaching the methods, skills and resources that go into learning languages. The following four semesters (S2 to S5) focus progressively on the four main language skills (listening, reading, writing, and speaking), while the final semester (S6) culminates in more autonomous language use including mediation. Students enrolled in the S6 English course must fulfil a series of tasks as part of their end-of-year assessment, working in small groups to decide a topic related to their studies to work on in English, as outlined in more detail in the next section.
12This course is open to students from a range of degrees in humanities and social sciences. The aim is for them to carry out a piece of research that will prepare them for the work they will have to do during their Master’s degree. It is an opportunity for them to learn topic-related vocabulary, improve their grammar and practise written and spoken skills, both receptively and productively. It also allows them to improve their group work skills, which will be crucial in their Master’s years and beyond. This course requires students to work autonomously in groups usually of three and choose a topic related to their studies that they will work on throughout the semester.
- 6 We prefer the term “assignment” rather than “dossier”, which has different connotations in English.
13The semester-long course is divided into two main parts: reading/writing and listening/speaking, with oral evaluations taking place over the last two weeks. For the first part (classes two to six), they are required to find and analyse three or four relevant written texts, at least one of which should be a scientific research article. They complete a template with information about their texts (source, date of publication, etc.), a short justification of their choices, plus a compilation of keywords and key expressions selected from those texts. Finally, they write a 300-word synthesis of their chosen texts (400 words for a group of four). The synthesis should be a coherent text linking themes from the different sources, not just a summary of their reading texts. They are given two group grades for this: one for the assignment6 as a whole and one specifically for the synthesis. The second part of the semester (classes seven to ten) follows a similar format but for listening and speaking, still in the same group and on the same topic. There is a second template to complete, but instead of a synthesis, they are required to write and act out a short sketch in English. An additional mark is attributed to self-directed learning, as students record their experiences in an individual online logbook they fill in throughout the semester. The focus of this study is on the first, written assignment i.e., short syntheses based on their reading of chosen documents.
14The basic course design has not changed since the advent of ChatGPT, and clearly there is potential for AI tools to contribute positively – or, conversely, represent a threat to the integrity of the course. Though there is no specific training for the students concerning the use of GenAI/ChatGPT as the UFR Lansad has no rules and regulations (unlike the MEEF case cited above), some teachers discuss the issues in their classes. Informal conversations with teachers show different approaches: some reject it outright while others endorse it almost wholesale, with most situated somewhere in between these poles. Our aim here is thus to examine how students are already spontaneously using ChatGPT and other AI in their marked assignments in and out of class.
15As we have seen, a large body of theoretical and epistemological papers have appeared on GenAI and ChatGPT. However, the few empirical studies to have investigated the impact of ChatGPT on students’ language skills have mainly relied on emic data from questionnaires, interviews and focus group discussions. Our study explores the direct impact of ChatGPT on student writing through the examination of linguistic aspects such as lexical frequency and range, parts of speech and participle clauses as a possible example of an AI-favoured grammatical construction. The study was conducted in the final semester of an undergraduate degree and was motivated by observations made during the assessment of the students’ assignments. It explores the impact that ChatGPT can have on these learners’ writing by comparing groups that use ChatGPT against those who do not.
16Given the context and the objectives, the current study is guided by three main research questions:
-
Which tools (especially GenAI) do the participants use to write their assignments, and what do they use them for?
-
What proportion of the groups use ChatGPT to write their assignments, especially the synthesis, as self-declared and as identified by an AI detection tool?
-
How does ChatGPT impact their writing, especially the final synthesis?
17The participants involved in this study included 753 students enrolled in humanities and social sciences at a French university who take a (usually compulsory) course in English as part of their degree requirements. They were in their early 20s at the time of data collection (mean age = 22 years, SD = 3.5) and come from different fields of study, mostly psychology, information and communication studies, or history (Figure 1).
Figure 1 – Disciplinary backgrounds of the students enrolled.
18Given the large number of students enrolled, seven teachers shared the course for a total of 20 classes. The data used for this research is collected from the written assignments submitted by the students via the course platform (Moodle) for all teachers, representing a more diverse and ecological dataset. The university Scientific Integrity commission was consulted for ethical approval.
19The corpus was compiled automatically to avoid time-consuming manual copying and pasting of individual texts in different formats: Microsoft Word, Open Office or PDF. For consistent manipulation, all formats were first converted to PDF, and subsequently into HTML/XML format using Pdftohtml (a Linux utility from the Poppler library), which gives the absolute positions (on each page) of the texts. A Python script retrieved these texts according to their position among the instructions in the template. For example, to retrieve the theme, we searched for what is between the texts “Theme and Coherence” and “Tell us about the tools.” This method has its limits, because some assignments were wrongly formatted (missing headings, wobbly pagination, etc.), and this particularly led to some automatic extraction errors. This was also the case for assignments where ligatures (“ti”, “fi”, “ff”, etc.) form a single symbol that is not included in the word (e.g. “position” appearing as “posi$n”).
20The raw corpus contained 245 assignments, of which 9 were duplicates and 46 others were returned as errors, therefore unusable without considerable manual work, resulting in a total of 190 that were successfully parsed and exploitable. Using a script coded in Python, we extracted, cleaned and sorted data from those assignments that were relevant to the research. The selection was based upon and corresponded to the different parts that constitute the template (see course description) with the following items.
-
The number of the class and the number of students in each group – one to four members (see Appendix A). A few students worked alone under a special arrangement called Régime Spécial d’Études, designed for those who cannot attend full time, usually where they have a job alongside their studies.
-
A list of all the tools they used to complete the assignment (dictionaries, DeepL, ChatGPT, etc.), required in the template along with the next two points.
-
Specific comments about how they used each tool and to what extent it helped them.
-
More general comments about the overall use of the tools.
-
The topic of their work, usually formulated as a title (e.g. “How effective is gamification of learning, and what are the limitations of this approach?”).
-
The theme of the assignment, explaining specifically what the topic was about.
-
The final written synthesis based on three or four written texts for a total of 300 words (for one, two or three students) or 400 words (groups of four), plus or minus 10%.
21The exploratory nature of this research led to the use of different analytical methods and software that we considered complementary, necessitating several different tools at different stages, from the compilation of the corpus to its analysis. As previously mentioned, Python was used once all the assignments were downloaded from the course platform to count the number of assignments in which the use of ChatGPT was mentioned, followed by descriptive statistics such as means, percentages, and distributions in graphs. It was also useful for exporting the data in different formats including CSV and for exporting the syntheses into .txt.
22Early on, we were surprised by the high occurrence of present participle clauses7 in the form of the sentence structure “comma + Verb + ing” (henceforth “cVing”). Many such sentences are long and complex and struck us as being unusual for our students, prompting us to explore it in more detail. For example: “The emphasis should shift towards the lives of victims, survivors, and those who thwart such acts, discouraging the glorification of offenders” (#57). Further examples are provided in Appendix B. To explore this, we compared cVing in assignments that declared using ChatGPT against those that did not, covering the three main sections: comments (general and specific), themes, and the syntheses themselves. It is worth noting that other items such as adverbs are sometimes found between the comma and the Ving, but this does not alter the sentence structure. These items include “and”, “or”, “thereby”, “thus”, “particularly”, “while”, “ultimately”, “potentially”, “especially”, “but”, “adversely.” Erroneous items were excluded: “sing”, “sibling”, “morning”, “shooting”, “ling”, “during”, “depending”, “bing”, “king”, “thanksgiving”, “marketing”, “according”, “marving”, “clothing”, “everything”, “cyberbullying.”
23The CSV file was opened in MS Excel to tally the tools students used and what they used them for, while the syntheses in .txt were uploaded into AntConc8 for the corpus analysis, supplemented by TagAnt9 and LexTutor VocabProfile.10 Then, GPT 3.5-turbo and 4.0 were used to “improve” the texts which did not contain the cVing patterns. Finally, all the syntheses were run through GPTzero which is an AI detector.
24Participants declared using many different tools to complete their written assignments (Table 1). Most of the groups (n = 180 or 94.7%) used more than one tool (M = 2.7) to work on their syntheses and assignments. Overall, they tended to favour online dictionaries (31.5%) such as Cambridge (n = 22) or Pons (n = 8) as well as Word Reference or Reverso, but also translation tools such as and Google Translate or DeepL.
Table 1 – Digital tools noted by the participants to complete their written assignments.
|
Online dictionaries
|
Word Reference
|
Reverso
|
Google Translate
|
DeepL
|
Linguee
|
Corpus-based
|
AI
|
Others
|
N
|
60
|
60
|
34
|
19
|
116
|
58
|
4
|
78
|
83
|
%
|
31.5
|
31.5
|
17.9
|
10
|
61.1
|
30.5
|
2.1
|
41.1
|
43.7
|
- 11 Both Linguee and Reverso feature parallel corpora, but can also be used as bilingual dictionaries.
25DeepL seems to have largely supplanted Google Translate since data collected by Nassau et al. (2022) at the same university; they found Google Translate to be the most favoured translation tool (65.2%), with DeepL at 34.3%. Corpus-based tools do not seem to be particularly popular (only 2.1%), with the exception of tools based on parallel corpora such as Linguee (30.5%) and Reverso (17.9%).11 Other tools (43.7%) include common word processors and wikis (Google Docs, Microsoft Word) and websites (AnglaisFacile.com, bab.la, LanguageTool.org, Scribbr.fr, Translatium.com), a plagiarism detector (not specified), websites for finding papers (databases offered by the university as Cairn, alternative platforms like Sci-Hub, etc.) and friends or colleagues who agreed to help in the writing or reviewing of the summary. Most interesting for present purposes are the 41.1% of the groups who declared using AI-based tools other than translators. These include ChatGPT at chat.openai.com (n = 65), Grammarly.com (n = 5), DeepLWrite (n = 2), PopAI.pro (n = 2), and a few others once each: Resoomer.com, QuillBot.com and Gemini.Google.com.
26One part of the written assignments required them to reflect on their use of digital tools. Content analysis of this part of the dataset revealed seven main uses (Figure 2). Disturbingly, the most frequent was actually to write text, which for our purposes includes summarising for the synthesis. This is followed by checking language use, where ideally we would like more information before deciding if this is appropriate behaviour to be encouraged, or goes against the objectives of the course. While students often asked the AI tool (a) to rephrase written text from their chosen documents in order (b) to better understand it, not all of them made a connection between the two in their commentaries; we therefore tagged them separately. Other uses (18.2%) included asking the AI tool to find information or documents for their task, finding supplementary material that went beyond the purposes of the task, finding keywords from the text they were studying (a required component of the assignment but not covered here), and polishing the final product.
Figure 2 – Self-reported uses of AI (number of occurrences).
27To tally the number of assignments that declared ChatGPT use (henceforth “GPT groups”) and those who did not (“non-GPT groups”), the items “chat” and/or “GPT” were parsed within the “name of tools” column. Out of the 190 assignments, 66 groups (34.74%) explicitly declared using ChatGPT, against 124 (65.26%) who did not. Narrowing down the search to focus only on assignments that claimed to use ChatGPT to write the synthesis itself (excluding the themes and comments sections), we found 24.7% (47 vs 143), about a quarter of the total number of groups (Figure 2). We also manually excluded two in which ChatGPT was mentioned to indicate that the tool was not used.
28The same two corpora were subjected to various analyses using AntConc and other corpus tools to identify differences potentially resulting from AI use. Initial results were unconvincing, leading us to consider that the students might have inaccurately represented their use of such tools – whether deliberately or through uncertainty of appropriate ethical uses. Rather than simply taking their word for it, all syntheses were thus loaded individually into the AI detector GPTzero.com, used by many journals and organisations including the American Federation of teachers. This tool relies on features such as ‘burstiness’, since humans tend to write somewhat more chaotically and less smoothly than AI. Like GenAI, GPTzero relies on LLMs and its output is just as much a black box as ChatGPT, for example, and we should not trust it blindly – especially in accusing students of ‘cheating’ (Heumann et al., 2023). Reassuringly, some of the few independent academic studies of the tool so far, albeit in other fields, have shown far more false negatives than false positives (e.g. Habibzadeh, 2023), meaning that it is relatively conservative and prefers to (mis)identify AI-generated text as human rather than vice versa. But for present purposes it may provide an additional indicator of the impact of AI tools. Among students who declared using AI to produce their synthesis, the mean quantity of text estimated to be AI-generated was 29.5% (SD = 37.7); for the others, it was 18.7% (SD = 32.3). The difference partly reflects students’ declarations, but clearly there are many in both groups who are relying heavily on AI to produce the text of the synthesis, in contravention of multiple warnings and recommendations for good usage, not to mention common sense. The breakdown is provided in figure 3 for each group (GPT and non-GPT). Between 8.4% and 8.5% of text in each group was estimated at 0%, with about two thirds of all syntheses (66.3%) estimated to have less than 10% of AI-generated text; however, the non-GPT group featured more heavily here (71.3% vs 51.1%). At the other end of the scale, 17.0% of the GPT and 10.5% of the non-GPT texts were estimated to be at least 90% AI-generated, including 8.5% and 7.0% respectively scoring 100%. Previous caveats notwithstanding, the high rate of probable AI-produced text in some syntheses is cause for concern.
Figure 3 – Estimated percentage of AI-generated text.
29Drawing on our initial assumption that GenAI tools impact student writing by “injecting” a high number of cVings in their assignments, we ran an in-depth analysis focusing on ChatGPT only. Although cVing structures were present in both GPT and non-GPT groups (560 occurrences in all), first results show two different distributions: there were assignments containing no cVing structures in both groups (6 in GPT and 42 in non-GPT), but the GPT groups’ assignments contained a higher proportion of cVing (274), with some outliers using up to 19 cVing patterns per 400 max words. This finding (Figure 4) means that the 34.7% of the GPT group employed almost half (49.0%) of the overall use of cVing compared to the 65.3% of the assignments that claimed not to use ChatGPT (51%).
Figure 4 – Number of cVing patterns in both GPT and non-GPT groups (assignments).
30When only the syntheses are considered, the results of cVing structures show a significant decrease for the GPT group, but the distribution is similar for the non-GPT group as for the assignments as a whole (Figure 5 and the first two boxes of Figure 6).
Figure 5 – Number of cVing patterns in both GPT and non-GPT groups (syntheses).
31On the hypothesis that cVing structures are characteristic of ChatGPT, we ran a quick verification by uploading the syntheses with no cVing structures to ChatGPT and asking it to “improve” them. Using both GPT 3.5-turbo and 4.0, the prompt “correct the grammar and improve the style of the following text” yielded one to seven cVing sentence patterns per text (400 words maximum). Thus, the 48 initial syntheses now contain 179 cVing patterns when revised by ChatGPT. This can be an indication that ChatGPT does indeed favour the cVing structure.
32A more systematic observation of the phenomenon involved comparison of the distributions of the cVing structure for the three groups: GPT and non-GPT, along with AI-improved texts on the right in figure 6. The results indicate a close similarity between the original GPT and the AI-improved syntheses. The even distribution can be explained by the automatic generation of the texts by ChatGPT. Also, there are no outliers as there were in GPT and non-GPT groups. One interpretation is that, while some groups have no doubt done more or less the same manipulations, others may have written their synthesis by assembling little bits of ChatGPT-generated sentences containing cVing patterns.
Figure 6 – Distribution of cVing patterns in both GPT and non-GPT groups (syntheses).
33The attempt to see whether there is a difference between the various forms of cVing occurrences within the three categories did not yield any salient results, other than some frequent verbs common to the GPT groups and the revised syntheses ("shedding light”, “underscoring”, “highlighting”, “emphasizing”, “leading”) but which are absent in the non-GPT groups, who tend rather to use simple verbs like “making”, “including”, “offering”, “having”, “allowing”). It therefore seems that these expressions are not naturally employed by the students – at least, not without the help of ChatGPT and the like.
34The recurrent cVing pattern introduced by ChatGPT is sometimes used to merge two sentences into one long one, so it might be assumed that ChatGPT produces texts with longer sentences. There were averages of 17.00 (SD = 4.06), 17.74 (SD = 2.64), and 18.79 (SD = 2.62) words respectively for GPT, non-GPT, and AI-revised syntheses, showing small differences that were not statistically significant. Thus this cannot be taken as a valid comparison indicator for sentence length between texts produced by ChatGPT and those written by the students.
35We then compared two corpora of texts which GPTzero estimated at the extreme levels: 21 syntheses at 98-100% and 56 at 0-1% (Table 2). While there can certainly be errors in the estimates, it is more likely that the extremes would outweigh slight errors, though it does reduce the size of the corpora available for comparison.
Table 2 – GPTzero polar opposite corpora.
Corpus
|
Files
|
Tokens
|
Mean tokens
|
Types
|
Lemmas
|
GPTzero 0-1%
|
56
|
19,065
|
340.45
|
3,474
|
2,796
|
GPTzero 98-100%
|
21
|
6,686
|
318.38
|
1,887
|
1,522
|
Total
|
77
|
25751
|
334.43
|
4,304
|
3,394
|
36Comparison now provides far more convincing results than between syntheses on the basis of students’ self-reported use or non-use of AI. First, table 3 shows that the syntheses estimated to be human-produced are heavy on very high-frequency lexical items: 71.5% of all words are among the 1000 most frequent words of English, compared to just 59.8% in the estimated AI-generated texts. The human texts show some relatively rare items, which we might attribute to lack of mastery, and are heavy on off-list items which probably come from the documents they read (including proper names), while the AI syntheses may avoid these.
Table 3 – Lexical frequency bands.
Vocab coverage, GPTzero 0-1% AI
|
|
|
Vocab coverage, GPTzero 98-100% AI
|
Level
|
Tokens
|
Percent
|
Cumul%
|
|
|
Level
|
Tokens
|
Percent
|
Cumul%
|
k-01
|
13,780
|
71.50
|
71.50
|
|
|
k-01
|
4,024
|
59.80
|
59.80
|
k-02
|
2,318
|
12.03
|
83.52
|
|
|
k-02
|
1,013
|
15.05
|
74.86
|
k-03
|
1,805
|
9.37
|
92.89
|
|
|
k-03
|
1,054
|
15.66
|
90.52
|
k-04
|
427
|
2.22
|
95.10
|
|
|
k-04
|
243
|
3.61
|
94.13
|
k-05
|
123
|
0.64
|
95.74
|
|
|
k-05
|
68
|
1.01
|
95.14
|
k-06
|
116
|
0.60
|
96.34
|
|
|
k-06
|
103
|
1.53
|
96.67
|
k-07
|
51
|
0.27
|
96.61
|
|
|
k-07
|
36
|
0.54
|
97.21
|
k-08
|
39
|
0.20
|
96.81
|
|
|
k-08
|
51
|
0.76
|
97.97
|
k-09
|
43
|
0.22
|
97.03
|
|
|
k-09
|
10
|
0.15
|
98.11
|
k-10
|
25
|
0.13
|
97.16
|
|
|
k-10
|
9
|
0.13
|
98.25
|
k-11
|
14
|
0.07
|
97.24
|
|
|
k-11
|
6
|
0.09
|
98.34
|
k-12
|
7
|
0.04
|
97.27
|
|
|
k-12
|
8
|
0.12
|
98.46
|
k-13
|
5
|
0.03
|
97.30
|
|
|
k-13
|
3
|
0.05
|
98.50
|
k-14
|
6
|
0.03
|
97.33
|
|
|
k-14
|
1
|
0.02
|
98.52
|
k-15
|
5
|
0.03
|
97.35
|
|
|
k-15
|
3
|
0.05
|
98.56
|
k-16
|
2
|
0.01
|
97.36
|
|
|
k-16
|
1
|
0.02
|
98.58
|
k-17
|
4
|
0.02
|
97.39
|
|
|
k-17
|
|
|
|
k-18
|
1
|
0.01
|
97.39
|
|
|
k-18
|
|
|
|
k-19
|
1
|
0.01
|
97.40
|
|
|
k-19
|
|
|
|
k-20
|
4
|
0.02
|
97.42
|
|
|
k-20
|
|
|
|
k-21
|
|
|
|
|
|
k-21
|
2
|
0.03
|
98.61
|
k-22
|
|
|
|
|
|
k-22
|
|
|
|
k-23
|
|
|
|
|
|
k-23
|
|
|
|
k-24
|
1
|
0.01
|
97.42
|
|
|
k-24
|
|
|
|
k-25
|
1
|
0.01
|
97.43
|
|
|
k-25
|
1
|
0.02
|
98.62
|
off-list
|
479
|
2.485
|
99.911
|
|
|
off-list
|
69
|
1.03
|
99.65
|
37Note: k-01 represents the 1,000 most frequent lexical items in the BNC and Coca, with each successive band adding a further 1,000 words; from https://www.lextutor.ca/vp/comp
- 12 Owing to a peculiarity of VocabProfiler, “online” and “healthcare” are automatically broken down in (...)
38Turning to the key lemmas in table 4 (i.e. those individual items that are statistically more frequent in one corpus compared to the other), the top key lemmas in the 0-1% corpus are all in the top 1,000 (K1) most frequent words of English in the BNC and Coca corpora (“be”, “we”, “see”, etc.), while those in syntheses most likely to have been AI-generated cover more bands: “and”, “shape”, “on” and “line” are all K1; “include”, “approach” and “criminal” are K2; “intervention”, “crucial”, “strategy” and “emphasize” are K3. All 26 key lemmas in the 0-1% corpus are in the K1-K3 band, while 22% of items in the 98-100% corpus are beyond that level.12 This would seem to suggest that learner-produced text is relatively heavy on common items, while AI generates lexically richer text.
Table 4 – Top 10 key lemmas; min. range = 3.
39We also looked at the most frequent parts of speech, following tagging with TagAnt using the Penn Treebank13 list. From table 5, the human-generated texts are heavy on verbs in the past tense (vbd), prepositions and adverbs, while the AI-generated ones have significantly more present participles, adjectives, plural and singular nouns and coordinating conjunctions (though it should be noted that all of these occurred in every file in both corpora).
Table 5 – Key parts of speech; min range = 5.
40The grammatical class most frequent preceding present participles in the AI-generated texts is preposition (88 of the 309 occurrences, or 28.5%), e.g. “crucial for reducing”; it should be noted that a classic problem for French learners is using an infinitive after a preposition (e.g. *”for to reduce”, *”before to understand”). More frequently still, present participles are preceded by punctuation (115 occurrences, or 37.2%). In 20.0% of cases (23) the participle is in sentence-initial position (i.e. after a full stop or similar), but the vast majority of preceding punctuation marks are commas – 92 occurrences, or 80.0%. Pursuing this, 83 of these are preceded by a noun (47 singular/mass nouns, 34 plural and 2 proper nouns); the only other parts of speech are 3 verbs, 2 adjectives, 2 adverbs, 1 personal pronoun and 1 particle.
41We now return to our research questions, beginning with RQ1: Which tools (especially GenAI) do the participants use to write their assignments, and what do they use them for? As outlined earlier, the primary use (writing texts or producing text summaries for reuse) was in fact an intrinsic part of the assignment. Adding self-reported use and use detected by GPTzero, it appears that a sizeable number of the respondents used AI to work on their behalf. However, since we were unable to conduct interviews, this remains a supposition at this stage.
42A quarter of the student population declared having used AI for purposes of revision and reviewing: “to improve our sentences” (#59); “in case we have poorly formulated sentences by having made grammatical or spelling mistakes” (#70); “to correct any errors or see if all our sentences were well written and understandable” (#89). This practice seems to have provided the participants with some measure of confidence, making the process of submitting the assignment less taxing than it could have been. AI appears then to be “reassuring” (#102) and may be useful for students who suffer from crippling linguistic anxiety (Fang & Tang, 2021) or insecurities (Langbach, 2020).
43This study also finds examples of poor use of digital tools and generative artificial intelligence. The following example is interesting: “We used ChatGPT so that he can explain certain terms that we didn’t understand in our articles. We used English dictionaries such as Word Reference and Cambridge Dictionary to translate some words and to help us define others” (#53). Using AI to understand terms when a bilingual dictionary would suffice does seem to be rather inefficient. Additionally, using a monolingual dictionary or translation tools for translating individual words might not be the best approach. Other participants caution against using AI blindly: “Sometimes the bot doesn’t understand the structure of your sentence, so do not blindly accept its corrections” (#118). Some feel the need to justify using AI rather than merely declaring it.
We only resort to using ChatGPT when we find ourselves stuck for ideas on projects or essays. We’re cautious not to rely too heavily on the tool, as it’s possible for some individuals to misuse its capabilities and allow it to do all the work. We believe in avoiding dependence on ChatGPT to ensure genuine learning experiences. Because we believe that human are capable to go pass beyond any intellectual limit. (#163)
44At the other end of the spectrum, we find students condemning any use of AI, considering themselves to be masters of their subject: “We haven’t used ChatGPT at all because we are studying history, and we already know the subject well enough. We considered that the documents provided us with sufficient information and that ChatGPT is a tool still in development and not very reliable” (#46). This finding seems to be representative of most of the student population as 65.3% do not consult any AI tools, despite teachers having given them the liberty to do so.
45RQ2: What proportion of the groups use ChatGPT to write their assignments, especially the synthesis, as self-declared and as identified by an AI detection tool? Of the 190 submissions, 66 (34.7%) overtly declared using ChatGPT, dropping to 47 (24.7%) using it to help write the synthesis itself. This does not take into account the excluded assignments, other AI tools (PopAI, Resoomer, QuillBot, Gemini), or the number of groups that may have used AI without admitting it. Adding those syntheses identified via an AI detector tool, it seems that large numbers of students are indeed using AI, and the chances are that this will increase in the near future. When these percentages are extrapolated to other universities in France, one can assume that at least one in two students is using ChatGPT and other GenAI tools for their academic work. This not only corroborates previous studies on the use of AI tools (Guhan & Chandramohan, 2024; Alm & Ohashi, 2024), but also calls for more research concerning the risk of “dependency” as raised by Moqbel and Al-Kadi (2023, p. 75): “ChatGPT […] encourages students to depend on ChatGPT to do their writing tasks, which signals a cautionary note to teachers and educators. Teachers ask students to perform in-class pen-and-paper writing tasks to prevent learners’ blind dependence on ChatGPT.” If this situation continues, and it seems likely to, then we can expect nearly the whole student population to engage in such practices. The need to establish clear regulations by institutions is therefore capital and this should come with training sessions for both teachers and students (Galmar, 2022).
46RQ3: How does ChatGPT impact their writing, especially the final synthesis? As far as the impact of GenAI/ChatGPT on the participants’ written production is concerned, the results indicate that the GPT groups used present participle clauses about twice as frequently, accounting for 49% of the 560 occurrences even though they represented only a quarter of the sample. Our tests found that ChatGPT is fond of the cVing construction, which students heavily draw on, consciously or not. What is more, the experiment carried out by ChatGPT (improving the 48 syntheses that did not originally contained cVing pattern) confirms our observations. This finding is interesting because in the process of “improving” these syntheses, ChatGPT injected 179 cVing patterns, comparable to the GPT groups’ writing. We attribute this to the tool rewriting texts by systematically combining two or more sentences using the cVing pattern as a welding element.
47We should note however that the non-GPT texts also contain a certain degree of cVing, which can be explained in two ways. First, it could be that participants really did not use any GenAI tool, and that their cVing structures constitute a naturally occurring construction within their language repertoire. This seems less likely than a second hypothesis: that students did not directly use ChatGPT, but did use other GenAI tools such as DeepL, which use similar algorithms to translate texts from French to English. Important here is the similarity in the nature of verbs employed by the GPT and the “improved” texts (“shedding light”, “underscoring”, “highlighting”, “emphasizing”, “leading”, etc.), as compared to the cVing structures within the non-GPT groups (“making”, “including”, “offering”, “having”, “allowing”, etc.).
48Running the texts through GPTzero enabled us to redistribute them, from self-reported GPT vs non-GPT to two other categories: very low and very high estimates of AI-generated text. The process found 29.5% of the GPT syntheses and 18.7% of the non-GPT syntheses had been substantially AI-generated. Also, the corpus analyses (lexical frequency bands, lemmas, parts of speech) suggest that learner-produced text is relatively weighty on common items such as verbs in the past tense, prepositions and adverbs, while AI generates lexically richer text, with a high number of present participles, adjectives, plural and singular nouns and coordinating conjunctions. The finding concerning the analyses of self-declared data does correspond to the corpus analyses in terms of the patterns that might have been induced by GenAI/ChatGPT. This also perfectly corroborates the distribution of cVing pattern in the sense that both ChatGPT and non-ChatGPT groups are found heavily using these structures. Given the likely uncertainty about what constitute ethical uses of AI, it would be imprudent to simply take students’ word for it, despite the multiple warnings and recommendations for good usage.
49This study analysed a collection of 190 third-year undergraduate assignments in English, to explore if and how they used ChatGPT and other AI tools, and what impact that had on their writing. Results show that the students use a variety of tools, most of them digital (including AI in at least one third of cases), to summarise and rephrase, review their syntax and grammar, find inspiration and ideas or understand texts by translating them. Also, corpus analysis indicates that there is actually a difference (use of present participle clauses, lexical sets, parts of speech, etc.) between assignments that rely heavily on GenAI tools and those that do not. An automatic detection tool suggests that not all students are happy to admit using AI, whether deliberately or through ignorance or uncertainly of ethical issues. Further research is needed to grasp the impact of GenAI tools on students’ Lx writing, with the aim of exploring other grammatical aspects including past and perfect participle clauses. The use of the present form (cVing) in our corpus could be compared to another similar corpus of written assignments collected prior to the release of ChatGPT. Individual use of GenAI could be explored through elicitation interviews or longitudinal data collection to better understand how learners use these tools.
50How frequently do learners use these tools? Why? And how do they know that what these tools generate is correct? How do they re-use the texts that are generated? Does the content help them to improve their language skills? What are the short, medium, and long-term impacts on their language development? What are the challenges for teachers? On that note, and as research has shown (Kostka & Toncelli, 2023; Hatmanto & Sari, 2023; Huang & Li, 2023), applications, policies and ethical issues relating to the use of GenAI tools are needed in the field of language teaching and beyond. It is also important to set up training courses for language teachers (Galmar, 2022), who will have to accept the change and rethink their role as mediators in terms of the nature of courses, activities, and assessment. Instead of wasting time correcting AI-generated texts or running them through AI-detection software, teachers could focus on meta- and epilinguistic aspects that enable them to work on skills related to critical thinking, argumentation, and reasoning through real-life sociolinguistic tasks (Ollivier et al., 2023). While there is no need to revive table-top writing and forced spontaneous speaking, the results of research such as this can facilitate the sociolinguistic aspects to be taken into account during assessments.