1Generative artificial intelligence models (GenAI) have entered the educational world breaking the door open, disrupting the class, and inviting teachers to rethink their methods, and learners to find or imagine new task strategies. They offer possibilities that can be timesaving for teachers (e.g., by generating questions from a text), or that can facilitate learning for students (e.g., by rephrasing an explanation). These tools are based on machine learning, deep learning and Large Language Models (LLM) capable of human-like text generation. But are they as intuitive as non-AI experts may imagine them to be? How will our students proceed with these GenAIs? They are already using Artificial Intelligence (AI) tools, but the scope of GenAIs is much wider in form and in content than most AIs they are used to. For this research, it was decided to investigate how students would use OpenAI's chatbot ChatGPT-3.5 if left the possibility to do so but without guidance. The chosen assignment is a “writing in the style of” imitation task as, up to today, this assignment has been done unmonitored at home and students can use the tools they wish. The LLMs will be or are part of these tools, and ChatGPT could seem like a perfect assistant for this kind of imitation task. It will allow the study of students' capacity to transfer their disciplinary skills in this new context and draw up a tentative picture of EFL (English as a foreign language) learners' GenAI uses.
2The debate over the use of AI in art divides artists and critics. Some seem to claim the use of AI corrupts art and the generated work cannot be considered as artistic: “Art is a uniquely human endeavour, using perspectives from philosophies of art and aesthetics” (Jiang et al., 2023, p. 363). However, when delving into the debate, one may realise that it is not only the presence of AI in art that is at stake, but among the multiple artistic and ethical questions, the conditions of its use. For Jiang et al. (2023), image generators might be used if they do not involve pillaging existing artistic projects, and do not supersede human-made works.
3The question then is how much original matter does the artist provide in the input? To what extent does he or she intervene on the output? To which degree is the human in-the-loop or out-of-the-loop? This stake of human in or out-of-the-loop raises many questions, especially in the field of AI and accountability (Goldenfein, 2019), and it needs to be broached when dealing with AI and art. Firstly, we humans appear to have a strong bias against AI art (Millet et al. 2023). Secondly, although Hitsuwari et al. (2023) tested discrimination on haiku and concluded that humans did successfully discriminate human made from machine made, other studies had previously concluded that we humans were unable to tell the difference between artwork made by a human and artwork made by an AI with a human out-of-the-loop process (Kobïs & Mossink, 2021). Consequently, human intervention is needed to improve the output, or to select the best one, if one wishes for a successful collaboration with the AI. For Hitsuwari et al. (2023), collaboration between the machine and humans delivered results considered as more beautiful than mere AI poetry.
4If we admit that humans need to remain in the loop to maintain authenticity in art, both in the output and in the input, there is nonetheless the challenge of making the LLM produce the wanted piece of art. Is the LLM capable of generating any piece of art?
5It seems that the less coded the type of art is, the more difficult it becomes for the LLM to produce what is demanded. This point was raised by Cheng et al. (2018), who questioned the ability of AIs to generate modern Chinese poems, in which creativity and imagination prevailed over the strict rules of ancient Chinese poetry. However, Badura et al. (2022) found no evidence that AI generated Shakespearean sonnets were more likely to be considered human made than AI generated free verse in the style of Walt Whitman. Shakespearean sonnets having strong and very formal poetic rules such as length of line, cadence, and rhyme patterns as opposed to Whitman’s poetry, their results contradict Cheng et al. (2018)’s. Nevertheless, it should be added that participants in Badura et al. (2022)’s study were not poetry experts and the samples were very short.
6In higher education, the aim of literature classes is to introduce works and authors believed to be fundamental in the history of literature, but also to develop cultural insights and tackle artistic perspectives on specific times or periods, so as to highlight their articulations and specificities. It allows learners to develop skills in stylistics and stylistic analysis, and is used as ‘a resource to develop linguistic competence’ (Zaro Vera, 1991, p. 169). The study of prose and poetry also allows learners to analyse the complexity of language in its symbolic content, and the connotations carried by a word, the variations in interpretations according to context, or the reader’s personal experience. These disciplinary skills are developed and assessed thanks to traditional assignments, such as comparing and contrasting texts, essays, text commentaries or imitation of an author. They are also used to develop creative skills.
7One of the traditional assignments used in the literature class is ‘Writing in the style of’. For this activity, students are meant to reproduce a piece of writing reflecting the style of a particular author. When presenting their work, they also need to justify their choice of theme, structure, cadence and imagery. According to Uwe Geist (2005, p. 172), ‘Imitation is closely connected with observation and analysis’ and students are taught to ‘supplement the personal-intuitive attitude with a technical-problem-solving one’. Imitation requires the imitator to thoroughly analyze the material to be imitated, gaining a deeper understanding of its function and style.
8Style is understood as the particular element that bridges form and content, as it ‘operates from the level of small mechanical units, like diction and punctuation, through the broadest conceptual levels, like content’ (Stodola, 2003, p. 59). It entails ‘variations in the presentation of this information [the content] that serve to alter its ’aesthetic quality’ or the reader’s emotional response’ (Abrams & Harpman, 2012, p. 386). For Riffaterre (1959, p. 155), it is ‘an emphasis (expressive, affective or aesthetic) added to the information conveyed by the linguistic structure, without alteration of meaning’. It may be understood as the specific use of literary means, vocabulary or syntax adopted by an author to express an idea. It also refers to ‘the way in which language is used in a given context, by a given person, for a given purpose, and so on’ (Leech & Short, 1981, p. 10). Style is not then dependent solely on the author, but also on context and other such mutable elements. Although style is far from monolithic both in its definition and in its expression, its identification constitutes a most relevant entry point into an author’s work, for the very fact that in spite of its eluding definition, it remains a writer’s signature (Compagnon, 1997).
9‘Writing in the style of’ may be seen as too constraining or too formal, yet it involves an accurate understanding of the original work through detecting and analysing elements of style before being able to imitate the original work. It allows students to be creative, but also to use their cultural, stylistic, and linguistic competence. ‘Writing in the style of’ is a task that can only come towards the end of a course, as it demands sharp mastery of devices, vocabulary, syntax, and stylistics components, but also a thorough knowledge of the author’s work and the capacity to implement a critical strategy.
10It is very likely that this type of assignment will undergo changes with the introduction of GenAIs as tools for our students, but will it make it useless? It seems that it may shift the focus, from reproducing the work itself to the critical approach students may have towards this imitation. Assuming that our students will test ChatGPT or other LLMs for this personal creation, we questioned their ability to use this GenAI to the best of their purpose. Our aim is to check their mastering of this new tool, but also to investigate possible new activities integrating LLMs to improve their understanding of literature.
11Conversing with a chatbot as a customer might be the most frequent first interaction one has had up to today. Studies have shown that anthropomorphic chatbots have a clear impact on the emotions of the people interacting (Crolic et al., 2022), especially because they align with the human interacting as if the conversation was natural. However, studies have also shown that humans reverted to clear commands under nominal clauses when confronted with bot mistakes (Spillner & Wenig, 2021). Therefore, the form of the interactions changes, sentences become shorter, the vocabulary used is not as rich as in human conversation (Hill et al., 2015), and there is not much attention paid to spelling or grammar (Park et al., 2018). We agree that GenAIs are not simple chatbots but, nevertheless, previous interactions students have had may influence the way in which they will consider prompting, prompting being ‘a natural language instruction that tells a large language model (LLM) to perform a task’ (Learn Microsoft, 2024). Consequently, prompting aims at initiating and shaping the output produced by the AI, it gives context, and details the expected output (White et al., 2023). Prompt engineering has thus become a new research area in the field of Human-Computer-Interaction (Korzyński et al., 2023).
12Users may have different types of conversations with GenAIs, depending on the format of the desired output (text or image). In the matter of texts, users may request an answer to a question, a summary, a classification, or content generation (Korzyński et al., 2023); their prompt needs to shape this output in form and content, and there are different ways of giving instructions to steer outputs, from keywords, brief search queries, to longer expanding conversations integrating numerous prompts and query refinements (Budiu et al., 2023a). The span of possibilities in the output intended is vast and, even if we limit the intention of the user to the area of writing fiction, the uses which are made vary very much from one user to another. Depending on the person’s expectations on the capacity of the AI, they will use it as ‘an active idea generator, a co-writer, or a writing assistant’ (Yang et al., 2022, p. 63). In all of these options, the GenAI comes in to support, if not emulate, human creativity. However, this use may also divert the writer’s attention from his or her own creative activity onto curiosity about what the AI can do or trying to ‘school’ the AI (Ghajarghar et al., 2022). The novelty of the tool, natural curiosity, and the speed of response might quickly take the user away from their first intention, their first prompt.
13To avoid wandering in prompting, a clear plan must be laid out with the students, prior to the interaction, so that they can make informed decisions during their conversation (Woo et al., 2023). However, it needs to be reminded that our purpose in this study is to evaluate students’ ability to interact with a GenAI uninformed, and to investigate their prompts.
14According to Cambridge dictionary online (2024), a prompt is defined as
(computer)
A sign on a computer screen that shows the computer is ready to receive your instructions
An instruction given to an artificial intelligence (=a computer system or machine that has some of the qualities that a human brain has, such as the ability to interpret and produce language in a way that seems human, recognize or create images, solve problems, and learn from data supplied to it) by a human using natural language rather than computer language (…)
(actor’s help)
Words that are spoken to an actor who has forgotten what he or she is going to say during the performance of a play
A person whose job is to help actors, during a performance, to remember words that they have forgotten.
15The term stems from the French prompt (quick, swift), which in turn comes from the Latin promptus (visible, evident, brought to light). Used as a verb, it is a synonym of ‘to cause’, ‘to make something happen’ (Cambridge Dictionary online, 2024).
16In the field of Second Language Acquisition (SLA), the use of prompts may recall behaviourism, structuring processes and drills. Prompts are closely associated with the structural drills which developed with the audio-lingual/army method, and the structuro-global audio-visual methodology later carried out in France in the 1960s and 70s. SLA researchers have also paid attention to prompting as a form of teacher feedback in immersion classrooms (Lyster & Ranta, 1997; Lyster, 2004) and considered it as a means to ‘push output’ (Swain & Lapkin, 1995), to call for accuracy or facilitate scaffolding (Gibbons, 2002). Lyster (2004, p. 405) identified four types of prompting: clarification requests, repetitions, metalinguistic clues and elicitation. The latter ‘entails direct questions such as ‘How do we say in French?’ or pauses that allow students to complete the teacher’s utterance’.
17Despite initial meanings which may suggest swiftness, visibility, and cause, in prompt engineering language, not all prompts aim for an output. A prompt can also be considered merely as an instruction, the textual input given to a GenAI.
18Henceforth, users may resort to prompts which are utterances deprived of apparent specific expectations (‘Thank you’; ‘Have a good day’, etc.). One may wonder whether these interactions may still be called prompts: rather than asking for the generation of a product, they recall the socio-interactional conversations held between humans and fulfil the polite function of language. Budiu et al. (2023a) identify these utterances as ‘filler prompts’ and consider them as signs of anthropomorphic language.
19Looking into the taxonomy of prompts has been fundamental to better structure the research planned. As of today, no standardised taxonomy is used in the area of prompting and research although it is called for (Santu & Feng, 2023). Studies show that prompting is still a question of trial and error (Dang et al., 2022; Zamfirescu-Pereira et al., 2023); strategies need to be developed (Ekin, 2023) as no specific strategy is developed by mainstream users, although fixed patterns or templates have been issued by researchers, for instance the text to image generator (Oppenlaender et al., 2023). Prompting remains an area to be explored, and a more thorough investigation can be done, be it in refining patterns but also in innovating perspectives on uses (White et al., 2023).
20Braun et al. (2024) have established a taxonomy encompassing three meta-dimensions: interaction, context, and outcome. ‘Interaction’ relates to the communication process and shaping; it integrates the format of the input and output as well as the CITL (computer-in-the-loop) or HITL (human-in-the-loop) criteria. ‘Context’ deals with the information fed into the computer in terms of style expected, role of machine and examples provided. Zero-shot prompts (or direct prompting) provide a simple instruction with no extra guidance or details; one shot prompts provide the model with an example of the output expected; few-shot prompts are used for more complex tasks. Few-shot prompting may also be combined with chain of thought prompting, which consists in providing intermediary reasoning steps (Wei et al., 2022). The last meta dimension, ‘outcome’, incorporates both goal and the way the prompts have been fed into the machine, in one single prompt, or following a step-by-step process, the latter being used for complex tasks. This taxonomy allowed the present research to retain general criteria and guidelines as regards relevant information to be found.
21Oppenlaender et al. (2023) have developed a more specific taxonomy, focusing on prompt modifiers in text to image generation. They have defined six prompt modifiers with different purposes: the ‘subject term’ used to define the subject, the ‘style modifier’ used to reproduce a particular style (be it through mentioning the materials used or a school), the ‘image prompt’ (that introduces an image through one or several URLs), the ‘quality booster’ (that is generally an added adjective meant to enhance the image), the ‘repeating term’ (meant to solidify the intended output through the repetition of a central term), the ‘magic term’ (an abstract concept or an incongruous word meant to add some surprising added element to the output).
22Numerous studies pinpoint the necessity of developing prompting skills and master the iterative nature of prompts (Dang et al., 2022; Oppenlaender et al., 2023; Robertson et al., 2024). The pitfall of a deceptively easy interaction with the AI is brought to the fore when one forgets it is an interaction with an LLM (Willey et al., 2023). The anthropomorphic characteristics may steer the user away from their focus on prompt structure and, even when prompt structure is implemented, a slight change can have a great impact on the output (Zhao et al., 2021). Nevertheless, the most common structure found in the internet pages or in research papers revolves around four components: request (instruction), references to either previous bot answers or external sources, format (or output indicator), framing (i.e., contextual details) (Budiu et al., 2023a). Their language format may vary: the prompt may be a question, a request (‘I need…’), a command (‘give me two examples’ …), or even an incomplete sentence. However, other special types of prompts do not use the typical structure (‘Can you do X’, ‘Give me more’, and ‘filler prompts’); prompts lacking core components may lead to long-lasting and inefficient conversations comprising numerous query refinements (Budiu et al., 2023a).
23Because prompt structure is key to the output, guides have been edited to help experts as well as laypeople. There is also a wide diversity of prompt courses available, promising breathtaking results and high-paying jobs. All offer users techniques to improve their prompt literacy. Prompting is not an intuitive skill; it is acquired through experimentation (Oppenlaender et al., 2023). Indeed, some strategies are not natural in human language, like iteration strategies, using the sandwich technique with long prompts (the same statement in two places), or resorting to caps or hyperbolic language. The intuitive approach of non-experts often appears flawed and inefficient.
24Budiu et al. (2023a) identify the reasons why prompt writing and integrating the four core components is difficult, including a lack of vocabulary or knowledge. Zamfirescu-Pereira et al. (2023) have stressed the opportunistic exploration of prompt designs by end users, as well as the difficulty to make effective progress. A limited understanding of prompt engineering and LLMs abilities, together with intuitions rooted in human social interactions and experiences, impact the quality and efficiency of the prompts (Zamfirescu-Pereira et al., 2023). Prompt engineering thus appears to be a creative skill for creating AI art (Oppenlaender et al., 2023), and a new competence meant to become essential (Korzyński et al., 2023), if not inevitable (Walter, 2024) in most jobs.
25However, it is a fast-evolving environment with rapidly changing circumstances, numerous recommendations, but no systematic rule; this may probably all be outdated when the article is published, and some points may become less important as technologies develop. Nevertheless, prompting is seen as a new essential, if not vital, skill for both teacher and students (Gattupalli, et al., 2023) and teaching prompt design needs to be carefully tailored, as it involves challenges such as a well-understood critical approach to AI. Using AI efficiently implies recognizing its fallibility (hallucinations, hits and misses, bias, etc.), and a social context that needs to be taken into account in order to prevent widening disparities in technological skills (Walter, 2024). Theophilou et al. (2023) conducted a pilot educational intervention and observed a shift in attitude, and improved prompting strategies, thanks to education. That is teaching ChatGPT prompting strategies as part of a global project to help students better understand LLM limitations, constraints and techniques, and thus tackle misuse as well as negative representations to make the most of AI (Theophilou et al., 2023).
26In this study, we explore whether AI generated poetry could be used to improve the analytical, interpreting, or writing skills of students enrolled in an English as a foreign language (EFL) literature university course. An AI-assisted creative writing poem activity was provided in the hope that the exercise, through the generation of prompts, would help students explicitly voice and transfer the literary analysis skills and literary knowledge taught in the course. Traditionally, this assignment is done as homework with students producing a poem and justifying their choices according to the specificities of the author under study. The main research question asks to what extent the exercise provided for the explicit application of the literary analysis elements taught in the course. There are three sub-questions: what are the characteristics of the prompts intuitively generated by the students? Do students intuitively manage to generate efficient prompts? What pedagogical implications can be inferred from the results?
27This study aims at finding answers to research questions. Hence it follows an inductive process relying on the formulation of research questions and the gathering of predominantly qualitative data through surveys and observations. We borrow the objectives and the methodology from different types of research, including exploratory research (through the exploration of the recent and under-investigated domain of AI within a literature class), explanatory research (e.g., we aim for a better understanding of students’ intuitive prompt writing), and action research (as the study is deeply rooted in an educational environment and aims for the improvement of language education practices) (Catroux, 2002). A number of features of correlational research have been embraced (although without manipulating or controlling variables) to facilitate the investigation of the relationship between the prompts and the explicit application of the literary analysis elements taught in the course, between the prompts and the intermediate ChatGPT answers or the final poem produced. Since participants were not randomly assigned, we are aware of potential response bias, and tried to minimise it as far as possible, mainly through anonymisation.
28A literature class of third-year students (L3) following a degree in EFL has been chosen for the experimentation (Licence Langues, littératures et civilisations étrangères, Anglais/bachelor’s degree in English, université de Guyane). The group is composed of 16 students (two boys and 14 girls). These students are familiar enough with literary analysis and creative writing not to be hampered by the instruction itself. Their AI savviness will be measured thanks to a survey.
29A pilot study was carried out with 16 pre-service teachers (M1) enrolled in a master’s course in Teaching English as a Foreign Language (TEFL) (Master Métiers de l'enseignement, de l'éducation et de la formation (MEEF) anglais, université de La Réunion). The two groups, L3 and M1, have been chosen to investigate the use of prompts by university students but also by future English teachers.
30Figure 1 below illustrates the timeline of study. The M1 pilot test was held on 30 January 2024, and the L3 experimental AI-assisted creative writing poem exercise was carried out on 6 February 2024.
Figure 1 – Phases of the study.
31The L3 experimental activity took place during the seventh class. During the preliminary L3 sessions, the approach to Langston Hughes was a conventional one with alternate lectures and workshops based on the analysis of his poetry. Among the poems studied were ‘Dream Variations’, ‘As I grew Older’, ‘The Negro Speaks of Rivers’, and ‘The Weary Blues’. Particular attention has been paid to the elements which have been chosen as representative of Langston Hughes’s style, such as a strong, vivid, poetic voice, radical language, a cause defended, an art celebrated, use of syntax, punctuation and unconventional grammar to create a rhythm, etc.
32All the students have been warned that this class would incorporate an AI tool and students have been asked to open a ChatGPT account beforehand. The day of the activity, the class began with a ten-minute conversation on students’ perception and use of AI. A single question was asked: ‘Opinion and uses of AIs?’. The participants were asked to react and interact. Next, the students were given a short comparative study to highlight Langston Hughes’s specific style. The participants then carried out the AI-assisted imitation task. No stake was given during 30 minutes, to avoid exerting pressure and influencing students in their use. To identify the teacher’s intervention within the transcripts, the students were asked to start the first prompt they wrote after the interruption with a special character (e.g., ‘9) why is this poem the same style as Langston Hughes?’ (ID13). The students then carried out a self-assessment, and the teacher evaluated the poems created. Table 1 below identifies the evaluation criteria, relying on the items expected within the poem.
Table 1 – Literary terms used during the course and expected within the AI-assisted creative writing poem in Langston Hughes’s style.
|
Themes & topics
|
racism, race relations, black pride, freedom, emancipation, struggle, elevation, change, music, American dream, working class, ordinary lives, universal man, past & present, a strong, vivid, poetic voice, radical language, or lyrical poem, etc.
|
|
Imagery & Symbolism
|
river, sun, body, wall, opposites, power, action, destruction; universality of imagery, etc.
|
|
Musicality
|
jazz, blues, repetitions, anaphora, alterations, cadence, musicality through sound; musicality through punctuation; rhymes, etc.
|
|
Structure & Prosody
|
syntax, structure of lines, irregularity, breaks, run-on-lines, punctuation, etc.
|
|
Grammar & Language
|
oral style rather than conventional written grammar rules; Black folk dialect, African tone, simple/everyday language, elegance, figurative, etc.
|
33Data were collected through two surveys and the observational study of transcripts. The L3 and M1 students carried out the experimental activity on ChatGPT. Then, they were asked to share the link of their conversations and, as a backup, to copy paste their conversations in a text document. In addition, anonymous Lime Surveys were taken online. The L3 survey included the link of the shared conversation, which allowed for the association of the survey and the conversation. The survey was taken at the beginning of the lesson for the M1 pilot test, and at the end of the lesson for the L3 experimentation. Anonymous ID numbers were automatically generated through Lime survey.
34For the M1 students, the survey included a set of eleven questions: questions 1 and 2 targeted students’ use of AIs in general; questions 3 to 6 were Likert scale questions, they tackled AIs in an educational context and the need to adapt, or not, to this new tool; questions 7 to 10 were inspired by the Turing test (Turing, 1950) to assess students’ ability to discriminate human-made or machine-made poems (two poems were human-made and two machine-made). Each question invited the participants to leave a comment to argue further his or her answer. The last question was an open question for any further comment students wanted to add.
35The L3 students were submitted a survey of sixteen questions. Questions 12 to 15 were identical to the questions asked to M1 students. Question 1 targeted their use of AI in general. Questions 2 to 11 considered the assignment itself and the output obtained with a focus on their prompting and the perceived interest of such an activity for their literature course. Questions 12 to 15 tested their ability to discriminate between human-made or machine-made poems and question 16 was an open question for any further comment they wanted to add. The survey was studied under three entries: context, interaction and feeling/opinion.
36The 16 conversation transcripts from L3 students were coded manually (mainly through inductive coding) with ATLAS.ti (version 23.2.1), a data analysis software facilitating qualitative data analysis. 178 codes were generated from scratch to cover the samples; each time a new code was created, a text search was conveyed to recode the samples already coded and code the following ones. Deductive coding was occasionally used, mostly to look for specific terms or terms not found (e.g., although ‘metaphor’ is used, ‘comparison’ was never used; ‘American dream’ appeared only once in a ChatGPT answer). Students’ prompts and ChatGPT answers were identified (<Student>; <GPT>). 501 quotations were created (240 GPT answers, 238 student prompts).
37To identify the characteristics of the prompts intuitively used by the students, we devoted 16 codes to the formal aspect of the students’ prompts. We thus highlighted capital letters (e.g., ‘FREE VERSE NOW’ (ID15)); familiar terms (‘Bro, you’re basically writing sonnets’ (ID14)); politeness; discourtesy (‘NO STANZA, NO STRUCTURE NOW’ (ID13)); language mistakes (spelling, grammar, punctuation mistakes); unclear antecedents; the absence of a verb, of initial capital letters, full stops or question marks. The prompts were identified either as commands (‘add a personal twist to it’ (ID19); ‘stop with the structure’ (ID23)); declarative sentences or segments (‘It is too simple’ (ID21)); personal comments (‘Nice thanks’ (ID21)); questions (‘can you structure the poem so it will look like a free form poem’ (ID6)); requests (‘You are my literature teacher in L3 LLCER, I need you to write a poem inspired by Langston Hughes’ (ID10)); and wishes (‘I’d like to delete all the rhymes’ (ID12)). All the students’ prompts were subject to a word count and an idea count identifying the number of different ideas or elements within the prompt (e.g., ‘Add more prosaic elements, punctuation, free verses’ (ID14) was coded <C nb ideas/elements 3>). The student’s initial prompt launching the chat was identified; so was the presence of contextual elements and the use of the exact same words as the teacher’s instructions. The clarity of the prompt and the use of idiomatic language were also coded. However, we consider the coding of such elements as a heavily biased enterprise, so conclusions need to be taken with caution.
38In order to scrutinize the type of process used by the participants and look at the way the prompts were generated, customized, and used to re-equilibrate the AI poem generated, we identified the total number of prompts used within the transcripts; the level of the poem targeted (poem, stanza, title, verse or word level); the purpose of the prompt (22 codes, e.g., <a broad task/Prompt> (such as ‘write a poem in Hughes’s style’); <adding an improvement>; <asking for information or a definition>; <asking for proof>; <asking for variation>; <asking for feedback>; <repeating>, etc.).
39To try and define whether the prompts were efficient, we marked the final poem created, and coded the intermediary ChatGPT answers: <answer seems to fit last prompt given>; <Answer inappropriate/not satisfying>; <Answer partially satisfying>. The subjectivity of such coding is evident, and here again we call for a cautious interpretation. Additional codes were used to analyse ChatGPT answers, such as <Answer provides literary information>; <ChatGPT explicits/justifies its choice> or <Answer highlights textual interpretation of instructions>. The final poem was marked A, B, C or D using five criteria: ‘something personal’, ‘Topic or theme corresponding to Langston Hughes’, ‘Imagery and musicality’, ‘Global structure’, ‘Grammar and punctuation’. A poem marked A meets four to five criteria, B meets three, C meets two and D meets only one or none.
40To identify the explicit application of the literary analysis elements taught in the course, we looked at the literary terms explicitly used. Only the literary terms explicitly mentioned within the participants’ prompts or the GPT answers were coded. Thus, the term ‘struggle’ was coded <struggle>; but ‘making violent efforts to get free’ would not be labelled <struggle>. The words within the poems were not taken into account. Thus, the word ‘Harlem’ was recurrent in the poems, but it was not included when used within a poem. As for all other codes, no matter the amount of repetitions, a code was only used once within each quotation (each student prompt, or each ChatGPT answer, no matter the amount of words). A general code was used to identify references to Langston Hughes or his style. The table below provides a synthesis of the terms expected and the codes created.
Table 2 – Codes used to codify the literary terms explicitly found in the conversations.
|
Themes & topics
|
<T African American/Black/Blackness/Black pride> ; <T authenticity>; <T culture/cultural>; <T discrimination>; <T freedom>; <T Harlem>; <T Harlem Renaissance>; <T hope/optimism>; <T identity>;< T loneliness>; <T ordinary>; <T personal twist>; <T poetic voice>; <T poverty>; <T racism>; <T resilience>; <T segregation>; <T setting>; <T social realism>; <T strength>; <T struggle>; <T theme>; <T topic>
|
|
Imagery & Symbolism
|
<IS imagery>; <IS metaphor>; <IS simile>; <IS symbolism>; <IS vivid/vividly>
|
|
Musicality
|
<M repetition.s>; <M alternate rhymes>; <M blues>; <M enclosed rhymes>; <M jazz>; <M musicality>; <M rhymes>; <M rhythm>
|
|
Structure & Prosody
|
<SP free flow>; <SP free form>; <SP free verse>; <SP lines>; <SP meter/metrical>; <SP punctuation>; <SP run on line>; <SP sonnet>; <SP stanza>; <SP structure.s>; <SP verse>
|
|
Grammar & Language
|
<L accessibility>; <L broad audience>; <L simplicity>; <L vernacular>
|
41The survey submitted to the students helped us get a glimpse of how familiar they were with AI, and what their opinions about this new tool were. It was also used to understand how they perceived their interaction with the AI, and their opinions on the output they had obtained. Two participants did not complete the survey in L3. In both classes, half of the students rarely or never use AI and, in L3, seven students out of 14 mention ChatGPT when asked if they use AIs; four of them use more specific AIs such as grammar checkers. All of these seven students mention an AI related to academic tasks. It is not the case with students at a master’s level. The latter mention using AIs for personal activities like cooking ‘to get examples of healthy recipes’ (N5) but also for their academic work, especially rephrasing. They never mention complex tasks involving prompts which would need to be elaborate.
42Most L3 students believed the result to be poor or very poor, and there is no correlation between their opinion on the result and their use of AI, as those who use it frequently find the result as poor or average as those who use it rarely or never. Eight students found it hard to find the right words or the right sentence structure for their prompts. But this has no correlation with their being familiar with AIs or not, as three of the seven students who never use AIs found it easy to find the right words. Nine students said they had followed intuition. Only one had prepared their prompt, while two had listed the elements they needed to focus on before their initial prompt. Four students declared they had used strategies which they did not usually use in class. Ten students believe that prompt writing should be taught at university, and only two of them believe it cannot be taught. Two students believe it should be a high priority, and six believe it can be included in the courses they already have, such as this literature class. Five participants believe these prompting classes should be given by IT engineers.
43As regards their feeling and the collaboration with the AI, 10 out of the 14 L3 students have experienced frustration, with one commenting ‘I was more focused on not insulting the AI than on the actual poetry’ (ID14) when asked if this class had helped them better understand Langston Hughes’s poetry. To this latter question, most participants answered ‘To some extent’ or less. Considering the authorship of the poems generated, six of the L3 students replied that the AI, themselves and Langston Hughes were the authors; one student left a comment explaining that ‘if the author’s style is so distinct that it can be replicated by a trained AI, the poet or the artist should be credited’ (ID21). None of the students believe they alone were the authors, and five students picked chat GPT as an author.
44Eight M1 students consider teaching prompting at university level as a high to essential priority, and only one less considers it high to essential for high school students. Ten consider it a moderate priority to not at all a priority for pupils under the age of 13.
45Both groups of students were presented with four poems. Poems 1 and 4 were AI produced. Poem 1 was deemed written by a human for three L3 students but eight M1 students. Poem 4 was deemed written by a human for more than half of each group. As for the human-made poems, no entire group agreed on the human origin, but overall, the L3 fared better in discriminating between human and AI poetry.
46The analysis of the formal aspect of the transcripts revealed that 66 out of the 239 student prompts comprised language mistakes (27,6%), including syntactic or grammatical errors (“*try again the poem” (ID10); “*Why this poem is like Langston Hughes” (ID6)), lexical errors (“*Do not jump line” (ID13); “*can you change the ponctuation even remove some” (ID16)), spelling slips (“*Delete all the rhymes fron this poem” (ID12); “*mae it only 3stanza” (ID18)), or the use of inadequate punctuation in questions (“*Can you write a poem in the style of Langston Hughes but with a personal twist.” (ID6)) The absence of a question mark was not coded as a mistake since absence of final punctuation, whether full stop or question mark, was coded separately: 75% of the prompts written by the students had no final punctuation (180 prompts). 48.5% did not start with a capital letter (116 prompts). Students often used elliptical sentences or segments (e.g., “*Can add a repetition in the third stanza” (ID17); “with less rhymes” (ID10)).
47Although all these elements could impact the clarity of the message, only 11% of the prompts were deemed unclear. Numbers however need to be taken with caution, the clarity of the message being a subjective element. Clarity seemed compromised in prompts such as: “*whats the structure that defines Langston Hughes style in his peace of works” (ID 20); “*POEM IN LANGSTON HUGHES WITH A PERSONNAL TWIST” (ID12); “*Do not jump line” (ID13). However, a teacher aware of the task instructions or an EFL teacher acquainted with loan translations may both consider these sentences comprehensible.
48The analysis of the transcripts also disclosed the use of common abbreviations (“More pls” (ID 15); unconventional language (“Do you know what Free verse is supposed to be like? Cause this poem is not” (ID7); “Bro, you’re basically writing sonnets, thats not what im asking…” (ID 14)); informal exclamations (“okay” (ID14); “GOOOOOOOOOOD JOOOOOOOOB” (ID21)).
49Politeness to the machine appeared in seven transcripts (ID5, 7, 10, 14, 15, 18, 20), through greetings (“Hello can you help me write a poem in Langston Hughes ? …” (ID 7); “Aloa Chat GTP !” (ID20); “hiii” (ID15)); the use of the adverb “please” (“can I have another one please” (ID5); “Make it into one big Stanza please” (ID14)); or the use of the modal “could” (“Could you define and analyse his poetry and literary style” (ID14); “*could you change the strucure a bit” (ID18)). Anthropomorphism reached a climax when ID20 asked ChatGPT its opinion (“Do you like this peace of work?” (ID20)), and ID9 its interpretation of life (“okay what is your interpretation of life” (ID9)).
50Dissatisfaction with ChatGPT’s answers appeared through the use of another prompt, or explicitly through negations (“It is not a free verse, try again” (ID10); “this one is not enough Langston Hughes” (ID5)), as well as the use of rhetorical questions (“Okay but wasn’t he also known for his free verse writing style?” (ID7)). Dissatisfaction was at times associated to capital letters or the adverb “now”, which we interpreted as signs of exasperation (“NO STANZA, NO STRUCTURE NOW” (ID13); “FREE VERSE NOW” (ID15)). Annoyance permeated some discourteous prompts which seemed to be scolding the machine (“Okay make the first stanza’s verses longer, add at least two words to each and stop the rhyming omg” (ID14); “you know what forget it” (ID18)). ID21 resorted to four extremely aggressive prompts abusing ChatGPT: “You really don’t have talent in art you fool”; “I’m embarrassed at your incompetence”; “i could never fail my parents like you fail me”; “Jump off a bridge” (ID21).
51Two out of the 16 initial prompts were single greeting words (“hiii” (ID15); “Hello” (ID10)); the second prompt then used by ID15 was “You know Langston Hughes?” while ID10 was the only one to provide some context, asking: “You are my literature teacher in L3 LLCER, I need you to write a poem inspired by Langston Hughes” (ID10). Three initial prompts were very similar to the wording of the teacher’s instructions (e.g., “Can you write a poem in the style of Langston Hughes but with a personal twist.” (ID6); “Write a poem in the style of Langston Hughes. It has to have my personal twist.” (ID20)); four other prompts comprised at least some of the words used in the teacher’s instructions (e.g., “write a poem in the style of langston hughes” (ID3). Seven prompts were thus rather broad and asked for the production of a poem. Five other prompts (and ID15’s second prompt) explicitly requested information on Langston Hughes, his poetry or the method to write in the style of Hughes: “Hi, what do you know of Langston Hugues” (ID14); “How to write a poem in the style of Langston Hughes?” (ID16); “*what do you know about langston hughs” (ID18). One prompt added a thematic content request: “*Hello can you help me write a poem in Langston Hughes? I want it to be about strength and loneliness” (ID7). The last prompt showed originality since the student copied and pasted Hughes’s poem “The Weary Blues” (241 words; ID13). Apart from ID13’s poem, only four prompts out of 15 began with a capital letter and ended with a full stop or a question mark. However, again, this did not seem to significantly affect the clarity of the prompt. At this stage, there were five commands (e.g., “Write…” (ID12), “Describe…” (ID21)) and eight questions (e.g., “Can you write…” (ID6), “What is distinct…” (ID11), “How to write…” (ID11). Seven of the ChatGPT answers following the initial prompts appeared to be partially satisfying (usually when the student had asked for a poem), and nine seemed to be satisfying (usually when the student had asked for information on Hughes).
52The final results of the poem production were somewhat unsatisfactory: only one student was awarded an A, five students obtained a B, and 10 a C. However, no production was given a D. Most poems equated a feeling of loneliness with a wandering of the poetic persona in the streets of Harlem, offering a clear focus on the urban setting; whereas Langston Hughes focuses on the people, and the theme of loneliness is used to support a sense of violent segregation. The musicality was never quite reproduced as ChatGPT remained rather conventional in syntax and in line structure, if not in stanza structure. ChatGPT generated mostly quatrains, even when prompted to do otherwise.
53Yet, within the conversation, 49% of ChatGPT answers seemed to fit the last prompt given by the student (118/240 GPT answers). Considering the previous student prompt given, 29% of ChatGPT answers appeared to be partially satisfying, and 19% were coded as non-satisfactory.
54Figure 2 shows that in total, each student used from 4 to 33 prompts. The distribution is positively skewed. The average is 15 prompts, which is also the number of prompts given by the sole student who was awarded an A Grade. The five students who obtained a B grade used on average 12 prompts. The 10 students who obtained a C grade used on average 17 prompts. ID21 (C grade) used 22 prompts including four prompts which appeared out of scope, or at least could barely lead to the improvement of the poem (“you really don’t have talent in art you fool”; “I’m embarrassed at your incompetence”; “i could never fail my parents like you fail me”; “Jump off a bridge” (ID21)). Therefore, no correlation was found in the corpus between the number of prompts given by the student and the quality of the final production.
Figure 2 – Number of prompts given by the students in the conversation.
55We could not identify any recurrent global method in the students’ requests. The students seemed to iteratively engage with ChatGPT in a conversational and rather unorganized manner. ID7, who was awarded an A, started by announcing a theme and working at poem level, primarily on the structure (Figure 3). They then introduced improvements focusing on musicality, free verse, and a title. ID 10 appeared at a loss with structure. Confronted with the production of sonnets (Figure 4), they were unable to stir the AI towards the required poetic form. Although they began with a promising contextualization and tackled style, structure, imagery and theme, their final production was poor. ID10’s transcript is characterised by a strong focus on structure (which the student tries to modify with references to sonnet and free verse); the repetition of similar prompts which had previously seemed abandoned (prompt 3: “With less rhymes”, prompt 13: “No more rhyme”); many word for word repetitions (which did not prove efficient here, although other transcripts showed that mere repetition was sometimes fruitful); and a change of theme after 12 prompts, which does not actually make the machine start from scratch (structure and punctuation remain the same).
Figure 3 – ID7’s prompts (15 prompts, A Grade).
Figure 4 – ID10’s prompts (23 prompts, C Grade).
56The literary terms explicitly used by ChatGPT and the students were distributed into the five domains (language and grammar, imagery and symbolism, musicality, structure and prosody, themes and topics). A code-document analysis revealed that every script comprised some literary terms, used either by ChatGPT or the students. The box-and-whisker plot below (Figure 5) displays the dataset on a five-number summary: the median value of the data set is 39 literary terms. The lower whisker shows that only two literary terms were found in the poorest script (ID3, <Harlem>; <personal twist>); the second lowest number is 17 (ID4). The upper whisker corresponds to the highest data point, which reaches 68 literary terms (ID14); the second richest script comprises 59 terms (ID7). The interquartile range is thus of 33, and the average number of terms per script is 38.
Figure 5 – Distribution of the number of literary terms used by ChatGPT or the students within the scripts.
57A co-occurrence analysis, however, shows that participants employed more literary terms pertaining to musicality and structure than ChatGPT (Figures 6 and 7). 31 occurrences of “free verse” and 21 occurrences of “stanza” were coded in the students’ prompts (“free verse” was coded six times in ChatGPT answers, and “stanza” 12 times). Conversely, ChatGPT made use of many terms which were not at all found in the students’ prompts (e.g., resilience (25); struggle (22); blues (21); jazz (20); identity (16); Harlem (12); rhythm (18), etc.).
Figure 6 – Imagery & symbolism, language, musicality, structure & prosody: table showing the terms explicitly used by ChatGPT and the terms explicitly used by the students.
Figure 7 – Themes and topics: table showing the terms explicitly used by ChatGPT and the terms explicitly used by the students.
58Together with the lexical difference of the terms used by ChatGPT and the students, the amount of literary terms employed by ChatGPT is remarkable. ChatGPT answers expose students to a large quantity of literary terms. Figures 8 and 9 highlight the number of words used by the students, and by ChatGPT. The student prompt “*What is distinct about Langstong Hughes poem writing style?” (L3ID11) thus triggered an answer providing coherent literary information pertaining to 18 codes (<IS imagery; IS symbolism; IS vivid/vividly; L simplicity; L vernacular; M blues; M jazz; M rhythm; T African American; T authenticity; T culture/cultural; T Harlem Renaissance; T identity; T ordinary; T poverty; T racism; T resilience: T struggle>).
Figure 8 – Sankey diagram highlighting the amount of literary terms used by the students.
Figure 9 – Sankey diagram highlighting the amount of literary terms used by ChatGPT.
59The traditional imitation writing assignment, especially because it comprises a self-analysis section, aims at having the students reflect on their choices in content as well as in form. We believed that the students would use an AI if asked to do this task today, and we wondered if their prompts would incorporate traces of their analysis in their command to generate a poem. Our study thus questioned the extent to which the GPT exercise provided for the explicit application of the literary analysis elements taught in the course. The analysis of the corpus showed that despite lukewarm final productions, ChatGPT exposed students to a large quantity of literary terms which were not always the same terms as the ones used by the students and pertained to the five domains expected (language and grammar, imagery and symbolism, musicality, structure and prosody, themes and topics). Moreover, these terms were used within answers fitted to the student’s previous question, which opens the path towards differentiated instruction.
60The study also scrutinised the characteristics of the prompts intuitively generated by the participants and questioned the efficiency of such prompts. We concluded that these prompts comprised language mistakes, unconventional language, abbreviations, elliptical sentences, typos, signs of anthropomorphism, etc., that is to say many characteristics of modern instantaneous communication and human-bot conversations. However, these elements did not necessarily impact the clarity of the message.
61The corpus revealed that participants engaged with ChatGPT in a conversational and rather unorganized manner. Writing a poem involves forethought about a subject or a situation, and a slow line by line process. This appears to have been completely skipped by all the students but one. None of the participants asked for a first line. The use of the AI utterly reversed the process from macro to micro, generating one poem which then had to be altered. Furthermore, most literary terms focused on the global aspect of the poem and on its layout on the page. Maybe one way of circumventing this reversed process would be to start the task in the class, make the students reflect on the process in action, and on how they would proceed with an LLM to maintain the same process.
62Although ChatGPT exposed students to a range of literary terms that differed from those they used, the students themselves hardly ever used hypernyms or hyponyms, preferring to repeat the same terms. It could be that the students were quickly taken over by their inability to steer the LLM towards the right output. Emotions, as well as impatience, prevailed, given the short if not elliptical prompts, conveying the idea that they expected the AI to lead the way, instead of using it as an assistant. Their frustration is palpable in form and content of prompts.
63It is also possible that students did not thoroughly read the generated poems, asking straight away for a change in the layout, for example in the number of lines in a stanza. The swiftness of the LLM responses may have led students to adapt to the LLM by adding immediacy in their interaction, turning the required reflexivity into spontaneity. The latter being also visible in the quick swaying of their emotional response to one and the same GPT response. Adding the LLM tool has changed stakes for the students; it could be that the novelty of the tool has turned the assignment away from its original target because of loss of academic landmarks and resurfacing of chatbot interaction habits.
64Our third sub-question considered the pedagogical implications which could be inferred from the results. The why teach prompt engineering has been demonstrated by researchers who described the blatant need for prompt engineering education both to tackle misuse (Theophilou et al., 2023) and develop what may become a vital skill (Korzyński et al. 2023; Smith, 2023). Similarly, the surveys and discussions showed that students needed guidance since they were unsure about how to use ChatGPT, and whether it was right or not. Moreover, within a literature course, a reflection on creativity, authorship and human AI cooperation appeared essential. We believe most students will use LLMs for their home assignment, and this reflection could help in keeping the academic stakes a priority in the task if they carry it out unsupervised.
65However, we have found only partial answers to the how to teach prompt engineering. Students-teachers reflections on digital literacy may integrate digital citizenship, ethical considerations or needs analyses. On the other hand, providing prompt writing recommendations is a thorny issue: it is difficult to go beyond general tips on source check, message clarity, needs and wants analyses, organisation, or common sense. What is sure is that the teacher’s traditional request for full, grammatical, polite sentences in class appeared irrelevant in a human-AI conversation. Nonetheless, the rude and impatient prompts used by a student call for abuse awareness and prevention, since psychologists have proven that online behaviours may have offline consequences. Our study demonstrated no correlation between the number and type of prompts used and the final result obtained. Moreover, within two transcripts, the same prompts did not yield the same results. Despite the general recommendations available online, prompt engineering is a fast-evolving field and the efficiency of the techniques depends on the environment.
66Finally, it appears we overlooked a certain amount of limitations, such as ChatGPT’s biases and its inability to get off the beaten track; indeed when asked if it could use the terms “Black girl”, the AI answered “As an AI language model, I strive to use language that is respectful, inclusive, and sensitive to diverse identities and experiences.” It also appeared socially biased as when asked about Langston Hughes, ChatGPT mentioned Harlem and associated it to “struggle”, “poverty”, and the more fashionable and pervasive term of “resilience”. This pitfall has been flagged by the BIG-bench Collaboration (Srivastava et al., 2023). ChatGPT also fared rather badly in the patterns produced, as it offered sonnets, traditional rhyme patterns and quatrains. It seemed unable to generate a poem not following Western mainstream conventions. Any ChatGPT production should therefore entail consideration of permeability to toxic content, simplification, bias, and the extent to which GenAI tools are WEIRD–Western, educated, industrialised, rich and democratic.
67We had not overlooked the additional skills that the use of a GenAI would involve, yet we had not anticipated the point to which the imitation process would be reversed, thus requiring a complete change of perspective. Furthermore, the contiguity between GenAI and mere chatbot conversation has amplified participants’ loss of focus on keeping some hindsight to guide the LLM towards the desired output. This point would have benefited from timing the ChatGPT conversation so as to have a clearer view of the time spent reading generated poems and explanations.
68The activity required not only literary but digital literacy and digital citizenship skills. It also entailed a redefinition of teaching posture, since the teacher lost control during the time span of the student-bot conversations, while machine tutoring was momentarily made possible. The exercise indisputably exposed students to a large quantity of literary terms fitted to their individual conversations with the chatbot, and it provided for the explicit use of literary vocabulary. Such a benefit is, however, tempered with limitations and complications. The spontaneous approach of students in their completing of the assignment using ChatGPT could be reined in with guidelines about prompting. However, the prompting skills needed to be efficient add another layer of difficulty to the disciplinary skills already at stake in carrying out the assignment. This could also widen the gap between computer-wise students and those who are not, and who may struggle further with this extra layer. This experiment has also shown how the emotional response prevails over a more objective, academic response which draws the assignment away from its purpose.