Navigation – Plan du site

AccueilVolumesVol. 25, n° 1RechercheAn Attempt at Describing Learning...

Recherche

An Attempt at Describing Learning Curves for a Complex Sub-Task in Reading Acquisition

Une tentative de description des courbes d'apprentissage pour une tâche intermédiaire complexe de l'acquisition de la lecture
Hannah Kermes et Kay Berkling

Résumés

Les courbes d'apprentissage sont un phénomène bien connu dans l'apprentissage, tout comme le fait de savoir que faire des erreurs fait partie du processus d'apprentissage. Il est également clair que ces courbes d'apprentissage sont individuelles et constituent donc un grand défi pour leur description et comparaison. Avec la possibilité de collecter de grandes quantités de données via des jeux d'apprentissage avec un contenu généré de manière adaptative, il est désormais possible de poser un nouveau regard sur ce processus. Grâce à l'utilisation de jeux de littératie déployés dans un cadre scolaire pour le projet iRead EU Horizon 2020, l'application Navigo fournit une tâche complexe déguisée en jeu, permettant aux apprenants de s'entraîner à distinguer la longueur des voyelles dans les mots bisyllabiques en allemand. L'engagement de l'apprenant dans le jeu a abouti au plus grand corpus longitudinal jamais collecté pour ce type de tâche auprès de 251 élèves d'une école élémentaire allemande. Les données résultantes présentent des courbes d'apprentissage décrivant le temps de réponse et l'exactitude sur plusieurs semaines pour les sessions de jeu de chaque élève. Le travail présenté ici tente (a) de modéliser et paramétrer ces courbes, (b) d'automatiser leur classification en formes communes à travers une population plus large, et (c) de détecter la maîtrise. Ce faisant, nous proposons une méthode de représentation et d'interprétation de la courbe d'apprentissage et nous l'appliquons aux données. Un modèle descriptible de traitement cognitif semble être observable et commun à toutes les courbes qui peuvent permettre une prédiction de la maîtrise et de la capacité de transfert de compétences vers d'autres environnements pour un sous-ensemble de joueurs. En conséquence, nous avons pu décrire cinq types généraux de progressions communes. Ces résultats sont en partie étayés par des données supplémentaires provenant du pré-test et du post-test sous la forme d'activités papier-crayon. Le travail présenté ici devrait avoir un impact en démontrant l'importance des données d'entrée à grande échelle dans la formation des compétences en littératie sur quelques exemples comme c'est la norme dans les manuels scolaires statiques.

Haut de page

Texte intégral

iRead is part of a project that has received funding from the European Union's Horizon 2020 research and innovation program under grant agreement No. 731724. The results presented reflect only the authors' view and the Agency is not responsible for any use that may be made of the information it contains. The authors have no conflict of interest with the content of this paper.

1. Introduction

1.1. Current Situation

1Schools in Germany are increasingly struggling to teach reading and writing to pupils and the proportion of children who meet or exceed the standards in the area of orthography has fallen from 65% to 55% according to the national standardised test IQB-Bildungstrend 2016 (Stanat et al., 2017). In 2017, 34.8% of children in their first year of elementary school were diagnosed with developmental disorders, which represents an increase of 27,5% from 2008. 82% of the issues relate to language acquisition, with boys being affected 1.5 more often than girls according to WiDO AOK (health insurance) (Waltersbacher, 2018). Given these developments, it is especially important to provide high-quality learning materials, personalised support and adequate amounts of training materials. Literacy games are able to provide this, and when combined with learning analytics, can offer insights into each pupil's learning. In order to achieve this goal, it is important to understand and interpret these data.

1.2. Language Acquisition Theory with a Focus on Reading and Writing

2Children are very sensitive to the orthographic regularities in their writing system. It has been shown that these regularities must be present in learning materials for early literacy education (Ouellette & Sénéchal, 2008; Pacton et al., 2001). Self-teaching theory (Share, 1995) states that children are able to establish specific orthographic knowledge on their own with sufficient reading experience (Caravolas et al., 2001; Caravolas & Volín, 2001; Martinet et al., 2004; Ziegler et al., 2014; Cunningham, 2006). However, this experience requires a combination of regularities and well-chosen "self-teaching" opportunities. Through correctly sequenced input, cognitive pressure to restructure internal representation results in improved lexical representation and the ability to generalise (Perfetti & Hart, 2002). By providing a common pattern with varying graphemes and phonemes in various combinations, the learner trains phonemic awareness through a mixed blending of grapheme-phoneme correspondences. (For example, Hase (rabbit), Wade, (calf), Rose (rose), Rede (speech), and Tube (tube) are examples of consonant-vowel sequences in German that follow the same prosodic pattern for long vowels with different combinations of consonants.) This process of forming internal representations of patterns requires large amounts of training, including unfamiliar words in familiar patterns (Melby-Lervåg & Hulme, 2010). The controlled input improves the inner lexicon (Ziegler & Goswami, 2005; Mann & Wimmer, 2002; Duncan et al., 2006). Mastering orthographic conventions in turn influences both reading fluency and writing ability (Ehri, 1987; Cunningham, 2006; Castles & Nation, 2006; Pacton et al., 2001; Anderson et al., 1978; Bhattacharya & Ehri, 2004). In summary, the literature states that pupils require a large amount of input material that is oriented along familiar patterns with unfamiliar words. Carefully sequenced training material should have an equal effect on reading and writing ability. The method of "Phonics" for English follows these principles. Literacy games are able to provide large amounts of training material that is sequenced according to linguistic rules describing difficulty. In contrast to teachers or static books, games can acquire their content from large databases to present as many combinations of input material as the language has to offer to the learners until they reach proficiency according to their own learning speed, all the while hiding the learning material behind motivating game-like adventures. The automation of games provides personalisation, direct feedback, and results in individual learning trajectories that are called "learning curves" in the literature.

1.3. Previous Research on Learning Curves

3The study of "learning curves" has a long history and was traditionally used for predicting industry productivity (Glock et al., 2019), but originates from terminology used in psychology based on Ebbinghaus's original work on the human mind (Ebbinghaus 1964). The "learning curve" describes the process of learning, typically by plotting time vs learning, using "whichever parameter to represent it" (Swift, 1903). Language learning as well has a long history of being modelled (Case, 2016) and is very often modelled as log functions with an asymptotic approach towards mastery. The authors agree that "learning curves" yield valuable insight but there is no agreement on how they can be modelled (Daller et al., 2013). In education, "learning curves" are employed as a model of pupils (Pavlik et al., 2021a) in order to optimally sequence the input material (Pavlik et al., 2021b). Learning is known to exhibit u-shaped phenomena, hence the technical term "curve" will be used throughout this paper. In language learning, this curve is believed to reflect a vacillation between mental grammatical models. Error-driven events lead to cognitive constraint-rerankings that cause learners' mistakes on previously correct items. The complexity of these learning processes are nicely explained in a case study by Stemberger et al. (1999) and explored by Tessier (2019). Worthwhile contributions to machine learning have shown a theoretical basis for the necessity of such u-shaped learning curves (Carlucci & Case, 2013) and their possible optimisations (Viering et al., 2020) that describe the learners' (or machine's) ability to generalise given the training set. The data for learning curves during gaming are automatically gathered for a large population of players. They contain valuable information regarding player background, game content, sequences, regularity of play, speed, and correctness and can yield enough information to add significant insight to the above research and potentially impact what is known about both machine and human learning. This paper explores the use of these data in order to learn more about the "y-axis" that is to be plotted against time to depict the learning process.

1.4. Literacy Games–Effects

4The effects of literacy games on learning outcomes have been studied in the sciences (Khan et al., 2020) and very often in the field of second language acquisition. Specific interest in games for elementary school-aged children is rare, especially in the German context (Acquah & Katz, 2020). While it is well known that games are very motivating to learners (Erhel & Jamet, 2013), it would also be helpful to know more about the effectiveness of games, which is a complex topic mixing emotions, cognition, input data design, and immediate feedback (Pattemore et al., 2019; Greipl et al., 2021; Elaachak, 2020). Very little is known about literacy games for elementary school-aged children regarding orthography acquisition and reading and writing effectiveness of the games in German. Nor is there much known about content and sequence with respect to the design constraints elaborated above. However, Berkling et al. describe a method for sequencing language input (Berkling, 2016; Berkling & Reichel, 2016), and some publications touch on the effectiveness of such games (Berkling, 2017; Berkling & Franken, 2019) and on game design for serious content (Berkling et al., 2017). A particular example of learning was published by Berkling and Kermes (2020); it explains the game and learning curves that will be analysed in more detail in this paper.

2. Research Motivation

5Digital literacy games can provide a way to collect learner data over longer periods of time. The game uses linguistically selected content from large databases in order to provide the player with familiar patterns with changing letters. This provides practice on specific cognitively challenging concepts and allows us to carry out a longitudinal study on the resulting learning curves. The various resulting data measurements provide material to determine the parameters that allow us to best capture and depict "learning" as a function of time and visualise a high-level view on individual learning development for the chosen task. We are interested in analysing the learning curves for tasks that are not taught explicitly in school in order to measure the effect of the game on learning. The paper will address the following research questions:

  • RQ1: How can learning curves be characterised?

  • RQ2: Are there commonalities or individual classifiable differences in "learning curves"?

  • RQ3: Is it possible to recognise and analyse outliers?

6Section 3 will describe the applied methodology, including instruments, participants, and data collection and processing. Section 4 will detail the results. Section 5 will end with a discussion of the findings and concluding remarks about remaining research questions to be addressed in future work.

3. Methodology

7This section describes the content for the game chosen for the study as part of Navigo, the participating pupils, and the resulting log data.

3.1. Instruments

(a) Navigo Game within iRead

8iRead is a 4-year (2017-2020) EU Horizon 2020 project1 that aims to develop personalised learning technologies to support reading skills. It is designed for elementary school children and was developed for multiple languages. The game was implemented for beginner-level L1 (native) readers in Germany. It is unique in that it is adaptive and based on a sequenced language model and rule-based automatic retrieval from a large underlying database of marked-up data2. The storyline in the mobile game Navigo takes place in an Egyptian setting of pyramids and oases in the desert with the goal of saving a village by working through the games. There are some 14 different game settings that are fed with content through a rule-based system or use hand-crafted exercises. Content is adapted to the player's performance. An underlying domain model provides content and sequence to the games (Mastropavlou et al., 2020). For the purpose of this study, we focus on one particular mini game within Navigo called "Watch your Step". It provides a graphic display of hopping across stones (Event) to get across a chasm (Session) as shown in Figure 1. At each event or step, a question is provided with around five possible answers with only one correct solution. The choices are words that are displayed on the stones. Feedback occurs in two rounds. After the first wrong step, the avatar is returned to the previous stone while the incorrect answer disappears into the chasm. After the second false answer, the avatar jumps onto the correct answer as the game progresses. Each mistake results in a loss of life. After five mistakes, the game ends. The impact of feedback on learning is beyond the scope of this paper but plays an important role in learning from mistakes in the process (Berkling & Kermes, 2020). The generated words in the game are extracted from the database using linguistic rules describing target (long vowel) and distractor (short vowel) words of trochee form3, such as Hase (rabbit) and Wiese (meadow) (long) vs retten (save) and Wasser (water) (short). The player is asked at each step to select the word with the long middle vowel. The distractors with short vowels are generally followed by a double consonant. The content is selected in a rule-based manner and presented randomly from a database of around 15,000 words. The goal for the player is to distinguish orthographic patterns for long and short vowels rather than memorising a finite set of words and their spellings. As described in the introduction, this sort of training should lead to cognitive restructuring to allow for generalisations, and we expect this to be visible in the learning curve of the player. An example of game session input is shown in Table 1 with Figure 1 showing the game interface. Five (multiple-choice question) steps lead to the other side of the chasm. Only one of the stones surrounding the player at each step has the correct answer, namely a long vowel in the middle. The correct word, containing a long vowel, is shown in parentheses in the table. The following data is collected from the game for each step:

  • Time taken for each answer

  • Correctness of answer

9We distinguish between two sets of vowel graphemes (<a>,<e>,<o>,<u>) vs (<i>/<ie>). The former set of vowels (KV) has no visible distinction of length. Their length is determined by the following consonant pattern of two consonant letters ('tt','mm'…). The latter vowel <i> vs <ie> denotes in itself whether it is short <i> or long <ie> while the short vowel is still followed by two consonants. These two subtasks are distinguished and tracked separately. The task is to select the bi-syllabic word with the long middle vowel. Top: Timed race. Bottom: Multiple-choice without time limit for "Watch your Step", showing a player crossing the chasm.

Figure 1–Navigo App example games.

Figure 1–Navigo App example games.

Table 1–Example answers for multiple choices given the task of finding the word with a long vowel.

Answer Choices by Step/Event for One Sample Session

1

(Wiese)

hassen

motzen

lecker

Type: ie

2

(Rasen)

Mutter

rennen

trennen

Type: KV or long vowel

3

netter

Betten

(Ruder)

flotter

Type: KV or long vowel

4

Kletten

Klassen

Flammen

(Strömen)

Type: KV or long vowel

5

Hammel

(dösen)

lassen

Risse

Type: KV or long vowel

(b) Word Riddles

10A written test in the form of a word riddle consisted of 20 items with a very high frequency of trochee-formed words with short and long vowels. Different tests were prepared for each school year using the same patterns but different words to avoid memorisation effects across tests. To avoid the ceiling effect (when the exam is too easy for the test taker), the difficulty of vocabulary was slightly increased from Grade 1 to Grade 4 while maintaining the balance between long and short vowel trochees patterns and word frequencies according to Schroeder et al. (2015), while allowing for the capture of over-generalisations. Each item is presented through a sentence with one missing word and a picture to support the semantics of the sentence in order to help the pupil identify and write the desired word into the sentence on paper. Figure 2 shows an example of one of these items. These word riddles, as they are called for the pupils, represent our pre- and post-test to allow us to track improvement in writing due to the intervention of the game.

Figure 2–Example of elicitation for orthographic skills from pupils.

Figure 2–Example of elicitation for orthographic skills from pupils.

11The expected answer in Figure 2 is "Das ist ein Hammer" (This is a hammer). The combination of sentence and image as shown here supports elicitation of the desired target word in almost all cases.

3.2. Participants

12Participants for the iRead project were recruited in the state of Baden-Württemberg, Germany, resulting in a total of nine different elementary schools, 20 classrooms and 528 pupils. The sample covers different types of schools representing a variety of socioeconomic classes or profiles, both urban and rural. Further detail can be found in a related publication (Pflaumer et al., 2021). We analyse the 251 pupils that engaged with the game for at least four weeks. Out of these, 102 have taken a written pre- and post-test. Table 2 lists the number of pupils by their school year with average number of sessions played. The numbers are not equally distributed but are based on the teachers that are willing to participate in the intervention and the parents that signed the ethics agreement to allow the data collection.

Table 2–Listing of participating pupils by school year and the average number of playing sessions.

Grade

Players Total

average Events

Players

> 4 weeks

average Events

Players

 + pre- and post-test

average Events

1

79

14

34

25

12

30

2

232

14

106

25

25

22

3

135

11

58

18

59

16

4

82

8

17

18

6

11

3.3. Data Collection

(a) Navigo Game within iRead

13Android tablets were distributed to classrooms in several grades where teachers adapted the use of the literacy games to the classroom activities. Pupils were asked to play Navigo regularly during school. In order to ensure that our chosen game was played sufficiently, the adaptive algorithm was regularly overwritten by pushing our game into the waiting list on a regular basis for all players. Thus, we were able to obtain a significant amount of data for "Watch your Step" for the "Vowel Length" exercise. The back-end system of the games collected all data as pupils were logged in with their accounts under a pseudonym. Data collected included session ID, date, time played as well as time taken for each step and correctness. This data was available to the researchers via an administrative interface.

(b) Word Riddles

14Two different but comparable tests were administered before and after the intervention and the pupils' spelling performances are compared. The resulting writings were transcribed and analysed with respect to two types of orthographic patterns: (1) The ability to spell the long "ie" correctly and (2) the ability to spell the other long and short vowels correctly, by adding the double consonants after short vowels. The tests were mostly written in a school setting. Teachers were asked not to pronounce the words when asked for help. Pupils were allowed to take their own time to finish the tests but did not spend more than 30 minutes on them. The pupils were allowed to skip words they did not know how to fill. Some noise in these tests was expected in case the teacher or parents helped with the spelling even though they were asked not to interfere. Not all pupils were administered all tests. Sometimes, pupils were sick on the day the test was administered. Covid school closings resulted in missing data for tests. 102 pupils had enough session data and two valid tests to be considered in the study of transfer skills.

15Three parameters were extracted from the written tests:

  • the number of correct achieved writings of a pattern;

  • the number of total target occurrence of a pattern;

  • % of correct writings of a pattern was computed from above.

16The target occurrence depended on the number of words completed in a test and the target words attempted by the pupil. This might differ as words chosen by the pupil may diverge from the designated word while still exhibiting the target pattern. The two patterns extracted from the tests were the correct spelling of

  • ie: grapheme <ie>, for example <ie> in Wiese (meadow) and

  • KV: double consonants to mark a short vowel, for example <mm> in Hammer (hammer)

17The tests were handwritten and collected and matched by pseudonym to the game player names. They were transcribed by hand according to target word and actual spelling and marked for the above patterns as correct or incorrect.

3.4. Modeling "Learning" in Learning Curves

18As discussed in Section 1.3 regarding previous work on learning curves, it is still unclear how to model "learning" as a function of time. The goal of this paper is to explore the collected data and parameters in order to gain a deeper understanding of how to display "learning" on the y-axis over time. In other words, in order to investigate the learning curves of pupils more closely, we need a set of features that allows for a detailed analysis of commonalities and differences among pupils. We need values and features that allow us to model the performance of pupils before and after playing as well as different ways of learning.
From the provided log data from Navigo we could derive the following basic parameters for this study.

19Data regarding the frequency of usage:

  • "Weeks": the number of weeks the game was played;

  • "Event": a single task in the game, ie one multiple-choice question;

  • "Sessions": the number of times the pupil crossed a chasm;

  • "Events per Session": the number of steps or events that it takes to get across the chasm.

20Data regarding correctness and speed of events:

  • "Correct": number of correct answers;

  • "Incorrect": number of errors;

  • "Duration" or "lack of speed": elapsed time for an event.

21The duration, meaning the elapsed time a pupil needs for a particular task, is an indicator of his or her mastery of the task. In general, the faster the performance, the more certain a pupil is about an answer, or the more automated his answer is. "Automatisation" means that the brain has fully acquired a skill and does not have to think about it anymore. Additionally, we also calculate error percentages for each week to cope with the problem that some pupils might simply be hitting buttons without concern for correctness. To express the learning curves, we take into account the following parameters:

  • duration, ie the elapsed time for each event;

  • duration average, ie the average time per event;

  • number of events played;

  • error percentage, ie the number of errors per number of events;

  • duration median, ie the median time per event;

  • duration variation in duration, ie difference in elapsed time for each event.

22We calculated the values for each of these parameters across all data that is accumulated for a given week that a pupil played the game. The results were then displayed as learning curves over time.

Figure 3–The subsumed learning curve for an average duration of the different grades.

Figure 3–The subsumed learning curve for an average duration of the different grades.

23Figure 3 shows a subsumed learning curve for the average duration of the different grades. The grey area indicates the confidence interval of the curve. As the learning curve in this case is subsumed from a variety of different values, the confidence interval indicates how well the learning curve fits the data. The learning curve of the first-grade pupils has the best fit. We can observe a learning peak at week 12 and a trough at week 15 and another peak towards the last week of play. Peaks indicate learning and that the task is not yet mastered as more time is taken to answer. First-grade pupils seem to have two larger learning peaks, indicating that the task has not yet been fully mastered. However, the curve subsumes a number of pupils and not all of them played the full range of weeks. Thus, we can only make a general statement. The other learning curves fit each other less. Third graders show development with several learning peaks. Fourth-grade pupils have a declining curve with a learning peak in week 4. Although the overall development indicates an improvement, there is a slight increase in slope at the right end, which might indicate another learning peak. Second-grade pupils also indicate an improvement, showing a decline in the learning curve. However, the learning curves of all three grades exhibit large confidence intervals at the end, making it difficult to make a valid statement for all students as a group. In order to get a better insight, we also looked at the individual learning curves of pupils. The goal is to further parameterise these learning curves in such a way as to understand learning stages and perhaps group learners according to their learning stage. Learning and the mastery of a task cannot be visualised by a single parameter. The graph yields insight only in combination with other parameters. Figure 4 shows an example of the learning curves for the parameters described above for one of the pupils. The following elements are depicted (from top left to bottom right): duration distributions for answers by week separated into correct and incorrect answers, average smoothed duration curve, number of events per week, error percentage of events played by week, variance of duration, and median duration per week. Inspection showed that these curves seem to be reasonable representations granting numeric representations of the observed learning curve.

Figure 4–Prototypical examples of learning curves and their representations.

Figure 4–Prototypical examples of learning curves and their representations.

24Looking at the first diagrams in Figure 4, we noticed that the variance of the duration indicates a cognitive event which depicts mastery in correctness and in duration. Diagrams 4 to 6 (error percentage, variance of duration, and duration median) break the distributional duration graph into its individual components. In this example, we can observe that the single components all show a similar trend. After a steep learning peak, the learning curve gradually declines, exhibiting some smaller peaks. Overall, we can see a large decline with a final low indicating an improvement in all observed parameters. Thus, we can assume a final mastery for this pupil. Using log data for Navigo, we are able to observe an oscillatory behaviour in the learning curves of pupils that dampens and converges over time as the player automates the learning goal. Learning breakthroughs seem to occur at the moment or shortly after a player takes a significantly longer time span for incorrect answers. This longer time span seems to mark important milestones in the acquisition process and may well be reflecting the cognitive restructuring we mentioned in the introduction.

3.5. Classification of Learning Curves

25While the individual learning curve gives us insight into the learning of an individual pupil, it is difficult and time-consuming to compare pupils and to identify differences and commonalities among them. For automatisation, we need a vector of features for each pupil that we can then feed into a classifier to identify common "learning" groups with comparable behaviour among these. We model the development of pupils, using the values of the first and last week the pupil played and calculating their difference for each of these parameters. Begin and end reflect the performance of pupils before and after playing. The difference depicts the development of pupils. A negative development indicates an increase in performance with a decline in error percentage, median duration, and variation in duration. The resulting vector contains the following nine features:

  • "varnorm_begin", "varnorm_end", "varnorm_diff"

  • "ernorm_begin", "ernorm_end", "ernorm_diff"

  • "medinorm_begin", "medinorm_end" , "medinorm_diff"

26We are also interested in the way the pupils develop, ie in the oscillations of the learning curves. We calculate the number of peaks as well as the average and maximum height of peaks per pupil for each of the three parameters (error percentage, median, and variance of duration). The result are the following nine features:

  • "ernorm_nopeaks", "ernorm_mean", "ernorm_max";

  • "medinorm_nopeaks", "medinorm_mean", "medinorm_max";

  • "varnorm_nopeaks", "varnorm_mean", "varnorm_max".

27Based on the above parameters, feature vectors are created for each of the 251 players. These are then used to cluster pupils with the overall goal of finding common learning patterns. For clustering, we experimented with different feature sets and two different clustering methods, the k-means algorithm and the PAM (partitioning around medoids) algorithm. For the final feature set (as described above) the PAM algorithm provided the best results. The PAM algorithm is a robust alternative to k-means clustering. Compared to k-means, it is less sensitive to noise and outliers using medoids as cluster centres instead of means. A medoid can be defined as the point in the cluster whose dissimilarities with all the other points in the cluster are minimal. The algorithm was proposed by Kaufman and Rousseeuw (1990) and maps a distance matrix into a given number of clusters. Based on the given feature vector, it tries to find the given number of medoids so that the dissimilarity within all clusters is the smallest.

4. Results

28Results of the above data analysis can be interpreted through the clusters that were obtained.

4.1. Clustering

29Before clustering, we need to determine the number of clusters that we want to obtain. We used the so-called elbow method to find the optimal number of clusters. We computed clustering algorithms for k from 1 to 10 clusters and calculated the total within-cluster sum of square (wss). This algorithm resulted in k=5 clusters as the optimal number of clusters. Figure 5 shows the resulting clustering trained and evaluated on the participants who played for at least 4 weeks. The depicted names are pseudonyms.

Figure 5–Cluster visualisation along the first two dimensions.

Figure 5–Cluster visualisation along the first two dimensions.

4.2. Characteristics of Clusters

30Examining the learning curves within the resulting clusters, we can make several observations. Even though all clusters show a mix of performance subgroups, they do seem to have certain characteristics that are describable. In particular, clusters 1 and 3 seem to describe the behaviour of the top and improving subgroups. Figure 6 depicts their characteristics.

31Cluster 1 represents mostly the pupils that improved while playing and secondly those that already did well on the written tests. It differs most significantly with respect to the other clusters in the parameters describing the path between the beginning and the end. The parameters show a larger variance of time spent to answer both in mean, median, and max values, despite a better performance in the beginning than the other clusters. Even the number of errors committed is much larger than in the other clusters. These values even increase with play. Amala is a good representative of this cluster. The feature values are visualised in Figure 6.

32Cluster 3 represents mostly the patterns for top pupils, that is those that performed well in both the written pre- and post-tests. Cluster 3 differs most significantly in the parameters describing mostly the path when compared to the other clusters. However, there are parameters referring to the oscillation that indicate top performers. The variance in the number of peaks in the learning curve is significantly lower. The duration is higher while the errors committed are also higher. These pupils also had a higher number of steps per session. A good example for cluster 3 is shown in Figure 7.

33While clusters 2, 4, and 5 did not have clear profiles with respect to the written tests, they do exhibit common patterns of learning curves. Cluster 2 has a large oscillation in the number of errors committed as well as variance and median number of peaks. These indicate non-converging learning curves that become apparent both in the time spent to answer and in the number of errors committed. They also have a lower number of events and fewer errors committed than other clusters overall and are slower towards the end than others. This can also describe the interaction for top and improving pupils as well as those making no progress. Figure 8 shows an example of a pupil in cluster 2.

34Cluster 4 parameters focus on the large improvement in duration and errors committed, as well as on a larger variance in duration at the beginning than others, with a smaller variance than others towards the end. Both duration and errors committed are higher among these pupils. A significant number of top and improved pupils fit this category. Figure 9 shows an example of this. Cluster 5 parameters describe another type of learner. Many of the pupils that do not improve are in this category. The number of errors committed is very small. Parameters reflect that there are no real learning curves with much fewer peaks than the other groups. These pupils were less exposed to making mistakes and played less as well. Figure 10 depicts an example from cluster 5.

Figure 6–Amala (Cluster 1, Grade 2), improved (both).

Figure 6–Amala (Cluster 1, Grade 2), improved (both).

Figure 7–Pille (Cluster 3, Grade 3), top (both).

Figure 7–Pille (Cluster 3, Grade 3), top (both).

Figure 8–Cos (Cluster 2, Grade 2), top (“ie”) none (“KV”).

Figure 8–Cos (Cluster 2, Grade 2), top (“ie”) none (“KV”).

Figure 9–Mona (Cluster 4, Grade 4), improved (both).

Figure 9–Mona (Cluster 4, Grade 4), improved (both).

Figure 10–Delfin (Cluster 5, Grade 3) improves (“ie”) no change (“KV”).

Figure 10–Delfin (Cluster 5, Grade 3) improves (“ie”) no change (“KV”).

4.3 Clusters as a Function of Pre-test and Post-test Results

35We distinguish four different groups based on the tests. These are

  • "Improved" (start with low performance and end with significantly improved performance on the written test);

  • "Top" (start and end with good performance on the written test);

  • "None" (no change in low performance on the written test);

  • "Worse" (a significant change from high to low performance on the written test).

36Figures 11 and 12 show the distribution of the above performance groups by grade. It seems that learning is already effective in Grade 2 and even in Grade 1 and that a heavy learning curve leading to worse performance before improvement could be avoided by training before entering Grade 3.

Figure 11–Distribution of performance groups based on pre- and post-tests for players by grade (ie).

Figure 11–Distribution of performance groups based on pre- and post-tests for players by grade (ie).

Figure 12–Distribution of performance groups based on pre- and post-tests for players by grade (short vowel).

Figure 12–Distribution of performance groups based on pre- and post-tests for players by grade (short vowel).

37Figure 13 shows the distribution of the pupils by grade into the clusters.

Figure 13–Cluster by grade distribution.

Figure 13–Cluster by grade distribution.

Figure 14–Distribution by performance on written tests by cluster.

Figure 14–Distribution by performance on written tests by cluster.

38Figure 14 depicts how the four groups of pupils (improve, none, top, worse) fall into the resulting five clusters. While all clusters have a mix of performers, Clusters 1 and 3 seem to capture mostly the group "improve" and "top" when looking at both "KV" (short vowel) tasks (identifying any long vowel against short vowels) and 'ìe" (identifying the long vowel "ie" against short vowels).

5. Discussion and Conclusion

39When inspecting log data for Navigo, we observe an oscillatory behaviour in the learning curves of pupils that dampens and converges over time as the player automates the task. Our goal was to model and parameterise these learning curves in order to detect common patterns and mastery of the material. Learning breakthroughs seem to occur at the moment or shortly after a player takes significantly longer to complete the task while registering incorrect answers. This longer time span seems to mark an important milestone in the acquisition process. This finding agrees with the literature; practice must include mistakes. Time factors and mistakes both indicate cognitive restructuring and form a major part of what identifies the successful writers in Cluster 3. At the same time, the individual learning curves posed some difficulty in interpreting their meaning or in grouping pupils. Through inspection, parameterisation, and non-linear clustering methods, we were able to answer the proposed research questions as follows.

40RQ1: How can learning curves be characterised? By clustering the pupils into five groups that could be meaningfully described, we could show different patterns in playing and learning. It could also be seen that some patterns are specific to pupils that have better writing skills and those that are improving their writing skills through gameplay.

41RQ2: Are there commonalities or individual classifiable differences? Clustering using the parameters that were derived through data inspection led to consistent groups with similar learning curves that can be described as specific and different from other groups.

42RQ3: Is it possible to recognise and analyse outliers? At this point, we were not able to recognise and analyse outliers. Due to the many different kinds of learning curves, it is possible to be a good writer and have several learning curve patterns. The inverse is also true that a specific learning curve does not lead to a guaranteed classification of writing proficiency or grade.

43We were able to identify some patterns that are common to subgroups of pupils. These patterns do not always predict their performance in a writing test. However, there are some patterns that seem to correlate mostly with top performers or improving pupils. It is not possible to find all pupils of a subgroup (performance in writing or grade) given a particular pattern in the learning curve. This may be due to many factors. First, the tests do not always accurately reflect a pupil's knowledge. Second, learning curves are highly individual. In addition, some pupils might have played badly on purpose or were unmotivated or had a sibling or friend play for them. Overall, however, the clusters reflect the patterns that were noticed when observing the data and allowed us to compute the interaction between the parameters in a non-linear manner in order to consider complex dependencies between the parameters to identify similar learning patterns.
To summarise, understanding learning curves is very complex and does not necessarily allow us to detect when learning is finalised. More research is needed in order to understand the various parameters. Much more data and probably more parameters will be needed in order to detect mastery. However, we showed a number of different learning curve patterns that seem to be common across a subset of players.
Future work will include further data collection as well as looking in more detail at external factors that influence the learning curves, such as prior knowledge, grade, learning disability, and (ir)regular practice.

Haut de page

Bibliographie

Acquah, E. O., & Katz, H. T. (2020). Digital game-based L2 learning outcomes for primary through high-school students: A systematic literature review. Computers & Education, 143, 103667. http://www.elib.sdnu.edu.cn/upload_files/file/20201124/1_20201124095608.pdf

Anderson, R. C., Stevens, K. C., Shifrin, Z., & Osborn, J. H. (1978). Instantiation of Word Meanings in Children. Journal of Reading Behavior, 10(2), 149-157.

Berkling, K. (2016). Item presentation in primers–An analysis based on acquisition research. Bochumer Linguistische Arbeitsberichte, 50.

Berkling, K. (2017). Phontasia: A phonics game for German and its effect on orthographic skills–first corpus explorations. In 6th Workshop on Child Computer Interaction (WOCCI 2017) (p. 7-16). ISCA.

Berkling, K., Faller, H., & Piertzik, M. (2017). Avoiding Failure in Modern Game Design with Academic Content–A Recipe, an Anti-Pattern and Applications Thereof. In Proceedings of the 9th International Conference on Computer Supported Education (p. 25-36). SCITEPRESS - Science and Technology Publications.

Berkling, K., & Franken, B. (2019). Transfer of Educational Skills from Games to Classroom tasks–A Case Study using iRead to Improve Reading and Writing. In International Symposium on Gamification and Games for Learning, Barcelona, Spain.

Berkling, K., & Kermes, H. (2020). Learning to take the next step: Knowledge acquisition of orthographic patterns in serious games with immediate feedback. In L. Gómez Chova, A. López Martínez, & I. Candel Torres (Eds.), EDULEARN Proceedings, EDULEARN20 Proceedings (p. 4883-4891). IATED.

Berkling, K., & Reichel, U. (2016). Wortstruktur, Orthographie und Didaktik: Die Relevanz der Vokallänge. In B. Mesch & C. Noack (Eds.), System, Norm und Gebrauch – Drei Seiten derselben Medaille? Orthographische Kompetenz und Performanz im Spannungsfeld zwischen System, Norm und Empirie (p. 200-228). WBV MEDIA.

Bhattacharya, A., & Ehri, L. C. (2004). Graphosyllabic analysis helps adolescent struggling readers read and spell words. Journal of Learning Disabilities, 37(4), 331-348.

Booij, G. (1999). The Role of the Prosodic Word in Phonotactic Generalizations. In T. A. Hall & U. Kleinhenz (Eds.), Current Issues in Linguistic Theory. Studies on the Phonological Word (Vol. 174, p. 47). John Benjamins Publishing Company.

Caravolas, M., & Volín, J. (2001). Phonological spelling errors among dyslexic children learning a transparent orthography: The case of Czech. Dyslexia (Chichester, England), 7(4), 229-245.

Caravolas, M., Hulme, C., & Snowling, M. J. (2001). The Foundations of Spelling Ability: Evidence from a 3-Year Longitudinal Study. Journal of Memory and Language, 45(4), 751-774.

Carlucci, L., & Case, J. (2013). On the necessity of U-shaped learning. Topics in Cognitive Science, 5(1), 56-88.

Castles, A., & Nation, K. (2006). How does orthographic learning happen? In S. Andrews (Ed.), From Inkmarks to Ideas: Current Issues in Lexical Processing (p. 151-179). Taylor & Francis.

Cunningham, A. E. (2006). Accounting for children's orthographic learning while reading text: Do children self-teach? Journal of Experimental Child Psychology, 95(1), 56-77.

Daller, M, Turlik, J., & Weir, I. (2013). Vocabulary acquisition and the learning curve: Vocabulary knowledge: Human ratings and automated measures. In S. Jarvis (Ed.), Studies in Bilingualism: v.47. Vocabulary Knowledge: Human ratings and automated measures (p. 185-217). John Benjamins Publishing Company.

Duncan, L. G., Colé, P., Seymour, P. H. K., & Magnan, A. (2006). Differing sequences of metaphonological development in French and English. Journal of Child Language, 33(2), 369-399.

Ebbinghaus, H. (1964). Memory: A contribution to experimental psychology. Dover Publications.

Ehri, L. C. (1987). Learning to Read and Spell Words. Journal of Reading Behavior, 19(1), 5-31.

Elaachak, L. (2020). Towards a New Platform Based on Learning Outcomes Analysis For Mobile Serious Games. International Journal of Emerging Technologies in Learning (IJET), 15(02), 42.

Erhel, S., & Jamet, E. (2013). Digital game-based learning: Impact of instructions and feedback on motivation and learning effectiveness. Computers & Education, 67, 156-167.

Glock, C. H., Grosse, E. H., Jaber, M. Y., & Smunt, T. L. (2019). Applications of learning curves in production and operations management: A systematic literature review. Computers & Industrial Engineering, 131, 422-441.

Greipl, S., Klein, E., Lindstedt, A., Kiili, K., Moeller, K., Karnath, H.‑O., Bahnmueller, J., Bloechle, J., & Ninaus, M. (2021). When the brain comes into play: Neurofunctional correlates of emotions and reward in game-based learning. Computers in Human Behavior, 125, 106946.

Holopainen, L., Koch, A., Hakkarainen, A., & Kofler, D. (2020). Predictors of Reading Skills at the First and Second Grade: The Role of Orthography. Reading Psychology, 41(5), 461-484.

Ise, E., Arnoldi, C. J., & Schulte-Körne, G. (2014). Development of orthographic knowledge in German-speaking children: a 2-year longitudinal study. Journal of Research in Reading, 37(3), 233-249.

Kaufman, L. & Rousseeuw, P. J. (1990). Finding groups in data: An introduction to cluster analysis. Wiley. https://onlinelibrary.wiley.com/doi/book/10.1002/9780470316801

Khan, A. Q., Khan, S., & Safaev, U. (2020). Serious Games and Gamification: A Systematic Literature Review.

Mann, V., & Wimmer, H. (2002). Phoneme awareness and pathways into literacy: A comparison of German and American children. Reading and Writing, 15(7/8), 653-682.

Martinet, C., Valdois, S., & Fayol, M. (2004). Lexical orthographic knowledge develops from the beginning of literacy acquisition. Cognition, 91(2), B11-B22.

Mastropavlou, M., Zakopoulou, V., Christou, T., Nerantzini, M., & Lekakou, M. (2020). The Design of New Digital Resources of Personalized Support of Reading Skills for Novice Readers and Children with Dyslexia: A Presentation of the iRead Software Linguistic Domain Models. In Conference on Software Development and Technologies for Enhancing Accessibility and Fighting Info-exclusion (p. 241-245). ACM.

Melby-Lervåg, M., & Hulme, C. (2010). Serial and free recall in children can be improved by training: Evidence for the importance of phonological and semantic representations in immediate memory tasks. Psychological Science, 21(11), 1694-1700.

Ouellette, G., & Sénéchal, M. (2008). Pathways to literacy: A study of invented spelling and its role in learning to read. Child Development, 79(4), 899-913.

Pacton, S., Perruchet, P., Fayol, M., & Cleeremans, A. (2001). Implicit learning out of the lab: The case of orthographic regularities. Journal of Experimental Psychology: General, 130(3), 401-426. https://psycnet.apa.org/record/2001-18060-004

Pattemore, M., Gilabert Guerrero, R., & Serra, J. K. (2019). Elaborative Feedback in L2 Reading Games. In International Symposium on Gamification and Games for Learning, Barcelona, Spain.

Pavlik, P., Eglington, L., & Harrell-Williams, L. (2021a). Logistic Knowledge Tracing: A Constrained Framework for Learner Modeling. IEEE Transactions on Learning Technologies, 14(5), 624-639.

Pavlik, P., Eglington, L., & Zhang, L. (2021b). Automatic Domain Model Creation and Improvement. In Sharon I-Han Hsiao, Shaghayegh (Sherry) Sahebi, François Bouchet, & Jill-Jênn Vie (Eds.), Proceedings of the 14th International Conference on Educational Data Mining, EDM 2021, virtual, June 29-July 2, 2021. International Educational Data Mining Society. https://educationaldatamining.org/EDM2021/virtual/static/pdf/EDM21_paper_208.pdf

Perfetti, C. A., & Hart, L. (2002). The lexical quality hypothesis. Precursors of Functional Literacy, 11, 67-86.

Pflaumer, N., Knorr, N., & Berkling, K. (2021). Appropriation of adaptive literacy games into the German elementary school classroom. British Journal of Educational Technology, 52(5), 1917-1934.

Rehm, G., Piperidis, S., Bontcheva, K., Hajic, J., Arranz, V., Vasiļjevs, A., Backfried, G., Gomez-Perez, J. M., Germann, U., Calizzano, R., Feldhus, N., Hegele, S., Kintzel, F., Marheinecke, K., Moreno-Schneider, J., Galanis, D., Labropoulou, P., Deligiannis, M., Gkirtzou, K., … Renals, S. (2021). European Language Grid: A Joint Platform for the European Language Technology Community. Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics: System Demonstrations, 221-230.

Schroeder, S., Würzner, K.‑M., Heister, J., Geyken, A., & Kliegl, R. (2015). Childlex: A lexical database of German read by children. Behavior Research Methods, 47(4), 1085-1094.

Share, D. L. (1995). Phonological recoding and self-teaching: Sine qua non of reading acquisition. Cognition, 55(2), 151-218; discussion 219-26.

Stanat, P., Schipolowski, S., Rjosk, C., Weirich, S., & Haag, N. (Eds.). (2017). IQB-Bildungstrend: Vol. 2016. Kompetenzen in den Fächern Deutsch und Mathematik am Ende der 4. Jahrgangsstufe im zweiten Ländervergleich [1. Auflage]. Waxmann. https://elibrary.utb.de/doi/book/10.31244/9783830987307

Stemberger, J. P., & Bernhardt, B. H., & Johnson, C. E. (1999). "Regressions" ("u"-shaped learning) in the acquisition of prosodic structure. Poster presented at the 6th International Child Language Congress, July 1999. http://roa.rutgers.edu/files/471-1101/471-1101-STEMBERGER-0-0.PDF

Swift, E. J. (1903). Studies in the Psychology and Physiology of Learning. The American Journal of Psychology, 14(2), 201.

Tessier, A.‑M. (2019). U-shaped development in error-driven child phonology. Wiley Interdisciplinary Reviews. Cognitive Science, 10(6), e1505.

Viering, T. J., Mey, A., & Loog, M. (2020). Making Learners (More) Monotone. In M. R. Berthold, A. Feelders, & G. Krempl (Eds.), Lecture Notes in Computer Science. Advances in Intelligent Data Analysis XVIII (Vol. 12080, p. 535-547). Springer International Publishing.

Waltersbacher, A. (2018, December 18). Sprach- und Bewegungstherapien bei Schulanfängern leicht rückläufig. WIdO – Wissenschaftliches Institut der AOK. https://www.wido.de/news-presse/pressemitteilungen/heilmittelbericht-2018/

Wiese, R. (2001). How prosody shapes German words and morphemes. Interdisciplinary Journal for Germanic Linguistics and Semiotic Analysis, 6, 155–184.

Ziegler, J. C., Bertrand, D., Tóth, D., Csépe, V., Reis, A., Faísca, L., Saine, N., Lyytinen, H., Vaessen, A., & Blomert, L. (2010). Orthographic depth and its impact on universal predictors of reading: A cross-language investigation. Psychological Science, 21(4), 551-559.

Ziegler, J. C., & Goswami, U. (2005). Reading acquisition, developmental dyslexia, and skilled reading across languages: A psycholinguistic grain size theory. Psychological Bulletin, 131(1), 3-29.

Ziegler, J. C., Perry, C., & Zorzi, M. (2014). Modelling reading development through phonological decoding and self-teaching: Implications for dyslexia. Philosophical Transactions of the Royal Society of London. Series B, Biological Sciences, 369(1634), 20120397.

Haut de page

Notes

1 iread-project.eu

2 The data is available under open license at https://live.european-language-grid.eu/catalogue/project/1980 (Rehm, et al., 2021).

3 Trochee are 2-syllable words where the first syllable is stressed and the second syllable usually contains a @-like reduction vowel represented by the grapheme <e>. This word-prosody is the most prevalent form in German language (Wiese, 2001; Booij, 1999) and impacts the most important phoneme-grapheme correspondences that make up the orthographic depth of German. Orthographic depth has a critical impact on reading ability (Ziegler et al., 2010; Holopainen et al., 2020; Ise et al, 2014)

Haut de page

Table des illustrations

Titre Figure 1–Navigo App example games.
URL http://journals.openedition.org/alsic/docannexe/image/6097/img-1.png
Fichier image/png, 265k
Titre Figure 2–Example of elicitation for orthographic skills from pupils.
URL http://journals.openedition.org/alsic/docannexe/image/6097/img-2.png
Fichier image/png, 34k
Titre Figure 3–The subsumed learning curve for an average duration of the different grades.
URL http://journals.openedition.org/alsic/docannexe/image/6097/img-3.jpg
Fichier image/jpeg, 84k
Titre Figure 4–Prototypical examples of learning curves and their representations.
URL http://journals.openedition.org/alsic/docannexe/image/6097/img-4.png
Fichier image/png, 50k
Titre Figure 5–Cluster visualisation along the first two dimensions.
URL http://journals.openedition.org/alsic/docannexe/image/6097/img-5.png
Fichier image/png, 431k
Titre Figure 6–Amala (Cluster 1, Grade 2), improved (both).
URL http://journals.openedition.org/alsic/docannexe/image/6097/img-6.png
Fichier image/png, 84k
Titre Figure 7–Pille (Cluster 3, Grade 3), top (both).
URL http://journals.openedition.org/alsic/docannexe/image/6097/img-7.png
Fichier image/png, 108k
Titre Figure 8–Cos (Cluster 2, Grade 2), top (“ie”) none (“KV”).
URL http://journals.openedition.org/alsic/docannexe/image/6097/img-8.png
Fichier image/png, 138k
Titre Figure 9–Mona (Cluster 4, Grade 4), improved (both).
URL http://journals.openedition.org/alsic/docannexe/image/6097/img-9.png
Fichier image/png, 117k
Titre Figure 10–Delfin (Cluster 5, Grade 3) improves (“ie”) no change (“KV”).
URL http://journals.openedition.org/alsic/docannexe/image/6097/img-10.png
Fichier image/png, 95k
Titre Figure 11–Distribution of performance groups based on pre- and post-tests for players by grade (ie).
URL http://journals.openedition.org/alsic/docannexe/image/6097/img-11.png
Fichier image/png, 5,4k
Titre Figure 12–Distribution of performance groups based on pre- and post-tests for players by grade (short vowel).
URL http://journals.openedition.org/alsic/docannexe/image/6097/img-12.png
Fichier image/png, 5,8k
Titre Figure 13–Cluster by grade distribution.
URL http://journals.openedition.org/alsic/docannexe/image/6097/img-13.png
Fichier image/png, 31k
Titre Figure 14–Distribution by performance on written tests by cluster.
URL http://journals.openedition.org/alsic/docannexe/image/6097/img-14.png
Fichier image/png, 28k
Haut de page

Pour citer cet article

Référence électronique

Hannah Kermes et Kay Berkling, « An Attempt at Describing Learning Curves for a Complex Sub-Task in Reading Acquisition »Alsic [En ligne], Vol. 25, n° 1 | 2022, mis en ligne le 01 juillet 2022, consulté le 14 août 2022. URL : http://journals.openedition.org/alsic/6097 ; DOI : https://doi.org/10.4000/alsic.6097

Haut de page

Auteurs

Hannah Kermes

Hannah Kermes is a project leader and researcher at Communication Lab. She develops concepts for text analysis with regard to readability and comprehensibility. She studied German and English linguistics and received her PhD in Computational Linguistics. She was a principal investigator in the Collaborative Research Center "Information density and linguistic encoding" and a substitute professor for linguistics at the University of Dortmund. Her expertise ranges from theoretical syntax, grammar writing, corpus compilation, (statistical) corpus analysis to data mining.
Affiliation: Communication Lab.
Email: hmtke@gmx.de
Address: CS Department, DHBW Karlsruhe, Erzbergerstraße 121, 76331 Karlsruhe, Germany.

Kay Berkling

Kay Berkling is a professor at the Cooperative State University in Karlsruhe (Germany). She teaches Software Engineering, Gamification and Games. Her research focuses on motivation of learning as well as orthography acquisition. Writing relates closely to reading, and both skills exhibit learning curves during acquisition, which are the focus of her studies. She received her PhD in 1996 in automatic language identification and has since applied speech technology to learning. Gamification and motivation help her to apply learning both to children through games and in adult education.
Affiliation: Cooperative State University Baden-Württemberg.
Email: berkling@dhbw-karlsruhe.de
Web: https://www.karlsruhe.dhbw.de/prof-berkling-phd.html
Address: CS Department, DHBW Karlsruhe, Erzbergerstraße 121, 76331 Karlsruhe, Germany.

Articles du même auteur

Haut de page

Droits d’auteur

CC BY-NC-ND 4.0

Creative Commons - Attribution - Pas d'Utilisation Commerciale - Pas de Modification 4.0 International - CC BY-NC-ND 4.0

https://creativecommons.org/licenses/by-nc-nd/4.0/

Haut de page
  • DOAJ - Directory of Open Access Journals
  • OpenEdition Journals
Rechercher dans OpenEdition Search

Vous allez être redirigé vers OpenEdition Search