iRead is part of a project that has received funding from the European Union's Horizon 2020 research and innovation program under grant agreement No. 731724. The results presented reflect only the authors' view and the Agency is not responsible for any use that may be made of the information it contains. The authors have no conflict of interest with the content of this paper.

1. Introduction

1.1. Current Situation

1Schools in Germany are increasingly struggling to teach reading and writing to pupils and the proportion of children who meet or exceed the standards in the area of orthography has fallen from 65% to 55% according to the national standardised test IQB-Bildungstrend 2016 (Stanat et al., 2017). In 2017, 34.8% of children in their first year of elementary school were diagnosed with developmental disorders, which represents an increase of 27,5% from 2008. 82% of the issues relate to language acquisition, with boys being affected 1.5 more often than girls according to WiDO AOK (health insurance) (Waltersbacher, 2018). Given these developments, it is especially important to provide high-quality learning materials, personalised support and adequate amounts of training materials. Literacy games are able to provide this, and when combined with learning analytics, can offer insights into each pupil's learning. In order to achieve this goal, it is important to understand and interpret these data.

1.2. Language Acquisition Theory with a Focus on Reading and Writing

2Children are very sensitive to the orthographic regularities in their writing system. It has been shown that these regularities must be present in learning materials for early literacy education (Ouellette & Sénéchal, 2008; Pacton et al., 2001). Self-teaching theory (Share, 1995) states that children are able to establish specific orthographic knowledge on their own with sufficient reading experience (Caravolas et al., 2001; Caravolas & Volín, 2001; Martinet et al., 2004; Ziegler et al., 2014; Cunningham, 2006). However, this experience requires a combination of regularities and well-chosen "self-teaching" opportunities. Through correctly sequenced input, cognitive pressure to restructure internal representation results in improved lexical representation and the ability to generalise (Perfetti & Hart, 2002). By providing a common pattern with varying graphemes and phonemes in various combinations, the learner trains phonemic awareness through a mixed blending of grapheme-phoneme correspondences. (For example, Hase (rabbit), Wade, (calf), Rose (rose), Rede (speech), and Tube (tube) are examples of consonant-vowel sequences in German that follow the same prosodic pattern for long vowels with different combinations of consonants.) This process of forming internal representations of patterns requires large amounts of training, including unfamiliar words in familiar patterns (Melby-Lervåg & Hulme, 2010). The controlled input improves the inner lexicon (Ziegler & Goswami, 2005; Mann & Wimmer, 2002; Duncan et al., 2006). Mastering orthographic conventions in turn influences both reading fluency and writing ability (Ehri, 1987; Cunningham, 2006; Castles & Nation, 2006; Pacton et al., 2001; Anderson et al., 1978; Bhattacharya & Ehri, 2004). In summary, the literature states that pupils require a large amount of input material that is oriented along familiar patterns with unfamiliar words. Carefully sequenced training material should have an equal effect on reading and writing ability. The method of "Phonics" for English follows these principles. Literacy games are able to provide large amounts of training material that is sequenced according to linguistic rules describing difficulty. In contrast to teachers or static books, games can acquire their content from large databases to present as many combinations of input material as the language has to offer to the learners until they reach proficiency according to their own learning speed, all the while hiding the learning material behind motivating game-like adventures. The automation of games provides personalisation, direct feedback, and results in individual learning trajectories that are called "learning curves" in the literature.

1.3. Previous Research on Learning Curves

3The study of "learning curves" has a long history and was traditionally used for predicting industry productivity (Glock et al., 2019), but originates from terminology used in psychology based on Ebbinghaus's original work on the human mind (Ebbinghaus 1964). The "learning curve" describes the process of learning, typically by plotting time vs learning, using "whichever parameter to represent it" (Swift, 1903). Language learning as well has a long history of being modelled (Case, 2016) and is very often modelled as log functions with an asymptotic approach towards mastery. The authors agree that "learning curves" yield valuable insight but there is no agreement on how they can be modelled (Daller et al., 2013). In education, "learning curves" are employed as a model of pupils (Pavlik et al., 2021a) in order to optimally sequence the input material (Pavlik et al., 2021b). Learning is known to exhibit u-shaped phenomena, hence the technical term "curve" will be used throughout this paper. In language learning, this curve is believed to reflect a vacillation between mental grammatical models. Error-driven events lead to cognitive constraint-rerankings that cause learners' mistakes on previously correct items. The complexity of these learning processes are nicely explained in a case study by Stemberger et al. (1999) and explored by Tessier (2019). Worthwhile contributions to machine learning have shown a theoretical basis for the necessity of such u-shaped learning curves (Carlucci & Case, 2013) and their possible optimisations (Viering et al., 2020) that describe the learners' (or machine's) ability to generalise given the training set. The data for learning curves during gaming are automatically gathered for a large population of players. They contain valuable information regarding player background, game content, sequences, regularity of play, speed, and correctness and can yield enough information to add significant insight to the above research and potentially impact what is known about both machine and human learning. This paper explores the use of these data in order to learn more about the "y-axis" that is to be plotted against time to depict the learning process.

1.4. Literacy Games–Effects

4The effects of literacy games on learning outcomes have been studied in the sciences (Khan et al., 2020) and very often in the field of second language acquisition. Specific interest in games for elementary school-aged children is rare, especially in the German context (Acquah & Katz, 2020). While it is well known that games are very motivating to learners (Erhel & Jamet, 2013), it would also be helpful to know more about the effectiveness of games, which is a complex topic mixing emotions, cognition, input data design, and immediate feedback (Pattemore et al., 2019; Greipl et al., 2021; Elaachak, 2020). Very little is known about literacy games for elementary school-aged children regarding orthography acquisition and reading and writing effectiveness of the games in German. Nor is there much known about content and sequence with respect to the design constraints elaborated above. However, Berkling et al. describe a method for sequencing language input (Berkling, 2016; Berkling & Reichel, 2016), and some publications touch on the effectiveness of such games (Berkling, 2017; Berkling & Franken, 2019) and on game design for serious content (Berkling et al., 2017). A particular example of learning was published by Berkling and Kermes (2020); it explains the game and learning curves that will be analysed in more detail in this paper.

2. Research Motivation

5Digital literacy games can provide a way to collect learner data over longer periods of time. The game uses linguistically selected content from large databases in order to provide the player with familiar patterns with changing letters. This provides practice on specific cognitively challenging concepts and allows us to carry out a longitudinal study on the resulting learning curves. The various resulting data measurements provide material to determine the parameters that allow us to best capture and depict "learning" as a function of time and visualise a high-level view on individual learning development for the chosen task. We are interested in analysing the learning curves for tasks that are not taught explicitly in school in order to measure the effect of the game on learning. The paper will address the following research questions:

• RQ1: How can learning curves be characterised?

• RQ2: Are there commonalities or individual classifiable differences in "learning curves"?

• RQ3: Is it possible to recognise and analyse outliers?

6Section 3 will describe the applied methodology, including instruments, participants, and data collection and processing. Section 4 will detail the results. Section 5 will end with a discussion of the findings and concluding remarks about remaining research questions to be addressed in future work.

3. Methodology

7This section describes the content for the game chosen for the study as part of Navigo, the participating pupils, and the resulting log data.

3.1. Instruments

(a) Navigo Game within iRead

• Time taken for each answer

• Correctness of answer

9We distinguish between two sets of vowel graphemes (<a>,<e>,<o>,<u>) vs (<i>/<ie>). The former set of vowels (KV) has no visible distinction of length. Their length is determined by the following consonant pattern of two consonant letters ('tt','mm'…). The latter vowel <i> vs <ie> denotes in itself whether it is short <i> or long <ie> while the short vowel is still followed by two consonants. These two subtasks are distinguished and tracked separately. The task is to select the bi-syllabic word with the long middle vowel. Top: Timed race. Bottom: Multiple-choice without time limit for "Watch your Step", showing a player crossing the chasm.

Table 1–Example answers for multiple choices given the task of finding the word with a long vowel.

Answer Choices by Step/Event for One Sample Session
1	(Wiese)	hassen	motzen	lecker	Type: ie
2	(Rasen)	Mutter	rennen	trennen	Type: KV or long vowel
3	netter	Betten	(Ruder)	flotter	Type: KV or long vowel
4	Kletten	Klassen	Flammen	(Strömen)	Type: KV or long vowel
5	Hammel	(dösen)	lassen	Risse	Type: KV or long vowel

(b) Word Riddles

10A written test in the form of a word riddle consisted of 20 items with a very high frequency of trochee-formed words with short and long vowels. Different tests were prepared for each school year using the same patterns but different words to avoid memorisation effects across tests. To avoid the ceiling effect (when the exam is too easy for the test taker), the difficulty of vocabulary was slightly increased from Grade 1 to Grade 4 while maintaining the balance between long and short vowel trochees patterns and word frequencies according to Schroeder et al. (2015), while allowing for the capture of over-generalisations. Each item is presented through a sentence with one missing word and a picture to support the semantics of the sentence in order to help the pupil identify and write the desired word into the sentence on paper. Figure 2 shows an example of one of these items. These word riddles, as they are called for the pupils, represent our pre- and post-test to allow us to track improvement in writing due to the intervention of the game.

11The expected answer in Figure 2 is "Das ist ein Hammer" (This is a hammer). The combination of sentence and image as shown here supports elicitation of the desired target word in almost all cases.

3.2. Participants

12Participants for the iRead project were recruited in the state of Baden-Württemberg, Germany, resulting in a total of nine different elementary schools, 20 classrooms and 528 pupils. The sample covers different types of schools representing a variety of socioeconomic classes or profiles, both urban and rural. Further detail can be found in a related publication (Pflaumer et al., 2021). We analyse the 251 pupils that engaged with the game for at least four weeks. Out of these, 102 have taken a written pre- and post-test. Table 2 lists the number of pupils by their school year with average number of sessions played. The numbers are not equally distributed but are based on the teachers that are willing to participate in the intervention and the parents that signed the ethics agreement to allow the data collection.

Table 2–Listing of participating pupils by school year and the average number of playing sessions.

Grade	Players Total	average Events	Players > 4 weeks	average Events	Players  + pre- and post-test	average Events
1	79	14	34	25	12	30
2	232	14	106	25	25	22
3	135	11	58	18	59	16
4	82	8	17	18	6	11

3.3. Data Collection

(a) Navigo Game within iRead

13Android tablets were distributed to classrooms in several grades where teachers adapted the use of the literacy games to the classroom activities. Pupils were asked to play Navigo regularly during school. In order to ensure that our chosen game was played sufficiently, the adaptive algorithm was regularly overwritten by pushing our game into the waiting list on a regular basis for all players. Thus, we were able to obtain a significant amount of data for "Watch your Step" for the "Vowel Length" exercise. The back-end system of the games collected all data as pupils were logged in with their accounts under a pseudonym. Data collected included session ID, date, time played as well as time taken for each step and correctness. This data was available to the researchers via an administrative interface.

(b) Word Riddles

14Two different but comparable tests were administered before and after the intervention and the pupils' spelling performances are compared. The resulting writings were transcribed and analysed with respect to two types of orthographic patterns: (1) The ability to spell the long "ie" correctly and (2) the ability to spell the other long and short vowels correctly, by adding the double consonants after short vowels. The tests were mostly written in a school setting. Teachers were asked not to pronounce the words when asked for help. Pupils were allowed to take their own time to finish the tests but did not spend more than 30 minutes on them. The pupils were allowed to skip words they did not know how to fill. Some noise in these tests was expected in case the teacher or parents helped with the spelling even though they were asked not to interfere. Not all pupils were administered all tests. Sometimes, pupils were sick on the day the test was administered. Covid school closings resulted in missing data for tests. 102 pupils had enough session data and two valid tests to be considered in the study of transfer skills.

15Three parameters were extracted from the written tests:

• the number of correct achieved writings of a pattern;

• the number of total target occurrence of a pattern;

• % of correct writings of a pattern was computed from above.

16The target occurrence depended on the number of words completed in a test and the target words attempted by the pupil. This might differ as words chosen by the pupil may diverge from the designated word while still exhibiting the target pattern. The two patterns extracted from the tests were the correct spelling of

• ie: grapheme <ie>, for example <ie> in Wiese (meadow) and

• KV: double consonants to mark a short vowel, for example <mm> in Hammer (hammer)

17The tests were handwritten and collected and matched by pseudonym to the game player names. They were transcribed by hand according to target word and actual spelling and marked for the above patterns as correct or incorrect.

3.4. Modeling "Learning" in Learning Curves

18As discussed in Section 1.3 regarding previous work on learning curves, it is still unclear how to model "learning" as a function of time. The goal of this paper is to explore the collected data and parameters in order to gain a deeper understanding of how to display "learning" on the y-axis over time. In other words, in order to investigate the learning curves of pupils more closely, we need a set of features that allows for a detailed analysis of commonalities and differences among pupils. We need values and features that allow us to model the performance of pupils before and after playing as well as different ways of learning.
From the provided log data from Navigo we could derive the following basic parameters for this study.

19Data regarding the frequency of usage:

• "Weeks": the number of weeks the game was played;

• "Event": a single task in the game, ie one multiple-choice question;

• "Sessions": the number of times the pupil crossed a chasm;

• "Events per Session": the number of steps or events that it takes to get across the chasm.

20Data regarding correctness and speed of events:

• "Correct": number of correct answers;

• "Incorrect": number of errors;

• "Duration" or "lack of speed": elapsed time for an event.

21The duration, meaning the elapsed time a pupil needs for a particular task, is an indicator of his or her mastery of the task. In general, the faster the performance, the more certain a pupil is about an answer, or the more automated his answer is. "Automatisation" means that the brain has fully acquired a skill and does not have to think about it anymore. Additionally, we also calculate error percentages for each week to cope with the problem that some pupils might simply be hitting buttons without concern for correctness. To express the learning curves, we take into account the following parameters:

• duration, ie the elapsed time for each event;

• duration average, ie the average time per event;

• number of events played;

• error percentage, ie the number of errors per number of events;

• duration median, ie the median time per event;

• duration variation in duration, ie difference in elapsed time for each event.

22We calculated the values for each of these parameters across all data that is accumulated for a given week that a pupil played the game. The results were then displayed as learning curves over time.

23Figure 3 shows a subsumed learning curve for the average duration of the different grades. The grey area indicates the confidence interval of the curve. As the learning curve in this case is subsumed from a variety of different values, the confidence interval indicates how well the learning curve fits the data. The learning curve of the first-grade pupils has the best fit. We can observe a learning peak at week 12 and a trough at week 15 and another peak towards the last week of play. Peaks indicate learning and that the task is not yet mastered as more time is taken to answer. First-grade pupils seem to have two larger learning peaks, indicating that the task has not yet been fully mastered. However, the curve subsumes a number of pupils and not all of them played the full range of weeks. Thus, we can only make a general statement. The other learning curves fit each other less. Third graders show development with several learning peaks. Fourth-grade pupils have a declining curve with a learning peak in week 4. Although the overall development indicates an improvement, there is a slight increase in slope at the right end, which might indicate another learning peak. Second-grade pupils also indicate an improvement, showing a decline in the learning curve. However, the learning curves of all three grades exhibit large confidence intervals at the end, making it difficult to make a valid statement for all students as a group. In order to get a better insight, we also looked at the individual learning curves of pupils. The goal is to further parameterise these learning curves in such a way as to understand learning stages and perhaps group learners according to their learning stage. Learning and the mastery of a task cannot be visualised by a single parameter. The graph yields insight only in combination with other parameters. Figure 4 shows an example of the learning curves for the parameters described above for one of the pupils. The following elements are depicted (from top left to bottom right): duration distributions for answers by week separated into correct and incorrect answers, average smoothed duration curve, number of events per week, error percentage of events played by week, variance of duration, and median duration per week. Inspection showed that these curves seem to be reasonable representations granting numeric representations of the observed learning curve.

24Looking at the first diagrams in Figure 4, we noticed that the variance of the duration indicates a cognitive event which depicts mastery in correctness and in duration. Diagrams 4 to 6 (error percentage, variance of duration, and duration median) break the distributional duration graph into its individual components. In this example, we can observe that the single components all show a similar trend. After a steep learning peak, the learning curve gradually declines, exhibiting some smaller peaks. Overall, we can see a large decline with a final low indicating an improvement in all observed parameters. Thus, we can assume a final mastery for this pupil. Using log data for Navigo, we are able to observe an oscillatory behaviour in the learning curves of pupils that dampens and converges over time as the player automates the learning goal. Learning breakthroughs seem to occur at the moment or shortly after a player takes a significantly longer time span for incorrect answers. This longer time span seems to mark important milestones in the acquisition process and may well be reflecting the cognitive restructuring we mentioned in the introduction.

3.5. Classification of Learning Curves

25While the individual learning curve gives us insight into the learning of an individual pupil, it is difficult and time-consuming to compare pupils and to identify differences and commonalities among them. For automatisation, we need a vector of features for each pupil that we can then feed into a classifier to identify common "learning" groups with comparable behaviour among these. We model the development of pupils, using the values of the first and last week the pupil played and calculating their difference for each of these parameters. Begin and end reflect the performance of pupils before and after playing. The difference depicts the development of pupils. A negative development indicates an increase in performance with a decline in error percentage, median duration, and variation in duration. The resulting vector contains the following nine features:

• "varnorm_begin", "varnorm_end", "varnorm_diff"

• "ernorm_begin", "ernorm_end", "ernorm_diff"

• "medinorm_begin", "medinorm_end" , "medinorm_diff"

26We are also interested in the way the pupils develop, ie in the oscillations of the learning curves. We calculate the number of peaks as well as the average and maximum height of peaks per pupil for each of the three parameters (error percentage, median, and variance of duration). The result are the following nine features:

• "ernorm_nopeaks", "ernorm_mean", "ernorm_max";

• "medinorm_nopeaks", "medinorm_mean", "medinorm_max";

• "varnorm_nopeaks", "varnorm_mean", "varnorm_max".

27Based on the above parameters, feature vectors are created for each of the 251 players. These are then used to cluster pupils with the overall goal of finding common learning patterns. For clustering, we experimented with different feature sets and two different clustering methods, the k-means algorithm and the PAM (partitioning around medoids) algorithm. For the final feature set (as described above) the PAM algorithm provided the best results. The PAM algorithm is a robust alternative to k-means clustering. Compared to k-means, it is less sensitive to noise and outliers using medoids as cluster centres instead of means. A medoid can be defined as the point in the cluster whose dissimilarities with all the other points in the cluster are minimal. The algorithm was proposed by Kaufman and Rousseeuw (1990) and maps a distance matrix into a given number of clusters. Based on the given feature vector, it tries to find the given number of medoids so that the dissimilarity within all clusters is the smallest.

4. Results

28Results of the above data analysis can be interpreted through the clusters that were obtained.

4.1. Clustering

29Before clustering, we need to determine the number of clusters that we want to obtain. We used the so-called elbow method to find the optimal number of clusters. We computed clustering algorithms for k from 1 to 10 clusters and calculated the total within-cluster sum of square (wss). This algorithm resulted in k=5 clusters as the optimal number of clusters. Figure 5 shows the resulting clustering trained and evaluated on the participants who played for at least 4 weeks. The depicted names are pseudonyms.

4.2. Characteristics of Clusters

30Examining the learning curves within the resulting clusters, we can make several observations. Even though all clusters show a mix of performance subgroups, they do seem to have certain characteristics that are describable. In particular, clusters 1 and 3 seem to describe the behaviour of the top and improving subgroups. Figure 6 depicts their characteristics.

31Cluster 1 represents mostly the pupils that improved while playing and secondly those that already did well on the written tests. It differs most significantly with respect to the other clusters in the parameters describing the path between the beginning and the end. The parameters show a larger variance of time spent to answer both in mean, median, and max values, despite a better performance in the beginning than the other clusters. Even the number of errors committed is much larger than in the other clusters. These values even increase with play. Amala is a good representative of this cluster. The feature values are visualised in Figure 6.

32Cluster 3 represents mostly the patterns for top pupils, that is those that performed well in both the written pre- and post-tests. Cluster 3 differs most significantly in the parameters describing mostly the path when compared to the other clusters. However, there are parameters referring to the oscillation that indicate top performers. The variance in the number of peaks in the learning curve is significantly lower. The duration is higher while the errors committed are also higher. These pupils also had a higher number of steps per session. A good example for cluster 3 is shown in Figure 7.

33While clusters 2, 4, and 5 did not have clear profiles with respect to the written tests, they do exhibit common patterns of learning curves. Cluster 2 has a large oscillation in the number of errors committed as well as variance and median number of peaks. These indicate non-converging learning curves that become apparent both in the time spent to answer and in the number of errors committed. They also have a lower number of events and fewer errors committed than other clusters overall and are slower towards the end than others. This can also describe the interaction for top and improving pupils as well as those making no progress. Figure 8 shows an example of a pupil in cluster 2.

34Cluster 4 parameters focus on the large improvement in duration and errors committed, as well as on a larger variance in duration at the beginning than others, with a smaller variance than others towards the end. Both duration and errors committed are higher among these pupils. A significant number of top and improved pupils fit this category. Figure 9 shows an example of this. Cluster 5 parameters describe another type of learner. Many of the pupils that do not improve are in this category. The number of errors committed is very small. Parameters reflect that there are no real learning curves with much fewer peaks than the other groups. These pupils were less exposed to making mistakes and played less as well. Figure 10 depicts an example from cluster 5.

4.3 Clusters as a Function of Pre-test and Post-test Results

35We distinguish four different groups based on the tests. These are

• "Improved" (start with low performance and end with significantly improved performance on the written test);

• "Top" (start and end with good performance on the written test);

• "None" (no change in low performance on the written test);

• "Worse" (a significant change from high to low performance on the written test).

36Figures 11 and 12 show the distribution of the above performance groups by grade. It seems that learning is already effective in Grade 2 and even in Grade 1 and that a heavy learning curve leading to worse performance before improvement could be avoided by training before entering Grade 3.

37Figure 13 shows the distribution of the pupils by grade into the clusters.

38Figure 14 depicts how the four groups of pupils (improve, none, top, worse) fall into the resulting five clusters. While all clusters have a mix of performers, Clusters 1 and 3 seem to capture mostly the group "improve" and "top" when looking at both "KV" (short vowel) tasks (identifying any long vowel against short vowels) and 'ìe" (identifying the long vowel "ie" against short vowels).

5. Discussion and Conclusion

39When inspecting log data for Navigo, we observe an oscillatory behaviour in the learning curves of pupils that dampens and converges over time as the player automates the task. Our goal was to model and parameterise these learning curves in order to detect common patterns and mastery of the material. Learning breakthroughs seem to occur at the moment or shortly after a player takes significantly longer to complete the task while registering incorrect answers. This longer time span seems to mark an important milestone in the acquisition process. This finding agrees with the literature; practice must include mistakes. Time factors and mistakes both indicate cognitive restructuring and form a major part of what identifies the successful writers in Cluster 3. At the same time, the individual learning curves posed some difficulty in interpreting their meaning or in grouping pupils. Through inspection, parameterisation, and non-linear clustering methods, we were able to answer the proposed research questions as follows.

40RQ1: How can learning curves be characterised? By clustering the pupils into five groups that could be meaningfully described, we could show different patterns in playing and learning. It could also be seen that some patterns are specific to pupils that have better writing skills and those that are improving their writing skills through gameplay.

41RQ2: Are there commonalities or individual classifiable differences? Clustering using the parameters that were derived through data inspection led to consistent groups with similar learning curves that can be described as specific and different from other groups.

42RQ3: Is it possible to recognise and analyse outliers? At this point, we were not able to recognise and analyse outliers. Due to the many different kinds of learning curves, it is possible to be a good writer and have several learning curve patterns. The inverse is also true that a specific learning curve does not lead to a guaranteed classification of writing proficiency or grade.

43We were able to identify some patterns that are common to subgroups of pupils. These patterns do not always predict their performance in a writing test. However, there are some patterns that seem to correlate mostly with top performers or improving pupils. It is not possible to find all pupils of a subgroup (performance in writing or grade) given a particular pattern in the learning curve. This may be due to many factors. First, the tests do not always accurately reflect a pupil's knowledge. Second, learning curves are highly individual. In addition, some pupils might have played badly on purpose or were unmotivated or had a sibling or friend play for them. Overall, however, the clusters reflect the patterns that were noticed when observing the data and allowed us to compute the interaction between the parameters in a non-linear manner in order to consider complex dependencies between the parameters to identify similar learning patterns.
To summarise, understanding learning curves is very complex and does not necessarily allow us to detect when learning is finalised. More research is needed in order to understand the various parameters. Much more data and probably more parameters will be needed in order to detect mastery. However, we showed a number of different learning curve patterns that seem to be common across a subset of players.
Future work will include further data collection as well as looking in more detail at external factors that influence the learning curves, such as prior knowledge, grade, learning disability, and (ir)regular practice.