Navigation – Plan du site

AccueilVolumesVol. 27, n° 1Pratique et rechercheUsing R to Apply Open Science Pri...

Pratique et recherche

Using R to Apply Open Science Principles in Second Language Acquisition and Didactics

Science Ouverte en didactique des langues : exemple de l’utilisation du logiciel R
Marie-Pierre Jouannaud

Résumés

Cet article commence par la description des apports du mouvement de la science ouverte : libre accès aux publications et aux résultats scientifiques, collaboration entre chercheurs, et surtout transparence méthodologique. Il montre ensuite que le logiciel R est un outil utile pour partager les méthodes d'analyse des données, parce qu’il permet de fournir des informations explicites sur le prétraitement des fichiers de données et sur les analyses statistiques utilisées. R aide également à la compréhension des résultats en facilitant la production de graphiques qui rendent compte des données de façon claire et esthétique. Son potentiel est illustré dans le contexte d'un projet de recherche à grande échelle nécessitant l'utilisation de statistiques pour analyser les résultats de prétests et posttests de Luciole, une application numérique permettant d'enseigner la compréhension de l’oral en anglais dans les classes de CP et CE1 en France.

Haut de page

Texte intégral

The development and field testing of the Luciole app described in this paper was funded by the Fluence and Trans3 projects (grant number ANR-22-FRAN-0008). I would like to thank Matthieu Rolland for his introduction to R, and my fellow Luciole team members Emilie Magnat, Coralie Payre-Ficout and Mathieu Loiseau (the team leader), without whom none of this would have been possible10.

1. Introduction

  • 1 The failure to reproduce the results of many published studies (originally in psychology), undermin (...)

1Scientific research is a collaborative enterprise: science rests on the sharing of observations, ideas and methods, on debate among competing hypotheses, and on the interpretation of results. Traditionally, this played out in scientific journals, society meetings and conferences, or through private correspondence between researchers. The advent of the Internet has opened up new spaces for collaboration, and more recently a number of academic journals have embraced an “open access” model. Awareness has grown, however, that sharing ideas and results may not be enough. In the context of the replicability crisis1 in social sciences (Open Science Collaboration, 2015), it is also important to share data (MacWhinney, 2000) and data collection materials such as questionnaires, tests, etc. (Marsden et al., 2015).

2In this paper, the advantages of using the statistical software R (R Core Team, 2022) to go one step further and share data analysis procedures together with collected data will be described. R is a free, open-source environment for statistics and graphics as well as a programming language. Because analyses are conveyed using lines of code that can easily be shared with other researchers, and run again using the same or similar data, it is easy to collaborate with colleagues who can reproduce analyses and graphics, point out flaws, and perhaps tweak the code to improve it. Although R requires initial training (as any statistical software), there are books, MOOCs and videos dedicated to its use, as well as a vibrant support community one can rely on when difficulties are encountered. The use of R will be illustrated by reference to the development and validation of an educational app designed to teach English listening comprehension to young learners in French elementary school classrooms.

2. Background: The Open Science movement

3At the end of the 1990s, scientists started to rebel against the power of the major scientific publishers, whose publishing costs decreased in the Internet era but who nonetheless charged steadily rising fees which did not seem justified (Larivière et al., 2015). Although researchers did the actual work of writing and reviewing articles, publishing companies forced them to sign away their rights and charged university libraries the world over for access to the scientific output of their own employees. Most articles were behind paywalls and thus difficult to access for the general public (who had, albeit indirectly, paid for the research), or for researchers with little or no resources or institutional support. This state of affairs inspired reactions at different levels, which correspond to three types of open access (OA) models, called “rogue OA”, “gold OA”, and “green OA” (Archambault et al., 2014).

4At the individual level, some researchers went “rogue” and called for rebellion, by illegal means if necessary (Swartz, 2008). The Sci-Hub repository of pirated academic papers is a child of this movement (Siew, 2017), belonging to rogue OA (also known as “Robin Hood OA”).

5More sustainably perhaps, some journals also decided to adopt the open access model, making their articles available without cost on the web. Some of these journals ask researchers to pay “author processing fees,” which cover costs that would otherwise have been paid for by subscription fees, such as the organization of the peer review process, or the technical maintenance necessary when publishing content on the web. Peer-reviewed articles thus accessible belong to gold OA. The easy-to-search Directory of Open Access Journals (DOAJ) compiles a list of credible peer-reviewed open access journals (Bohannon, 2013), and assigns “gold DOAJ” badges to such articles. The subtype of gold OA that does not demand fees of either readers or authors is called “diamond OA.” A list of diamond OA journals for applied linguistics is curated by Al-Hoorie (2022). ALSIC and Language Learning and Technology (LLT), launched respectively in 1999 and 1997, had an open access strategy from the very beginning and were early examples of the diamond OA model in the field.

  • 2 The CC-BY Creative Commons Licence “allows users to distribute, remix, adapt, and build upon the ma (...)

6The third source of evolution toward open access came from governments adopting resolutions (at the European level) or laws (at the national level) pushing for government-funded research to be available to the public (Enserink, 2016). In France, the HAL platform was conceived as an open repository of each public researcher′s published output. Documents are available after a short embargo or immediately upon publication if the CC-BY licence2 is used (MESR, 2022). Research output thus shared by their authors on institutional platforms is referred to as “green OA.”

7Of course, the open access model is not without its downsides. First, the three broad categories presented above (green, gold and rogue) do not cover all possibilities, and aggregator sites such as ResearchGate are categorized as “other OA” (Archambault et al., 2014). The rise of legitimate peer-reviewed open access journals has also seen a simultaneous increase in the number of predatory journals whose only goal is to collect authors' evaluation and publication fees without conducting meaningful peer review before publication, leading to scientific output of dubious quality (Bartholomew, 2014). Although this diversity makes the effects of the open access model difficult to evaluate and somewhat controversial (Basson et al., 2020), most studies seem to find that OA is delivering on its promise of greater accessibility, visibility and impact of research (Archambault et al., 2014). More than half of (recent) papers are available in open access, and the proportion keeps growing, through the decommissioning of old papers as well as the publication of new papers directly in OA. OA also leads to more citations, especially for papers published in open institutional repositories (green OA). The initial proponents of open access declared that:

removing access barriers to this [scientific] literature will accelerate research, enrich education, share the learning of the rich with the poor and the poor with the rich, make this literature as useful as it can be, and lay the foundation for uniting humanity in a common intellectual conversation and quest for knowledge (Chan et al., 2002, §1).

8Their prophecy may sound grandiose and overly optimistic in retrospect, but the positive role of the web as a virtual space for the dissemination of knowledge seems to have been partially realized.

3. Methodological transparency

9However, Marsden (2020) argues that open science goes much further than open access and the free and permanent availability of research output to the general public. The adjective “open” should also refer to methodological transparency – a philosophy with significant consequences at all stages of the research process.

  • 3 The Open Science Framework repository serves a similar purpose, but for science in general: osf.io.

10First, when designing research protocols and before data collection, permission must be sought to make (anonymized) data publicly available, and provision must be made for public storage of all instruments designed and used, such as questionnaires, pre- and post-tests or experimental protocols. The IRIS database (Instruments and data for research in language studies, Marsden et al., 2015) is a natural repository for such documents, as it was designed specifically to facilitate sharing and collaboration between Second Language Acquisition (SLA) and Applied Linguistics researchers and is free and open to the public3.

11Second, knowing that our instruments will be publicly available and open to criticism by the community has a positive washback effect on instrument design (a washback effect is the influence of a subsequent event – here, publication, on a previous event – here, design). According to Marsden (2020), if we know that other teams are likely to re-use our instruments, we are motivated to make sure their usage is self-explanatory, and that they are accompanied by tutorials if necessary (to describe a scoring scheme, for example). Knowing our productions will be shared eventually also works on instrument quality, and leads to better proofreading, better quality items through prior piloting, as has been shown to happen in other related domains (Wicherts et al., 2011).

12Making our instruments (questionnaires, tests, interview protocols, etc.) available is also essential if we want other researchers to conduct replications of our studies. The reproducibility crisis in human sciences started when highly publicized results in social psychology were questioned and large-scale replication efforts failed to reproduce them (Open Science Foundation, 2015), thus undermining confidence in earlier results. The main way to ensure that results are reliable is to have other researchers reproduce them with other participants (the definition of reliability is that results are stable and resistant to changing conditions or subjects, ALTE, 1998). With this goal in mind, some SLA researchers have also started to choose important results from the field as targets for replication. Morgan-Short et al. (2018), for example, attempted a large-scale replication of Van Patten (1990), according to whose results attention to grammatical form (such as determiners or grammatical suffixes) detracts from text comprehension. Unlike the initial study, the results of the replication did not support the initial hypothesis. The authors were then able to conclude that the picture was still not clear enough to draw firm conclusions on whether learners are able to focus on both form and meaning when processing texts for comprehension. The availability of initial material and data is essential to be able to conduct such studies.

13Large-scale collaboration made possible by open science methods and procedures is important for another reason in social sciences. Dealing with human subjects vastly increases the number of variables involved and the complexity of the analyses needed to answer our questions. Because several methods are usually possible to analyze the data collected, results might differ depending on which method is chosen. Each research article usually only presents one possible analysis, but using a different method might yield different results (Silberzahn et al., 2018). Even using the same method but with different decisions at each step of the process (selection of participants, etc.) can greatly alter the conclusions reached (Simmons et al., 2011). This problem is unavoidable, and being open about each decision is the easiest way to mitigate it. Other research teams might then hopefully be inspired to try to replicate the results to eventually obtain a clearer picture. For these reasons, it is essential, after data collection, that all the steps taken to “clean” the data before analysis be described and justified (for example, stating whether outliers, or incomplete data from some participants, were removed, and on what grounds).

4. Quantitative methods

  • 4 The extent to which results are distributed (un)symmetrically around the mean, i.e. there is a long (...)

14Applied linguists and SLA researchers often need to use quantitative analyses to summarize their observations, describe their results, offer interpretations and answer their research questions. Detailed descriptive statistics (mean, standard deviation, skew4, reliability of instruments, etc.) must also be provided to enable reanalysis, future comparison with other data sets or meta-analyses. Best practice recommendations for “robust statistics” (Larson-Hall, 2012) also advise against the exclusive use of means to describe group results and p-values to assess significance, and require additional information such as confidence intervals, or an explicit and predetermined rationale to get rid of outliers.

15However, most applied linguists and didacticians come from the humanities and are not entirely comfortable with the use of statistics. Although statistics and graphs are ubiquitous in our daily environment, and are required to understand a majority of Second Language Acquisition research articles, statistical reasoning is rarely taught in language departments. Several studies have pointed out that statistical literacy is not high, and that many researchers feel underprepared for the comprehension and use of statistics (Loewen et al., 2014). Nevertheless, the same studies show that most researchers know how to interpret basic descriptive statistics (mean, median, and standard deviation), are familiar with a few language testing concepts (validity and reliability) and can interpret some inferential statistics (p-value, t-tests, etc.). Two caveats are in order here: one is that “interpreting” means understanding the results, and not necessarily being able to use these tests; the other is that Europeans tend to be less knowledgeable than North Americans (there is no breakdown by nationality in the studies mentioned, so that the level of statistical literacy among French researchers in particular is not known).

16One way to simplify access to statistics is by using visual means of representation: graphics are not “a mere frill” (Larson-Hall, 2017, p. 244), in that they help us grasp patterns more easily than when we read statistical results in numerical form: “of all methods for analyzing and communicating statistical information, well-designed data graphics are usually the simplest and at the same time the most powerful” (Tufte, 2001, p. 9, in Larson-Hall, 2017, p. 244). They are useful tools to examine our own data in the exploration phase of a research study, to detect outliers and aberrant data, or to notice subtle patterns, which we can later try to confirm through statistical analyses. According to Larson-Hall (2017), graphics should also be “data-rich,” i.e. they should give the reader information about the distribution of the data, or even “data-accountable,” i.e. they should represent all data points (for example with a box plot overlaid with points, cf. below). This information can also be conveyed through numbers or statistical tests, but graphs are easier to understand, and the absence of data-rich or data-accountable graphs might lead readers to think that there is less variation in the data than there actually is.

5. The R software

17In order to conduct quantitative analyses, a majority of applied linguists use the IBM SPSS software (Statistical Package for the Social Sciences), and a small minority (15 to 20% ) use R (Loewen et al., 2014; 2020)5. R is a statistical environment for statistical computing and graphics, and it has several advantages. It is free, open source, and statisticians and computer scientists regularly develop extensions to the core functions in R. Because of this, the range of possible analyses and graphics is virtually unlimited and very flexible.

18Reproducibility of data analysis, however, is “perhaps the most compelling advantage provided by R” (Mizumoto & Plonsky, 2016, p. 2). R makes it easy to reproduce analyses because each action it performs corresponds to a line of code (one or several functions) that is controlled by the user. Each line of code can be commented so that users can explain to someone else using the code (or to themselves, at a later date) what it does and why. For example, there is a function to import files in table format (.csv), functions to add or delete rows or columns, or to perform basic operations (means, medians, etc.). Other functions allow users to produce graphics. In addition, libraries of functions called “packages” automate additional operations or offer more elaborate graphics. Analyses are performed using successive function calls stored in an R “script” (a sequence of commented functions) that can then be run again as is, by the same or a different person, without the need to go through each step anew.

19By contrast, in less open software such as SPSS or even Excel, the sequence of actions necessary to perform an analysis with a given data file is not memorized automatically, and the steps taken to clean the files before analysis are not transparent. As shown above, this transparency is nevertheless essential, because using a different method, or the same method but different criteria for selecting relevant data (such as for the exclusion of outliers), might yield different results (Silberzahn et al., 2018; Simmons et al., 2011). R provides a natural environment and culture to share these decisions and analyses.

20The open science principles outlined here are mostly uncontroversial: everyone is probably in favor of an easier access to scientific results, and most researchers agree on the necessity of other scientists being able to conduct replications thanks to access to the instruments or the findings of previous studies. However, scientists work in the real world, and it is not always easy to live up to these principles, especially when they compete against other realities, such as the incentive to publish as much as possible, or the need to spend a non-negligible amount of time looking for funds and responding to calls for papers or projects. In the rest of the paper, I will try to show how a research team designing an English comprehension game (Luciole) attempted to follow some of these principles, and how the use of R helped in this respect. The research questions that will be explored are the following: to what extent can open science principles be applied to a large-scale research project involving young school children's data learning English in the French context? How does the use of R help in such an endeavor?

6. Presentation of the study

6.1. Context: The Fluence project

21Luciole is one of the apps developed in the Fluence project (Mandin et al., 2021), selected and funded by the French National Research Agency as part of a program focusing on digital learning tools and attempting to develop evidence-based education in the French context (Gentaz, 2017). The idea behind evidence-based education is that educational tools should be tested to try to demonstrate their efficiency in the classroom, in order to then help educators choose methods applicable to their contexts and corresponding to identified best practices. The main goal of the project was to test three apps for first- and second-graders. The first two (EVAsion and ELARGIR) are designed to improve reading fluency in L1 French. EVAsion deals with cognitive mechanisms inherent to reading (visual and visuo-attentional processing), while ELARGIR trains holistic processing (orthographic units, prosody, breath groups), which ultimately leads to reading fluency. The third app (Luciole) focuses on English listening comprehension. Many theory-based educational interventions work when initially tested with small samples or in controlled laboratory conditions, but fail to bring about any significant improvements when deployed on a larger scale in actual classrooms (Goigoux et al., 2021). Because of this, the study was a large-scale endeavor involving a large number of schools (n = 35) and classrooms (n = 52).

6.2. Study design

  • 6 Because of the Covid-19 pandemic, only the first two years were actually implemented.

22The study design was a classic pre-test – intervention – (delayed) post-test experimental design, with an experimental group and an active control group working on different educational apps. The project was a longitudinal study following a cohort of students from first- to third-grade (2018–2020)6. The three apps were used symmetrically: Luciole groups (English listening comprehension) served as active controls for EVAsion and ELARGIR (L1 reading in first- and second-grade respectively), while they in turn acted as control groups for Luciole (used in both firs-t and second-grade). Active controls were used to test the effectiveness of the apps because studies have suggested that comparing a passive “business-as-usual” control group to groups using a novel educational technology might not entirely be fair, and part of the effect might be due to the introduction of a digital tool and new activities in the classroom (Boggio et al., 2023).

23The study also tried to be fairly ecological, in that classroom teachers received a one- or two- day training explaining the conceptual underpinnings of the apps and were given guidelines for their use, but they were free to schedule their lessons and organize their time as they wished. It was hypothesized that this would mirror the usual use of such apps or educational resources by teachers in their classrooms. They at best get a short training on how to use them, but are then on their own afterwards.

6.3. Participants

24730 first-grade students in 52 classrooms (35 schools) were originally recruited, and 679 students completed the post-tests. All schools participated on a voluntary basis, and parents were asked to give their informed consent. As the sample was deemed to be representative of the French school population and this was a study carried out in ecological conditions, all results were kept. Almost all children were complete beginners in English, having had at most eight weeks of English lessons in first-grade before the experiment (amounting to 12 hours), although some of them had also learnt a few words (one, two, three, etc.) and expressions (Hello, how are you?) in kindergarten or through older siblings.

6.4. Game design

25The Luciole game is compatible with the existing curriculum for French elementary schools, which calls for an emphasis on oral skills. Listening in particular is a foundational skill as it provides the linguistic input on which subsequent learning will be based. Listening comprehension depends to a great extent on aural word recognition (Wang & Treffers-Daller, 2017; Zoghlami, 2016), but work on isolated words is not sufficient, as learners also need to be able to recognize words in the stream of speech, surrounded by other words (Matthews & O'Toole, 2015). Prior work has shown that young French learners of English indeed tend to be unable to understand more than one word per sentence and have to guess overall meaning based on this unique word (Audin, 2003). In order to help them integrate the meaning of several elements, Luciole uses systematic activities where recognition of several words is task-essential (DeKeyser & Prieto Botana, 2015). In other words, the task can only be performed successfully if several elements are attended to and their meaning integrated (for example, when one character tells the hero to “take two red shirts” in preparation for a trip, the player needs to drag two red shirts—not shoes—into the suitcase; the number must be two, not three, and the shirts must be red).

26All the activities are embedded within a narrative in which a French child (Sasha) has to learn to understand English and interact with English speakers in order to help a team of British spies recover kidnapped animals. The dialogues are mostly in French at the beginning of the game, but progressively more English is integrated as the story progresses. Game design elements that can be classified as “gamification” (Sailer et al., 2017) include the use of points (stars) to provide feedback during each exercise, badges to represent achievement visually, one character (Sasha) serving as an avatar for the player, and above all, a narrative to contextualize the language introduced in the game, and justify the activities the player engages in. In this respect, the game is also compatible with the action-oriented approach in which language is hypothesized to be acquired when it is used in the pursuance of authentic non-language related (real life-like) goals.

6.5. Instruments (pre- and post-tests) and procedure

27The apps were designed to be used three times a week for 20 minutes each time, over a period of around ten weeks, or three months including school holidays. The timeline was as follows: the pre-tests were administered in November, the children started using the apps in February, and the post-tests took place in June (around one month after the end of the experiment).

28The English listening comprehension pre- and post-tests were made up of 21 multiple-choice questions. For each question, students heard an English word or sentence and had to point to one of four pictures corresponding to what they heard. The recordings did not use any of the visuals or voices used in the game, and the tests were administered one-on-one by trained research assistants.

7. Application of open science principles

7.1. Informed consent and anonymity

29All actors playing characters in the Luciole game signed release forms and gave their permission to use their voices.

30At the beginning of the project, ethical approval was received from the University of Grenoble ethical committee (CERNI), as well as the French personal data protection agency (CNIL) for the research protocol. Signed parental consent was obtained for all participants in the final study (children whose parents did not sign the consent form were still able to use the apps, but their data was not included in the study). Data collection involved pre-tests, post-tests, and traces of student activity within the apps. Although the CERNI advised the project leaders to try to collect children's informed consent as well as their parents', this was considered difficult because of their young age as well as the setting. Schoolchildren (especially at this age) usually trust their teachers and tend to obey them implicitly. Also, many of them were shy with the research assistants (who were strangers to the class) to begin with. Because of this, children were considered willing if they showed their agreement to start the tests verbally or non-verbally (with a nod, or moving to an attentive position to get ready, etc.), displaying “assent” rather than full-fledged “consent” (Gallagher et al., 2010; Waligora et al., 2014).

31Anonymity of data is a major concern in contemporary research practices. It is important to protect the identity of participants, both so that researchers might not be influenced by this knowledge, and because participants' data must not be traceable back to their source when the data are stored or shared. Every child received a “Fluence ID” for the duration of the project. The teachers, school heads and school board authorities had access to the list of students' names and corresponding IDs, but researchers only had access to Fluence IDs. When taking pre- and post-tests, students, teachers or research assistants wrote both their names and student IDs on answer sheets, but the names were removed before the sheets were retrieved by the researchers and answers were entered into CSV files. When children used the apps on class tablets, their Fluence IDs were also used to link their identifier (a pictogram) to the traces of their activity within the app.

7.2. Choice of R

  • 7 e-FRAN (Espaces de formation, de recherche et d'animation numérique) is a research funding program (...)

32As outlined above, SLA researchers and didacticians are usually not very knowledgeable about statistical analysis, and the Luciole team was no exception at the beginning of the project. This problem was alleviated in two ways. The first was that because of the scale of the project and thanks to the funding obtained (e-FRAN, PIA 27), a post-doc researcher was hired to help with statistics. The second was that this researcher started the English team (who, being the original “control” group, did not have nearly as much access to his time as the other groups) on the path to (limited) statistical autonomy. He recommended the use of R because of his previous experience with it, because it was free (an important consideration as the researchers involved did not all belong to the same lab and did not all have access to statistical programs other than Excel), and because it was easy to share data between members of the team and reproduce analyses.

33At the beginning of the project, in preparation for the pre-test analyses, the post-doc researcher shared the R files with the English team and walked us through them. In this way, we were able to replicate the initial analysis on our own time and understand the effect of each line of code (to which we added our own comments). At the same time, some of the project members followed a MOOC on the FUNMOOC platform, which serves as both an introduction to statistics and a guide to using R (Falissard & Lalanne, 2017). Thanks to this initial training and the frequent use of the Google search engine as an omniscient tutor, the next iteration of analyses was largely conducted independently.

7.3. Use of R for data cleaning

  • 8 Because the actual code for the iteration presented in the article is not very informative, the exa (...)

34Below is an example of the code produced for the cleaning of the data file before running any statistical analysis8. The lines beginning with a hashtag correspond to comments, and the others are lines of code, with “foo” being the new name of the data file loaded into R, “read.csv” being the function used to load it (“sep” is an attribute indicating that the field separators used in the CSV file are semi-colons, not commas), and “na.strings” indicating which symbols correspond to absent data. The “filter” function is used to select rows in the data file (in order to keep some participants' data and not others'). In this way, the fact that we did not use data from a bilingual class, or from students whose parents did not sign the consent form or who were absent for a test, is stated explicitly, both in the comments and the commands.

#import data file

foo <⎯ read.csv ("PrePo_CP_Luc_Scores_data.csv", sep = “;", na.strings=c("","NA"))

#enlever 22 eleves classe bilingue (classe num 32)

foo <⎯ filter (foo, Classe!=32)

#enlever 43 non consentement

foo <⎯ filter (foo, consent==1)

#enlever 30 absents prétest ou posttest

foo <⎯ filter(foo, abs_pretest==0)

foo <⎯ filter(foo, abs_posttest==0)

#il reste 520 enfants

35Each decision to remove data or participants is completely transparent, and any user running these lines again will see the effect of these decisions immediately. It is also easy to tweak the code and observe the potential effects of making different decisions on the end result. This transparency makes it difficult to doctor the data or tamper with the analysis in such a way that they correspond to the hoped-for results.

7.4. Use of R for the visualization of results

36After the initial data cleaning process comes the visualization of results and the analysis. The choice of R was not initially motivated by this, but R is known for the quality of its graphics, and the range of data visualization it offers. The main variable of interest in the Fluence study was the English comprehension post-test score in Luciole groups versus the control groups working on a French app (EVAsion). This result can be conveyed in numerical form, stating that the Welch two-sample t-test shows that pupils in the Luciole group had significantly better English scores at the end of the school year than pupils in the EVAsion group: Evasion = 8.41 vs. Luciole = 11.18 ; t = -10.59, df = 676, p < .0001. However, the use of graphs allows readers to grasp these results more intuitively.

37As we will now see, however, all graphs are not created equal, and the amount of information they convey has different effects on readers' comprehension. Figure 1 presents two ways of visualizing results without sufficient information about dispersion. Figure 1 is a simple bar plot showing the results at pre-test (beginning of the first-grade) and post-test (end of first-grade). The figure clearly shows that both groups improved, and that the Luciole groups (in blue) improved more than the French groups (in red). The bar plot only takes mean results into account, and thus hides any information about variability.

Figure 1 – Visualization of results at pre- and post-test with a bar plot.

Figure 1 – Visualization of results at pre- and post-test with a bar plot.

Figure 2 – Visualization of results at pre- and post-test with a line graph.

Figure 2 – Visualization of results at pre- and post-test with a line graph.
  • 9 The fact that the scale is not shown in full in figure 2 might also give the impression that the im (...)

38Figure 2 presents a line graph with some information about dispersion in the error bars at the beginning and end of each colored line. However, this information is hard to interpret, and not very prominent in the graph, so that readers still have a chance of underestimating the presence of variation in the results. Error bars often give rise to misunderstandings (Belia et al., 2005). For example, the bars might lead to the misinterpretation that they represent the range of observed values (rather than an estimation of the range of possible values for the mean given the dispersion observed)9.

Figure 3 – “Data-rich” visualization of the results at post-test (boxplot).

Figure 3 – “Data-rich” visualization of the results at post-test (boxplot).

39The plot in figure 3 shows that variability is much higher than what the previous figures might have led readers to expect. We now have what Larson-Hall (2017) calls a 'data-rich' graphic showing the dispersion of data with a box-plot (also called box-and-whisker plot), providing explicit information about the median (the black line dividing each box horizontally), the lowest and highest values (including a few outliers represented by small circles), and the lower and upper quartiles (the bottom and top lines enclosing each colored box). Readers can now clearly see that the range of values for both groups is identical, with students scoring zero and perfect scores in both groups. The main difference seems to be that there are fewer low scores in the Luciole group, and that its median is higher than that of the French group (reaching the top quartile of the French group, which means that half of the Luciole group received results as high as the top quarter of the French group).

Figure 4 – “Data-accountable” visualization of the results (stripchart boxplot).

Figure 4 – “Data-accountable” visualization of the results (stripchart boxplot).

Figure 5 – “Data-accountable” visualization of the results (line graph).

Figure 5 – “Data-accountable” visualization of the results (line graph).

40The last figures present results for Luciole (English) and EVAsion (French) groups with all data points represented: stripchart boxplot (Figure 4, post-test) and line graph (Figure 5, pre- and post-test). These are two “data-accountable” graphs in Larson-Hall (2017)'s terms, because they attempt to convey all the information available and they allow the reader to unambiguously visualize the amount of individual variation among both groups. The boxplot in figure 4 is the same one as in figure 3, on which individual data points have been overlaid (because many students have the same scores, these individual points are difficult to separate). The dispersion of scores and the score overlap between Luciole and EVAsion groups are even more obvious, as well as the upward drift of scores for Luciole groups.

41The graph in figure 5 is a line graph with one line per participant linking results at pre- and post-test (as well as two overlaid lines of slightly thicker width representing mean results for each group). The resulting graph is very crowded and perhaps less clear than the one presented in figure 4, but it also informs about learning trajectories and shows that although the general trend was upward, some participants saw their results decline from pre- to post-test.

42All these graphs were produced using the GGPlot package in R, and the corresponding R scripts can be found, together with the data files and the post-tests, in the IRIS database.

8. Conclusion

43The aim of the Open Science movement is to produce transparent and trustworthy research with results freely available to everyone, and to encourage collaboration among researchers. It has been instrumental in the rise of open access publications as well as in the development of a number of platforms dedicated to the sharing of instruments or data files between researchers and with the general public.

44In this paper, I have argued that the R software is a useful tool which can help fulfill these aims by opening up new spaces for collaboration. Indeed, R contributes to the sharing of data analysis methods with explicit information about the pre-treatment of data files and the statistical analyses used. It also helps with the comprehension of results by facilitating the production of visually appealing data-accountable graphics. This was illustrated in the context of a large-scale research project requiring the use of statistics to analyze data from pre- and post-tests to validate the use of Luciole, an educational app teaching English listening comprehension in first-grade classrooms in France.

45It must be said, however, that while most researchers share the lofty goals of open science presented here, the reality of conducting research is sometimes far removed from them. R does have a steep learning curve (mainly because we usually need to learn both the statistical methods and the lines of code required to apply them), but learning how to use it can also be done collaboratively, with peers working on the same project, with the help of tutorials and user websites, or with one of several available MOOCs (e.g., Falissard & Lalanne, 2017). It is accessible to researchers previously unfamiliar with coding (or even statistics), provided they are able to put in the time necessary to learn it and continuously refresh their skills.

Haut de page

Bibliographie

Al-Hoorie, A. (2022). Applied Linguistics Open Access Journals. Ali H. Al-Hoorie. https://www.ali-alhoorie.com/applied-linguistics-open-access-journals

Archambault, É., Amyot, D., Deschamps, P., Nicol, A., Provencher, F., Rebout, L., & Roberge, G. (2014). Proportion of Open Access Papers Published in Peer-Reviewed Journals at the European and World Levels – 1996-2013. Copyright, Fair Use, Scholarly Communication, etc. https://digitalcommons.unl.edu/scholcom/8

Association of Language Testers in Europe [ALTE] (Ed.). (1998). Multilingual glossary of language testing terms. Cambridge University Press.

Audin, L. (2003). L'apprentissage d'une langue étrangère à l'école primaire : Quel(s) enseignement(s) en tirer  ? Les Langues modernes, 3, 10-19.

Bartholomew, R. E. (2014). Science for sale: The rise of predatory journals. Journal of the Royal Society of Medicine, 107(10), 384-385. https://doi.org/10.1177/0141076814548526

Basson, I., Blanckenberg, J. P., & Prozesky, H. (2021). Do open access journal articles experience a citation advantage? Results and methodological reflections of an application of multiple measures to an analysis by WoS subject areas. Scientometrics, 126(1), 459-484. https://doi.org/10.1007/s11192-020-03734-9

Belia, S., Fidler, F., Williams, J., & Cumming, G. (2005). Researchers Misunderstand Confidence Intervals and Standard Error Bars. Psychological Methods, 10(4), 389-396. https://doi.org/10.1037/1082-989X.10.4.389

Boggio, C., Zaher, A., & Bosse, M. (2023). ECRIMO, an app to train first graders’ spelling: Effectiveness and comparison between different designs. British Journal of Educational Technology, 54(5), 1332-1350. https://doi.org/10.1111/bjet.13354

Bohannon, J. (2013). Who's Afraid of Peer Review? Science, 342(6154), 60-65. https://doi.org/10.1126/science.2013.342.6154.342_60

Chan, L., Cuplinskas, D., Eisen, M., Friend, F., Genova, Y., Guédon, J.-C., et al. (2002). Budapest open access initiative. Declaration. Open Society Foundations. https://www.budapestopenaccessinitiative.org/read/

Charles, E., Magnat, E., Jouannaud, M.-P., Payre-Ficout, C., & Loiseau, M. (2022). Effect of an EFL listening comprehension learning game on phonemic awareness in French [poster]. ATFLY 2022 Advances in Teaching Foreign Languages to Young Learners, Leipzig. https://hal.science/hal-03865677/file/Charles_et_al_ATFLY_EFT_awareness.pdf

DeKeyser, R., & Prieto Botana, G. (2015). The Effectiveness of Processing Instruction in L2 Grammar Acquisition: A Narrative Review. Applied Linguistics, 36(3), 290-305. https://doi.org/10.1093/applin/amu071

Enserink, M. (2016). In dramatic statement, European leaders call for 'immediate' open access to all scientific papers by 2020. Science. https://doi.org/10.1126/science.aag0577

Falissard, B., & Lalanne, C. (2017). Introduction à la statistique avec R [Cours en ligne]. FUN MOOC. http://www.fun-mooc.fr/fr/cours/introduction-a-la-statistique-avec-r/

Gallagher, M., Haywood, S. L., Jones, M. W., & Milne, S. (2010). Negotiating Informed Consent with Children in School-Based Research: A Critical Review. Children & Society, 24(6), 471-482. https://doi.org/10.1111/j.1099-0860.2009.00240.x

Gentaz, E. (2017). Un « levier » prometteur pour développer des liens entre des acteurs de la recherche, de l'école et du numérique : E-FRAN. A.N.A.E., 150, 515-517.

Goigoux, R., Renaud, J., & Roux-Baron, I. (2021). Comment influencer positivement les pratiques pédagogiques de professeurs expérimentés ? In B. Galand & M. Janosz (Eds.), Améliorer les pratiques en éducation : Qu'en dit la recherche ? (pp. 67-76). Presses universitaires de Louvain. https://hal.science/hal-03153950

Larivière, V., Haustein, S., & Mongeon, P. (2015). The Oligopoly of Academic Publishers in the Digital Era. PLOS ONE, 10(6), e0127502. https://doi.org/10.1371/journal.pone.0127502

Larson-Hall, J. (2012). Our statistical intuitions may be misleading us: Why we need robust statistics. Language Teaching, 45(4), 460-474. https://doi.org/10.1017/S0261444811000127

Larson-Hall, J. (2017). Moving Beyond the Bar Plot and the Line Graph to Create Informative and Attractive Graphics. The Modern Language Journal, 101(1), 244-270. https://doi.org/10.1111/modl.12386

Loewen, S., Gönülal, T., Isbell, D. R., Ballard, L., Crowther, D., Lim, J., Maloney, J., & Tigchelaar, M. (2020). How Knowledgeable are Applied Linguistics and SLA Researchers about Basic Statistics?: data from North America and Europe. Studies in Second Language Acquisition, 42(4), 871-890. https://doi.org/10.1017/S0272263119000548

Loewen, S., Lavolette, E., Spino, L. A., Papi, M., Schmidtke, J., Sterling, S., & Wolff, D. (2014). Statistical Literacy Among Applied Linguists and Second Language Acquisition Researchers. TESOL Quarterly, 48(2), 360-388. https://doi.org/10.1002/tesq.128

MacWhinney, B. (2000). The CHILDES project: Tools for analyzing talk: Transcription format and programs, Vol. 1, 3rd ed (p. xi, 366). Lawrence Erlbaum Associates Publishers.

Mandin, S., Zaher, A., Meyer, S., Loiseau, M., Bailly, G., Payre-Ficout, C., Diard, J., Valdois, S., Blavot, A., Bosse, M.-L., Briswalter, Y., Chalon, N., Godde, E., Ingremeau, S., Jouannaud, M.-P., Lequette, C., Magnat, E., Masperi, M., Piat-Marchand, A.-L., … Zanoni, M. (2021). Expérimentation à grande échelle d'applications pour tablettes pour favoriser l'apprentissage de la lecture et de l'anglais oral. EIAH 2021 – 10e Conférence sur les Environnements Informatiques pour l’Apprentissage Humain, 118-129. https://hal.archives-ouvertes.fr/hal-03292798

Marsden, E. (2020). Methodological transparency and its consequences for the quality and scope of research. In J. McKinley & K. Rose (Eds.), The Routledge Handbook of Research Methods in Applied Linguistics (pp. 15-28). Routledge. https://doi.org/10.4324/9780367824471-2

Marsden, E., Mackey, A., & Plonsky, L. (2015). The IRIS Repository: Advancing Research Practice and Methodology. In A. Mackey & E. Marsden (Eds.), Advancing Methodology and Practice (pp. 1-21). Routledge.

Matthews, J., & O'Toole, J. M. (2015). Investigating an innovative computer application to improve L2 word recognition from speech. Computer Assisted Language Learning, 28(4), 364-382. https://doi.org/10.1080/09588221.2013.864315

Ministère de l'Enseignement supérieur et de la Recherche. (2022). Mettre en œuvre la stratégie de non-cession des droits sur les publications scientifiques (Ouvrir la Science). https://www.ouvrirlascience.fr/mettre-en-oeuvre-la-strategie-de-non-cession-des-droits-sur-les-publications-scientifiques

Mizumoto, A., & Plonsky, L. (2016). R as a Lingua Franca: Advantages of Using R for Quantitative Research in Applied Linguistics. Applied Linguistics, 37(2), 284-291. https://doi.org/10.1093/applin/amv025

Morgan‐Short, K., Marsden, E., Heil, J., Issa II, B. I., Leow, R. P., Mikhaylova, A., Mikołajczak, S., Moreno, N., Slabakova, R., & Szudarski, P. (2018). Multisite Replication in Second Language Acquisition Research: Attention to Form During Listening and Reading Comprehension. Language Learning, 68(2), 392-437. https://doi.org/10.1111/lang.12292

Open Science Collaboration. (2015). Estimating the reproducibility of psychological science. Science, 349(6251), aac4716. https://doi.org/10.1126/science.aac4716

R Core Team. (2022). R: A language and environment for statistical computing. R Foundation for Statistical Computing. http://www.R-project.org

Sailer, M., Hense, J. U., Mayr, S. K., & Mandl, H. (2017). How gamification motivates: An experimental study of the effects of specific game design elements on psychological need satisfaction. Computers in Human Behavior, 69, 371-380. https://doi.org/10.1016/j.chb.2016.12.033

Siew, K. (2017). The open science movement. The Physiological Society. https://www.physoc.org/magazine-articles/the-open-science-movement-revolution-is-underway/

Silberzahn, R., Uhlmann, E. L., Martin, D. P., Anselmi, P., Aust, F., Awtrey, E., Bahník, Š., Bai, F., Bannard, C., Bonnier, E., Carlsson, R., Cheung, F., Christensen, G., Clay, R., Craig, M. A., Dalla Rosa, A., Dam, L., Evans, M. H., Flores Cervantes, I., … Nosek, B. A. (2018). Many Analysts, One Data Set: Making Transparent How Variations in Analytic Choices Affect Results. Advances in Methods and Practices in Psychological Science, 1(3), 337-356. https://doi.org/10.1177/2515245917747646

Simmons, J. P., Nelson, L. D., & Simonsohn, U. (2011). False-Positive Psychology: Undisclosed Flexibility in Data Collection and Analysis Allows Presenting Anything as Significant. Psychological Science, 22(11), 1359-1366. https://doi.org/10.1177/0956797611417632

Swartz, A. (2008). Guerilla Open Access Manifesto. http://archive.org/details/GuerillaOpenAccessManifesto

Tufte, E. R. (2001). The Visual Display of Quantitative Information (2nd edition). Graphics Press USA.

VanPatten, B. (1990). Attending to Form and Content in the Input: An Experiment in Consciousness. Studies in Second Language Acquisition, 12(3), 287-301. https://doi.org/10.1017/S0272263100009177

Waligora, M., Dranseika, V., & Piasecki, J. (2014). Child's assent in research: Age threshold or personalisation? BMC Medical Ethics, 15(1), 44. https://doi.org/10.1186/1472-6939-15-44

Wang, Y., & Treffers-Daller, J. (2017). Explaining listening comprehension among L2 learners of English: The contribution of general language proficiency, vocabulary knowledge and metacognitive awareness. System, 65, 139-150. https://doi.org/10.1016/j.system.2016.12.013

Wicherts, J. M., Bakker, M., & Molenaar, D. (2011). Willingness to Share Research Data Is Related to the Strength of the Evidence and the Quality of Reporting of Statistical Results. PLOS ONE, 6(11), e26828. https://doi.org/10.1371/journal.pone.0026828

Zoghlami, N. (2016). La compréhension de l'anglais oral (L2) : Processus cognitifs et comportements stratégiques. Recherche et pratiques pédagogiques en langues de spécialité. Cahiers de l'Apliut, 35(1). https://doi.org/10.4000/apliut.5322

Haut de page

Notes

1 The failure to reproduce the results of many published studies (originally in psychology), undermining confidence in current scientific methods and results in human sciences.

2 The CC-BY Creative Commons Licence “allows users to distribute, remix, adapt, and build upon the material in any medium or format, so long as attribution is given to the creator. The license allows for commercial use.” https://creativecommons.org/about/cclicenses/

3 The Open Science Framework repository serves a similar purpose, but for science in general: osf.io.

4 The extent to which results are distributed (un)symmetrically around the mean, i.e. there is a longer ‘tail’ on one side of the mean than on the other.

5 New software programs such as JASP (https://jasp-stats.org/) or Jamovi (https://www.jamovi.org/) are also used, as well as other programming languages (Python), but the extent of their use has not been documented yet.

6 Because of the Covid-19 pandemic, only the first two years were actually implemented.

7 e-FRAN (Espaces de formation, de recherche et d'animation numérique) is a research funding program started by the French government in 2017, as part of the PIA-2 (“investing for the future”) investment plan.

8 Because the actual code for the iteration presented in the article is not very informative, the example below comes from another iteration of the experiment (Charles et al., 2022).

9 The fact that the scale is not shown in full in figure 2 might also give the impression that the improvement (and the difference between the two groups) is greater than it really was.

10 A full list of credits can be found here: https://wiki.lezinter.net/_/LUCIOLE:Crédits

Haut de page

Table des illustrations

Titre Figure 1 – Visualization of results at pre- and post-test with a bar plot.
URL http://journals.openedition.org/alsic/docannexe/image/7486/img-1.png
Fichier image/png, 5,3k
Titre Figure 2 – Visualization of results at pre- and post-test with a line graph.
URL http://journals.openedition.org/alsic/docannexe/image/7486/img-2.png
Fichier image/png, 5,1k
Titre Figure 3 – “Data-rich” visualization of the results at post-test (boxplot).
URL http://journals.openedition.org/alsic/docannexe/image/7486/img-3.png
Fichier image/png, 12k
Titre Figure 4 – “Data-accountable” visualization of the results (stripchart boxplot).
URL http://journals.openedition.org/alsic/docannexe/image/7486/img-4.png
Fichier image/png, 6,6k
Titre Figure 5 – “Data-accountable” visualization of the results (line graph).
URL http://journals.openedition.org/alsic/docannexe/image/7486/img-5.png
Fichier image/png, 247k
Haut de page

Pour citer cet article

Référence électronique

Marie-Pierre Jouannaud, « Using R to Apply Open Science Principles in Second Language Acquisition and Didactics »Alsic [En ligne], Vol. 27, n° 1 | 2024, mis en ligne le 31 août 2024, consulté le 16 décembre 2025. URL : http://journals.openedition.org/alsic/7486 ; DOI : https://doi.org/10.4000/12a6r

Haut de page

Auteur

Marie-Pierre Jouannaud

After working as a language teacher and teacher trainer for many years, Marie-Pierre Jouannaud defended her PhD in didactics in 2021. She is now an assistant professor of second language acquisition in the English department of Université Paris 8. Her research interests within applied linguistics include language teaching methods and language assessment, with a focus on listening comprehension and lexical acquisition. She is also interested in the development of digital resources for language teaching, from elementary to university level.
Affiliation : université Paris 8, Saint-Denis, France.
Courriel : marie-pierre.jouannaud@univ-paris8.fr
Toile : https://transcrit.univ-paris8.fr/-JOUANNAUD-Marie-Pierre-MCF-644-
Adresse : laboratoire Transcrit, université Paris 8, 2 rue de la liberté, 93200 Saint-Denis, France.

Articles du même auteur

Haut de page

Droits d’auteur

CC-BY-NC-ND-4.0

Le texte seul est utilisable sous licence CC BY-NC-ND 4.0. Les autres éléments (illustrations, fichiers annexes importés) sont susceptibles d’être soumis à des autorisations d’usage spécifiques.

Haut de page
Rechercher dans OpenEdition Search

Vous allez être redirigé vers OpenEdition Search