1. Introduction

1In documenting linguistic variation, phonetic analyses typically use non-spontaneous elicitation strategies, in order to guarantee high quality, carefully controlled data. However, in the field, this is not always practical; sound quality, speech register, and social setting are just a few of the numerous considerations affecting the quality and accuracy of phonetic documentation. Thus, the driving force behind this work is two-fold: firstly, this work describes a strategy for performing phonetic analyses (specifically vowel-space-related research) using natural speech recordings that better represent real language usage. Secondly, at heart, this work is a documentation project. Pure phonetic documentation is often overlooked by researchers when documenting a language on the verge of extinction, prioritizing phonological and higher level (morphology, syntax, etc.) analysis. Thus, this work presents a way to perform phonetic documentation of vowels in any language, using speech (natural or elicited) recorded for other research purposes. This can be an asset in less-documented languages.

2We describe a methodology for measuring the frequency space for any given vowel, as a subset of a language’s vowel space. Typically, this will be presented within the confines of the classic vowel trapezoid, as a 2-dimensional space defined by linear functions as edges. This methodology applies various data scraping and pre-processing techniques, as well as a machine learning strategy incorporating an artificial neural network model. This process is one of quasi-unsupervised machine learning—there is a need for a small ‘seed’ of labeled data in order for a preliminary model to be generated, but this is followed by numerous unsupervised training rounds, with data being added to the set of labeled data based on the previous round of training. While we demonstrate this for one vowel, the process can be repeated for each vowel phoneme that has been identified in the language.

3In this case study, a variationist phonetic analysis of the minority and little-documented Faetar language is performed. The space of the tense high front vowel [i] is measured in two speaker groups: homeland (native speakers from the town of Faeto, Italy), and first-generation immigrants, who are heritage speakers in Toronto, Ontario, Canada. The audio data comes from the Heritage Language Variation and Change Corpus (HLVC), directed by Dr. Naomi Nagy (University of Toronto).

4Before elaboration of the above topics, a few terms must be defined. A language is considered to be a group of one or more speaker groups. It is defined as an arbitrarily-delineated group of speakers whose trends are being analysed. Although this is a rather broad definition, when considering the scope of this work, it allows for groups to be quantitatively compared; for example, in the Faetar analysis which follows, two speaker groups (homeland speakers and first-generation immigrant or heritage speakers) are compared. Hereafter, language will only be used to describe the two groups of speakers inclusively. Thus, a speaker of the Faetar language refers to any speaker from either group analyzed.

5A brief description of Faetar is also in order. Faetar is a Francoprovençal-based dialect spoken in two small villages in the south of Italy, by < 700 speakers (Nagy, 1996, 2000). In addition, there are small communities of Faetar speakers in a number of North American cities, but we estimate fewer than 1,000 speakers exist overall (Kasstan & Nagy, 2018; Zulato et al., 2018). There is limited documentation of this language: a bibliography exists at <http://projects.chass.utoronto.ca/​ngn/​HLVC/​1_6_refs_Faetar.php>. Thus, it serves as a fine example of a language whose degree of documentation can be significantly increased by automated approaches.

1.1. Neural Networks

6The design of the multi-layer neural network used in this work is not extremely complicated or novel—the construction of the network is based on Boersma, Benders, and Seinhorst’s (2012) work on the application of neural networks to solve phonological problems. This application varies from the above work, as this is a documentation project at its roots; it is not an attempt to create a generative phonological model using machine learning. However, the essence of the data structuring (multiple layers of synapse vectors, arranged as a matrix) and training process (back-propagation) is similar. It should be noted that the concept of neural networks in the machine learning field is rather old (i.e. Schidhuber first published a polished version of multi-layered neural networks in 1992), but Boersma, Benders, and Seinhorst’s (2012) work was pioneering in applying machine learning to a purely phonetics-phonology interface study.

1.2. The Faetar language

7The fact that the phonetics of Faetar is not extensively studied is exciting, as it allows a first analysis to be performed on the phonetic nuances between different generations of speakers. The previous literature related to this project is phonological in nature (as seen in the aforementioned bibliography of the HLVC project), but it is still valuable to this analysis in that it provides a general picture of the phonological vowel landscape of Faetar, from which phonetic analysis can begin. Relating specifically to the vowel in question, [i], Nagy (1996) shows that the phoneme /i/ exists in the vowel inventory of Faetar, but also that there may be a conditioned lax allophone [ɪ], along with the expected [i]. However, when hand-coding seed data before training, only [i] was considered as a valid sample.

2. Methodology

2.1. Terminology (what is a vowel?)

8The following machine learning-related definitions will be used. Tagged and untagged data refer to data points which are known or not known to be a certain vowel, respectively. In this work, tagged data is hand-coded by the first author (not a speaker of Faetar), who listened to the data and decided which portions of the audio-file represented the vowel in question, using the phonetic transcriptions produced by the second author (an L2 speaker of Faetar) for guidance.

9A vowel is any unit of speech which shows a lack of constriction, which implies five distinct and well-defined formant bands. Also, only vowels which are the nucleus of their respective syllables are considered; this is discovered by looking for relative maxima in the intensity curve of a syllable. More precisely, a vowel sample will be mathematically defined as a vector of five n formants, in Hertz:

v = [f₁ f₂ f₃ f₄ f₅]

10In this work, the first 5 formants are considered from a training perspective, but only the first two are visualized within the trapezoid, as f₁ and f₂ correspond roughly to height and frontness of a vowel, respectively. Also, a valid sample is one which is assumed to represent the vowel in question.

11The vowel set of any given vowel is defined as the set of all vowel samples considered to be a member of this set, decided through either hand-coding or testing against a model. Furthermore, the vowel space is defined as the convex hull bound by a set of k linear functions. This is generated by creating the polygon with the largest area, such that all its vertices are convex and belong to a measured sample. A vowel space in a speaker group is not considered to be unique, as vowels can (and almost always do) overlap in natural speech, due to conditioned allophones. However, in this work, overlapping phonemes are not considered—only [i] is examined.

2.2. Methodology overview

12Data collection was conducted (in the 1990s in Faeto and around 2009 for the heritage variety spoken in Toronto) for other purposes. The method used is that of the standard sociolinguistic interview (Labov, 1984), designed to elicit and record relaxed spontaneous speech in surroundings comfortable to the speaker. In particular when working with minoritized and un(der)codified languages, the differences in speech elicited through this vs. more formal methods can be large. Extracts are taken from digitized version of the Faeto audio recordings (originally made on cassettes) and the raw digital audio recordings made in Toronto. 167 speakers, representing each speaker group as shown in the table below, are analysed.

13From these 115 hours of recorded speech data, all samples containing 5 formants are extracted, at a rate of 100 samples per second. From this, a subset of 100 samples is hand-coded as either the vowel in question or not; it is on this hand-coded sample that the neural network model is initially trained. Once the model is trained using the hand-coded data, the set of untagged vowel samples is analyzed by the neural network, which gives a score between 0 and 1; samples scoring closer to 0 are less likely to be samples belonging to the vowel in question, while scores closer to 1 indicate that the sample likely belongs to the vowel in question. After this first round of testing the entire dataset, samples scoring above a very high threshold are added to the set of valid samples, and then the neural network model is retrained on this slightly larger training set. This is repeated, until no other samples are found with a high enough confidence level to merit inclusion.

2.3. Data scraping and pre-processing

14In order to scrape the formants from the raw audio files, the acoustic analysis software Praat (Boersma, 2001) was used to extract the first five formants (assuming they existed) from samples taken every 1/100th second across the entire duration of each recording. As this produces millions of data points, they must be cleaned and vowel samples extracted. One of the largest challenges encountered in this work is automatically deciding what is and is not a vowel. Depending on the language being analysed, there may be many consonant phones showing similar acoustic patterns to vowels (semi-vowels, liquids, and rhotic consonants). Thus, this is accomplished using a two-pronged strategy: naive quantitative filtering and machine learning.

15Firstly, most samples can be excluded simply by looking for a few known vowel ‘markers’, the presence of many (in this case 5) formants, formant continuity (across a timespan of 10 msec), being the nucleus of the syllable as indicated by relative intensity (Arciuli et al., 2013), and not showing known trademarks of certain non-vowel consonants. To discover whether a sample showed enough formants, Praat was used to measure formants and any sample showing fewer than five was discarded. Also using Praat, formant continuity was measured by considering eight samples around the sample in question; four before and four after. If, for each formant, the frequency difference (in Hertz) between the sample in question and the same formant of another sample in the window was greater than 1 standard deviation, the sample in question was discarded. A consequence of this is that only longer monophthongs were considered; any vowel not showing continuity in the 9/100th second frame was likely discarded.

16In order to find the nucleus of a syllable, samples were only considered if they appeared near the maximum intensity of their syllable. As seen in Ladefoged (2001), typically only vowels and syllabic consonants appear as syllable nuclei, thus this method of filtering is valuable in vowel extraction. Finally, there are certain known phenomena which can be used to filter. For example, [r] typically shows a low third formant (< 2000 Hz) while [l] often has marked separation between F₂ and F₃ (Johnson, 2012). Using these cues, phonemes that show spectrographic similarities to vowels can be filtered.

17The structure of the machine learning algorithm also leads toward the discovery of vowels more generally in the sample. As the hand-coded data only contains valid vowel samples, and the learning process only allows for samples showing similar patterns to known samples to be trained upon, any sample which is not a vowel should show a low score after running through the neural network, and thus not be considered for the next round of training.

2.4. Neural network design

18The neural network design was not extremely complex; it was an n-layered network of k synapses each. The design is similar to the illustration in Figure 1, in which each input node represents a formant, each hidden node represents a synapse, and the output node is a value between 0 and 1, signifying the suitability of the input sample being described by the trained model. The number of layers and synapses was variable; as typical in the previous literature. This was optimized to be large enough to provide accurate results when testing with the hand-coded data, but not so large as to be computationally unwieldy. In the case study below, a balance of 12-15 layers and 150-170 synapses in each gave the best results while still being relatively quick to compute. ‘Quick’ in this sense is on a scale of hours to days—with the larger Homeland dataset, a typical round of training with the above parameters took approximately 12 hours. Using more modern hardware (i.e. GPU acceleration), this practical time limitation could be minimized. The process is written in Python, based around Scikit-learn’s (2011) neural network implementation. NumPy (Hugunin, 1995) was used for pre-processing and practical data structuring.

2.5. Training process

19The neural network was trained iteratively, with each round incorporating the highest-scoring samples from the previous round. A round contains two steps: a training phase, and a testing phase. The first phase involves taking 90% of the currently valid known samples, and training the neural network through back-propagation, as described in Boersma, Benders, and Seinhorst (2012). The labeled training data in each round contains the hand-coded data, as well as any very high-scoring sample from previous rounds. In the testing phase, the other 10% of the scraped data (after pre-processing) is run through the neural network model and is scored based on suitability within the model. As previously stated, very high-scoring samples are added to the training data for the next round, and they are weighted based on distance from the hand-coded data (with a lower weight the further away the samples are). Once the accuracy of the 10% test set falls below a certain threshold (90% accuracy in this study) the algorithm is terminated and the convex hull is generated using the samples in the last training set. Using this similarity cut-off causes the algorithm to select all phonetically-similar samples, but not cross any gaps in data. Thus, it is relying on the dataset to show enough distinctness in vowel clusters in order to find an accurate vowel space; in testing, this problem is solved by using the aforementioned intensity cut-off to find the nucleus of each syllable.

3. Faetar case study

20This project relies on large amounts of data for training and testing, thus audio quality was not a main concern when selecting data. The entire set of Faetar recordings contained within the HLVC corpus was used. This amounted to the number of hours of audio for each speaker group shown in Table 1.

Table 1. – Data sample.

Speaker Group	Speakers	Hours
Homeland	123	90
1st Generation Heritage	44	35
Total	167 (but check)	115

21Although there is a discrepancy between the number of speakers and hours for each speaker group, this does not cause an issue with analysis, apart from the smaller dataset being slightly less likely to show certain nuances regarding the vowel space.

22After naively scraping for vowels (but before neural network training), the following counts of potential vowel samples were found:

Table 2. – Number of vowel samples per group.

Speaker Group	Likely Vowel Samples
Homeland	8,077
1st Generation Heritage	2,607
Total	10,684

23These take on the typical vowel space triangle shape when graphed as F1 x F2, as seen in Figures 1–4, representing different speaker group estimates at different points in the training process. In this visualization, [i] appears in the top-left, [a] in the bottom, and [u] in the top-right. The yellow points represent the hand-tagged [i] data, along with a yellow convex hull delimiting the hand-tagged homeland vowel space. The red points represent the potential vowel samples (for all vowels), and the enclosing red convex hull represents the total vowel space observed. This distribution does not differ markedly from the expected. For example, it is quite similar to the sample of measurements shown for southern speakers of standard Italian, a geographically proximate variety, in Renwick and Ladd (2016).

24The concentration toward the centre is due to the fact that unstressed vowels in natural speech tend toward the central vowel [ə] (schwa). Moreover, it is likely that, due to naive nature of the vowel scraping, a fair proportion of samples are not actually vowels; rather, they could be glides, semi-vowels, liquids, etc. However, this is the purpose of the machine-trained neural network filtering. As it is hand-trained on a sample including only vowels, the hope is that only the actual vowel samples are discovered. It can be hypothesized that the green convex hull tends far from the centre due to the hand-tagging process. Naturally, when selecting samples, a sample that sounds very ‘in-the-category’ is more likely to be selected, compared with a phonetically-borderline sample. Thus, it is fair to say that the following results are biased toward a conservative representation of the vowel space.

25After 5 rounds of training, the convex hull has expanded, as shown in Figure 2.

26And then, after 14 rounds of training, when the algorithm finished (due to the arbitrary setting of 14 rounds), the vowel space of [i] is as shown in Figure 3.

27Thus, by this methodology, the yellow convex hull in Figure 3 marks the boundaries of a conservative [i] in Homeland Faetar.

28Repeating the same process for first-generation heritage speakers, the Figure 4 is produced after 11 rounds of training, with the blue convex hull marking the [i] vowel space.

29And finally Figure 5 shows the two speaker groups superimposed. The red convex hull represents the Homeland total space, with the Homeland [i] represented by the yellow hull. The orange and blue hulls represent the respective vowel space and [i]-space for first generation heritage speakers.

30We note first an interesting shift in the shape of the overall vowel space—it becomes more trapezoidal (like Canadian English with the two low vowels /æ/ and /ɑ/), while the Homeland shape is more triangular, with only one low vowel, /a/. This Heritage vowel space shape is not unlike that of Toronto English, for example as shown in Roeder and Hunt Gardner (2013: 165). This potential evidence of a language contact effect is echoed when we zoom in on the shape of the convex hull for [i] in the two speaker groups: in the first-generation data, a new cluster appears at F1 = ~850 Hz, F2 = ~1750 Hz. This articulates close to a North American English [ε], although slightly further back, not unlike the location of /ɪ/ for Toronto English speakers undergoing the Canadian shift (ibid.).

4. Discussion

31Based on the plot in Figure 5, it appears that in the Heritage speaker variety, [i] is articulated slightly further back in the mouth (i.e. F2 is lower), and with a wider F1 range. Hypothesizing why this occurs, two possibilities come to mind. Firstly, Heritage speakers are pronouncing [i] slightly further toward the back of the mouth, possibly as a result of contact-induced language change, as noted in the previous section. This phenomenon could be caused by a merger of [i] and its lax variant [ɪ], causing a loss of distinction between the two. Or, the above results could stem from a simple lack of data—it could be a quirk of the individuals studied.

32The first hypothesis is based on the observation of lax allophones in Faetar (Nagy, 1996), which mentions a lax variant of /i/, realized as [ɪ], in Faetar. Thus, the expansion of [i] to include [ɪ] in first-generation Heritage speakers could be due to a merger of [i] with its potentially-attested lax variant, [ɪ]. As the [i] vowel space seen in G01 speakers reaches lower and further back, it includes much of range of the ‘typical’ [ɪ]. Thus, with the distinction between the two becoming less pronounced, the machine-trained neural network model includes both [i] and [ɪ] as similar to the hand-tagged [i]. Although great care was taken in attempting to hand-tag both the heritage and G01 [i] seed samples identically, [i] tagged from G01 is pronounced slightly (nearly imperceptibly) further back.

33Alternatively, the possibility of the classic machine learning criticism must be addressed: not enough data. With any machine learning-based methodology, the training data must be accurate in order for the algorithm to succeed (the “garbage in, garbage out” phenomenon). In this case, the seed data was not tagged by a native speaker of Faetar; rather, it was tagged by a native speaker of North American English. Such allowances are often necessary in work with endangered languages: there is not always an overlap between speakers and those with the time, training and resources to conduct computer-aided research. Thus, although care was taken to avoid tagging errors, the possibility of an implicit bias is present. However, unless the [i] produced by a native Faetar speaker is radically different than the [i] of a native speaker of North American English, the results should be accurate. It is possible that, with more data, the pattern seen in this work could be levelled out. Again, this is often the unavoidable situation in endangered language documentation work.

5. Future Directions and Conclusion

34The methodology described in this work lies firmly in the realm of starting points. Although it appears to accurately document the vowel space of a specific vowel in two speaker groups, more testing must be done to confirm its accuracy. To increase the confidence of this work, the hand-tagging should optimally be done (or checked) by a native speaker of the language in question. Phonetic perception is inherently skewed toward one’s native language, which could cause inaccuracy in the trained model (and this would be more extreme if the vowel in question were not found in the inventory of the tagger’s native language). Secondly, typical across any data-oriented methodology, a bigger sample leads to more representative results. However, regardless of the above points, it is an area that will benefit from more research and case studies—using this methodology, phonetic documentation and variationist analysis of vowel space could be automated, preserving and expanding linguistic knowledge at minimal cost and time to researchers.