These authors contributed equally.
§1: Elia Scapini and Federico Iezzi; §2: Federico Iezzi; §3,1: Federico Iezzi; §3,2: Elia Scapini e Federico Iezzi; §3,3-4: Elia Scapini; §5: Elia Scapini and Federico Iezzi.
1This paper aims to explore the application of computational linguistics techniques to semantic tasks in Ancient Greek, presenting an investigation of the state of the art in this field, with a particular focus on PLMs based on the transformer architecture. Although these models are inherently multitasking and thus adaptable to various Natural Language Processing (NLP) tasks, their application to semantic analyses of Ancient Greek texts remains underexplored. By critically examining existing transformer-based PLMs, this study also aims to observe their current capabilities and explore their potential to contribute to semantic research in the context of Ancient Greek.
- 1 Since this review focuses only on encoders and does not include attempts to apply GPT-like models t (...)
2We focus here on those PLMs that are deep, multi-layered neural networks specifically trained for language understanding tasks. These models are generally classified as Large Language Models (LLMs) when they exceed 6–10 billion parameters (Liu et al. 2025). Depending on the specific language processing tasks, these models can employ either an encoder-only architecture or a combined encoder-decoder architecture1. According to this standard, we can outline the structure of our contribution as follows. After the introduction (sec. 1), we describe the rationale behind distributional semantics and sentence embedding, showing how it has led to the development of Vector Space Models (VSMs) for Ancient Greek (sec. 2). Then, we illustrate the main Distributional Models in Semantic Scholarship on Ancient Greek (sec. 3) firstly numbering context-count, context-predict and bayesian projects, switching secondly to transformer-based models and finally opening the issue of the benchmark. We then discuss advantages and limitations of statement retrieval performed with cosine similarity (sec. 4). We conclude by cautiously suggesting that transformer-based models and statement retrieval using cosine similarity, despite their limited transparency to human users, still provide valuable tools for scholarly work.
- 2 Quoting Firth and Harris has become almost a ritual practice. For a deeper discussion and compariso (...)
3The core principle that semantically related words tend to co-occur in shared contexts inspires all the techniques of distributional semantics. This discipline aims to represent the relatedness of terms through spatial proximity, where geometric closeness reflects their semantic similarity. Such representations enable NLP systems to tackle a variety of tasks. "Similarity" must be understood in the broadest sense possible; it encompasses not only synonymy and antonymy but also the broader, self-implicating relationships between words. The rationale behind this approach is to consider terms not as isolated units but as dense tensors. This idea traces back to the principle famously summarized by Firth (1957) as «you shall know a word by the company it keeps» and to the view of Harris (1954) that language is organized within a distributional structure2.
- 3 For a comprehensive overview of the development of VSMs for semantic purposes, from their origins i (...)
- 4 An overview on the history of the development of these tools to state-of-the-art transformers is pr (...)
4Efforts to apply this principle to concrete algorithms for measuring semantic similarity often lead to the use of vectors, matrices, and higher-order tensors. According to Turney and Pantel (2010)3, the intimate connection between the distributional hypothesis and VSMs was a strong motivation to closely examine VSMs. In their survey, they classified previous work on VSMs based on the type of matrix involved: term–document, word–context, and pair–pattern. Although these three types of matrix cover most of the existing work, Turney and Pantel did not believe they exhausted the possibilities. They anticipated that future research would introduce new types of matrices and higher-order tensors, expanding the potential applications of distributional semantics. They concluded their paper by openly stating that they expected the introduction of new types of matrices and more complex tensors to open up new applications for VSMs. In this sense, their prediction was partially realized. In fact, just a few years after the publication of their work, one of the most promising techniques in the field of distributional semantics was developed: embeddings. With "embedding", a technique is meant that transforms words (and sentences) into multi-dimensional vectors, statistically encoding information about grammar, morphology, syntax and context in a tensor made of a fixed quantity of real numbers. Embeddings are usually distinguished between "static" and "dynamic"4.
5Static embeddings, either calculated with context-count models or predicted with neural networks (Baroni, Dinu, and Kruszewski 2014), are unequivocally assigned to a vocabulary word, regardless of its contextual behavior. These can be generated using tools like Word2Vec proposed by Mikolov et al. (2013a, 2013b). Word2Vec is a neural network-based model that maps words to a continuous vector space, where semantically similar words are represented by similar vectors. The two main architectures of Word2Vec are as follows:
-
Skip-gram, which predicts the context words given a target word;
-
CBOW (Continuous Bag of Words), which predicts a target word given its surrounding context words.
6This ability to "predict" often allows the model to outperform so-called context-count models in many NLP tasks, particularly those related to semantics (Baroni, Dinu, and Kruszewski 2014). However, this is not always the case, as shown by Lenci et al. (2022), who provide a comparative evaluation of approaches. For Ancient Greek specifically, see also Stopponi et al. (2023) for an assessment of distributional semantic models. It is also true that, the outputs of context-count models remain more interpretable and transparent (Keersmaekers and Speelman 2023). Word2Vec has nonetheless revolutionized computational linguistics by enabling highly precise word embeddings in large corpora and facilitating the use of machine learning models for complex language tasks.
7The year after the launch of Word2Vec, Stanford researchers introduced GloVe (Pennington, Socher, and Manning 2014). GloVe, which stands for Global Vectors for Word Representation, is similar to Word2Vec but relies on a word co-occurrence matrix rather than a neural network. This matrix captures how words co-occur within a corpus of text, making GloVe particularly effective at encoding global information about the relationships between words.
8In 2016, FastText (Bojanowski et al. 2017), a model that represents words as sequences of character subsets (n-grams), was introduced. This approach enables FastText to handle rare or unknown words (out-of-vocabulary) more effectively by generating embeddings for unseen words based on their constituent parts. Thanks to this capability, FastText has significantly influenced the development of models for languages with complex morphologies or numerous compound words.
9Dynamic embeddings, on the other hand, provide different vector representations for the same word depending on its context and are most effectively generated using PLMs based on Deep Neural Networks. For this reason, dynamic embeddings are often referred to as "contextual" embeddings. These embeddings are closely associated with the transformer architecture introduced in Attention is All You Need (Vaswani et al. 2017). The transformer employs self-attention mechanisms that allow it to evaluate the importance of each word in relation to others, regardless of their order in a sentence. This represents a major advancement over older models like RNNs (Recurrent Neural Networks) and LSTMs (Long short-term memory), which process words sequentially and struggle with long-range dependencies. By processing words in parallel, the transformer produces rich, context-sensitive word representations. This architecture, first implemented by Devlin et al. (2019) in models like Bidirectional Encoder Representations from Transformers (BERT), has been pivotal for generating dynamic embeddings that adapt to surrounding words, significantly outperforming traditional word embeddings such as Word2Vec on many NLP tasks. Moreover, these models have advanced not only general language tasks but also research in the study of ancient languages. The implications of these developments for semantic tasks in Ancient Greek language processing will be examined in the following sections.
- 6 Wishart and Prokopidis (2017) adapted a PoS tagger and lemmatizer for Hellenistic Greek and analyze (...)
- 7 However, it is important to highlight the study by Keersmaekers (2020), which integrated the use of (...)
10It is interesting to note that the first use of NLP for Ancient Greek involved a Part-of-Speech (PoS) tagger created to assist students in learning the language (Packard 1973). Only later was the potential of computational resources for advanced research recognized, with applications extending to historical linguistics and philology. In any case, it should be noted that research applying and investigating semantics in this area remains relatively scarce, leaving many opportunities for further exploration within computational linguistics. Some studies have focused on the identification of hidden topics within a corpus of documents using statistical models like LDA (Latent Dirichlet Allocation)6. However, the primary application of distributional models in the semantic field for Ancient Greek has been the study of semantic change in words7.
11To our knowledge, the first attempt to apply distributional semantics to Ancient Greek (specifically using a context-count model) for the study of semantic changes in the language over time was made by Boschetti (2010) 8. In the third part of his doctoral thesis, the author focused on the study of semantic spaces. Using a varied corpus, both in terms of genre and chronology (8th century B.C. – 15th century A.D.), Boschetti examined the meanings of words across different literary genres and observed semantic changes over time through the calculation of cosine distance between the vectors associated with words. His study deserves recognition for pioneering the application of distributional semantics to Ancient Greek. However, the considerable variety of literary genres in the corpus introduces a significant amount of noise. Similarly, Rodda, Lenci, and Senaldi (2017) employed distributional models that utilized Positive Pointwise Mutual Information (PMI) to analyze the semantic lexical change in Ancient Greek. Their study used a smaller corpus than Boschetti’s (7th century B.C. – 5th century A.D.), divided into two sub-corpora: one representing the pre-Christian era and the other the Christian era. Applying representational similarity analysis (RSA) to measure semantic changes, the study identified the extent to which certain terms shifted in meaning over time, shedding light on the influence of Christianity on the lexicon. Also the doctoral thesis of Grewcock (2018) falls within this framework, aiming – through the use of a context-count model – to identify patterns, structures and variations in the syntax and semantics of movement verbs. On the other hand, Homeric formulas were the focus of the analysis of Rodda, Probert, and McGillivray (2019). This paper described the creation of a vector space model and its validation against a new benchmark9 introduced by the authors. The benchmarks were derived from scholarship from the ancient world, modern lexicography, and an NLP resource. Finally, to demonstrate the applicability of the model, authors use the model for a study on epic formulae to track semantic variation.
12On the other hand, regarding context-predict VSMs, Burns (2019) must be recognized as a forerunner. He presented an experimental model based on Word2Vec in version 0.1 of the CLTK (Classical Language Toolkit), which would later merge into version 1.0 of the CLTK (Johnson et al. 2021), offering a standard API (application programming interface) and pre-configured processing pipelines for several ancient languages. Another example of the application of Word2Vec is List (2022), who used vector spaces to examine kinship terms, demonstrating how the distributional analysis of words can contribute to the creation of lexical (dictionary) entries by suggesting relevant meaning associations to the target word. The author tests the method on three kinship terms. The embedding technology, along with other resources, plays also a central role in the recent project funded by CLARIN A New CLARIN Resource Family for Lexical Semantic Change Research10 (McGillivray, Khan, and Marongiu 2023). The project’s aim is to centralize essential resources for lexical semantic change research. These resources include datasets annotating word meanings, word embeddings derived from diachronic corpora, automated algorithms to detect semantic changes and lexical resources. In January 2024, Marongiu, McGillivray, and Khan (2024) published a paper describing a workflow example for the application of the tools provided by CLARIN to English, Ancient Greek and Latin. A few months later, Stopponi et al. (2024) stated that they were following a workflow that largely overlaps with that defined by Marongiu and her colleagues. First, they divided the Diorisis corpus (Vatri and McGillivray 2018) into five time windows. They then applied two measures of lexical semantic change detection to Word2Vec embeddings trained on the selected corpus: VC (Vector Coherence) and J (Jaccard). The analysis revealed that both measures were effective at identifying stable words, with Vector Coherence being more reliable at detecting changed words. However, a low Jaccard score also indicated semantic change when combined with low Vector Coherence. Both measures were affected by lemmatization errors in the training corpus, which hindered the detection of changed words11. Other attempts to study semantic change in Ancient Greek have been made using Bayesian models, which, in some papers, intersect with embedding technology. Perrone et al. (2019) developed GASC (Genre-Aware Semantic Change), a Bayesian model that uses metadata about the genre of texts to improve the inference of semantic change in Ancient Greek corpora. This approach helped to distinguish between polysemy and semantic change. McGillivray et al. (2019)12, starting from the paper just described, applied the SCAN model – originally developed for English by Frermann and Lapata (2016) – to Ancient Greek, comparing it with the GASC model, which performed better. The authors, in conclusion of their paper, underlined the difficulty of the challenges related to the evaluation of computational models, particularly when comparing automatic analyses with those of experts. The paper by Perrone et al. (2019) is explicitly considered the starting point for the study of Zafar and Nicholls (2022), which further refined this framework by integrating concepts of sense and time as additive effects into their DiSC model (Diachronic Sense Change), thus achieving a significant improvement in the performance of the model. Two years later, the same authors presented EDiSC, a model that extends DiSC, combining it with word embeddings to improve the analysis of sense change (Zafar and Nicholls 2024). The main novelty consists in having integrated topic-based models and embedding-based models to optimize performance. As a final example of the use of a Bayesian model – which in this case was applied to both Greek and Latin – we report the study of Perrone et al. (2021), in which GASC was compared with embedding-based models.
13The authors of this latest work showed that Bayesian models offer explicit and easy-to-interpret representations, while embedding-based methods, despite being extremely sophisticated and widely used, are less transparent. As we will discuss later, interpretability remains a pivotal feature to negotiate with when evaluating which methodology best suits the scholar’s research purpose.
- 13 This is a natural process in the field of digital humanities, which often adapts techniques and mod (...)
- 14 Appendix A provides a summary table of the models discussed in this section.
- 15 It is common to consider the first layers as representing lower-level information, particularly rel (...)
- 16 Dimensions raise to 1024 in larger BERT models.
14Being it a language that is more computationally studied, provided with the largest textual corpus available and currently associated with a significantly larger number of resources, English naturally served as the basis for the development of transformer-based models. Techniques and solutions were often adapted to Ancient Greek13. As a result, there is a physiological latency period that separates the appearance of tools designed for English and the correspondent attempt to apply the same technique to ancient languages, including Ancient Greek14. Indeed, there was a three-year gap between the release of the first BERT (Devlin et al. 2019) and the first Ancient Greek BERT (AGB) model, released by Singh, Rutten, and Lefever (2021). Between the two models, a key intermediate step was represented by a Modern Greek BERT model (Koutsikakis et al. 2020). This tool has to be mentioned because, before training it with data from corpora such as First1KGreek Project, Perseus Digital Library, PROIEL and fine-tuning it to perform PoS and morphological tagging, Singh, Rutten, and Lefever (2021) based their model on that of Koutsikakis et al. (2020). As BERT models, they share several common features: their base structure consists of twelve transformers layers, each with its attention head15. The output representation they provide is a vector with 768 dimensions16. One distinctive aspect of their operation is the use of special tokens, among which the most relevant for the purpose of this review is the [CLS] (Classification) token. This token is added at the beginning of each input sentence and serves as a context aggregator for the entire sequence. The output vector corresponding to each [CLS] token can be used for various tasks, having captured relevant information from the entire sequence.
15The first BERT model by Devlin et al. (2019) was initially released in both an UNCASED and a CASED version. In the former, the corpus was fully converted to lowercase while in the latter uppercase letters were taken into account by the model. In contrast, the initial AGB models did not include this distinction. In fact, not only did the AGB model inherit its lack of diacritics from the Modern Greek model (Koutsikakis et al. 2020) on which it was based, but the same approach was also adopted by the models released by Yamshchikov et al. (2022) and Spanopoulos (2022).
16Yamshchikov et al. (2022) released a BERT model for Ancient Greek specifically optimized to predict authorship attribution, with a validation accuracy of around 80%. As in Singh, Rutten, and Lefever (2021), the model was not created from scratch for Ancient Greek but was derived from Modern Greek BERT17 and fine-tuned using transfer learning on a Masked Language Modeling (MLM) task. The goal in this case was not PoS tagging, but rather to clarify the attribution of three documents to Pseudo-Plutarch18. Moreover, this model was initialized with Modern Greek (Koutsikakis et al. 2020), which raises the issue of the omission of certain diacritics in Ancient Greek. Indeed, authorship attribution, like statement retrieval, is not the task for which these models were initially designed, nor is it the one where they are expected to perform best. In fact, among the tasks for which BERT encoders were originally designed, ’fill in the gap’ stands out. Typically, training data are divided into a training set and a validation set, with an approximate ratio of 80% to 20%. The validation corpus is then randomly masked by replacing some words with an anonymous tag (e.g., [MASK]), which the model has the task to predict. This technique is known as MLM. A number of training epochs are then iterated to adjust the internal weights of the neural network in order to improve the accuracy of the model’s representations.
- 19 Access to digitized and interconnected data related to historical languages plays a crucial role in (...)
17We have mentioned the model from Spanopoulos (2022): this is a RoBERTa (Robustly Optimized BERT Pre-training Approach) model that was released one year after the first AGB. Similarly, the first RoBERTa model was released around one year after BERT. RoBERTa is an encoder-only PLM first introduced by Liu et al. (2019). RoBERTa and BERT share a similar structure; however, the former often outperforms the latter due to specific adjustments in the training process, including dynamic masking and the removal of the next sentence prediction (NSP) task. Disclosing the performance of their model, Spanopoulos noted that although it performed reasonably well on PoS tagging, it showed signs of underfitting, likely caused by insufficient training data19. The lack of an adequate amount of training and validation data is one of the key challenges in developing state-of-the-art transformer-based models. This issue is particularly pronounced in the case of Ancient Greek, since the available corpus is relatively small and confined to a fixed body of texts, with no possibility of new material being produced. Furthermore, Ancient Greek poses additional difficulties due to its high degree of inflection, its considerable diachronic variation, and its substantial genre variation (Perrone et al. 2019, 2021).
18Yousef et al. (2022a, 2022b) published an XLM-RoBERTa-based model optimized for multilingual text alignment. This model was developed using aligned material from UGARIT20 (Palladino, Foradi, and Yousef 2021), a translation alignment editor. The development was informed by observations on the application of translation alignment in ancient language courses conducted at Tufts University and Furman University between 2017 and 2019. With this work, the acronym XLM, which stands for "Cross-Lingual Model", entered the field of transformer-based models applied to Ancient Greek. In fact, models can also be trained in multiple languages (Artetxe and Schwenk 2019; Lample and Conneau 2019), opening up new possibilities thanks to their ability to represent concepts in a language-agnostic way and to transfer knowledge, adapting it to new tasks. This is particularly interesting because multilingual models sometimes outperform single-language models in zero-shot configuration. This means that if a model is pre-trained to perform a specific downstream task in one language, it can adapt to a second language without requiring fine-tuning to accomplish the same task. For example, a model trained in English and fine-tuned on a downstream task for that language may, after multilingual adaptation, outperform a native Ancient Greek model on the same task. Following m(ultilingual)BERT, which supports more than a hundred languages, Lample and Conneau (2019) introduced Translation Language Modeling (TLM) and proposed training a XLM model. Later, other models such as XLM-R(oBERTa) (Conneau et al. 2020) were developed based on this architecture. The XLM-RoBERTa-based model by Yousef et al. (2022a) was created with the assumption that multiple fine-tuning steps can enhance the model’s performance on a downstream target. More specifically, they have fine-tuned multilingual model XLM-R on 12M Ancient Greek tokens with MLM and subsequently tailored the improved model on language pairs taken from UGARIT. These models, while designed for alignment and translation, can also be used for other NLP tasks, such as improving Named Entity Recognition (NER), as shown by and Yousef, Palladino, and Shamsian (2023) and Yousef (2023).
19Since PLMs are trained to model patterns in the use of a language, they can be exploited to suggest how to complete missing portions of text21. This approach has been applied to both inscriptions and manuscripts. Assael, Sommerschield, and Prag (2019), Assael et al. (2022), within the framework of the PithiaPlus project22, worked on a tool designed to restore gaps in inscriptions caused by physical damages. The first (LSTM-based) tool named PYTHIA (2019) was capable of predicting how to fill missing parts of inscription. It was then followed by ITHACA (2022)23, a model that leverages a transformer-based architecture to restore inscriptions and predict their location and date. Not only was the training performed on the largest pre-modern Greek corpus to date, but, interestingly, the authors also trained ITHACA with augmented data, i.e. by generating multiple artificial variants of the same inscription through segmentation and masking. This procedure increased the number of training examples and simulated the fragmentary nature of the sources, allowing the model to improve its ability to restore damaged texts. Serving as a bridge between ITHACA and PYTHIA was the Blank Language Model (BLM) by Shen et al. (2020), a transformer-based architecture evaluated on the dataset by Assael, Sommerschield, and Prag (2019). This model achieved similar performance to PYTHIA but stood out for its ability to generate arbitrary sequences without needing to define the target length. By the side of manuscripts, Logion BERT, announced in the concept-paper of Graziosi et al. (2023) and released by Cowen-Breen et al. (2023), was specifically customized for Greek philology. This BERT model was trained to perform a fill-gap task on errors in a long hand-copy transmission queue to achieve correction of scribal errors in manual transcription. It should be noted that none of the studies reported in this paragraph evaluated their model for semantic purposes.
20Up to this point, it is important to note that no transformer-based model among those discussed above has specifically been created from scratch for Ancient Greek. The first model trained directly on Ancient Greek was introduced by Frank (2023a) from the Computational Linguistics department of the University of Heidelberg. Their work presented four PLMs for Classics: two RoBERTa encoder-only models, GrεBERTa and PhilBERTa, and two T5 encoder-decoder models, GrεTa and PhilTa. It is the first time that we see the construction of T5 models for Ancient Greek. Raffel et al. (2020) firstly suggested this technique: aiming to adapt one model to several tasks, authors introduced Text-to-text Transfer transformer architecture (5 "T"s). These models consist of an encoder-decoder structure so that they can create intermediate representations of the input before sending it to the decoder that generates the output. This approach enables the transformation of each task into a text-to-text problem, giving generalization capacity to the model. Riemenschneider and Frank’s GrεBERTa and GrεTa are, respectively, encoder-only RoBERTa and T5 encoder-decoder trained exclusively on Ancient Greek texts. In contrast, PhilBERTa and PhilTa are the trilingual counterparts of the previous models, pre-trained on datasets in Greek, Latin and English. Since all these models were trained on a significantly larger corpus of Ancient and Medieval Greek (185.1 million tokens; see Table 1) compared to previous BERT models, and incorporate a text processing system that retains diacritics, the result is that GrεBERTa outperformed all existing models in morphological and syntactic tasks.
Table 1: The four models trained by Riemenschneider and Frank.
|
Greek
|
Multilingual
|
|
Encoder only
|
GrεBERTa
|
PhilBERTa
|
|
Encoder-decoder
|
GrεTa
|
PhilTa
|
- 24 https://sbert.net/.
- 25 Research on intertextuality in ancient texts is a field that has become more and more involved in h (...)
21In the same year, Riemenschneider and Frank (2023b) presented SPhilBERTa, a cross-trilingual model designed for Classical Philology, "distilled" from a multilingual model called all-mpnet-base-v2. Here, two new concepts are introduced: the distillation technique and the sentence model. The first term, distillation, refers to a method for adapting a model trained on a highly attested language (e.g., English) to work on less-resourced languages (e.g. Ancient Greek and Latin). This technique, described by Reimers and Gurevych (2020), involves transferring knowledge from the original model to improve performance in languages with limited data. This method operates teaching a "learning" model to assign the same vector representations to a sentence in the target language as those assigned by a "teaching" model to the same sentence in the more attested language. To achieve this, authors need to compose (and sometimes synthesize) a dataset of parallel sentences in English and Ancient Greek, and set the learning function to implement sharing of representation. Precisely because the smaller model is trained to replicate the predictions of the large model, this method has proven particularly useful for languages with limited available data. Regarding the second concept (sentence PLMs), BERT models were primarily trained to encode individual words after tokenizing them. Although it is possible to represent a sentence by summing or averaging its word vectors, or by leveraging the [CLS] token, this approach often performs poorly, especially for sentence retrieval tasks in languages with limited available data, such as Ancient Greek. To address this limitation, more recent BERT models have been enhanced to perform sentence embedding, as first suggested by Reimers and Gurevych (2019). Sentence-BERT (sBERT) models24 integrate a siamese and triple network structure into the BERT architecture, making them more efficient and less time consuming for sentence embedding tasks, such as similarity, clustering and information retrieval. SPhilBERTa, designed for Classical Philology, has been specifically tailored for cross-linguistic semantic understanding and the identification of identical sentences across Ancient Greek, Latin and English. also presented a case study on the Aeneid and the Odyssey, demonstrating the ability of SPhilBERTa to facilitate the automated detection of intertextual parallelisms25.
22In the same year, another sentence transformer-based model for Ancient Greek, SHLM-grc-en, was introduced by Krahn, Tate, and Lamicela (2023). The accompanying paper describes their distillation method, which involves adapting a larger multilingual model. They created aligned vector spaces for Ancient Greek and English, following the alignment approach introduced by Liu and Zhu (2023), creators of Bertalign. The authors then evaluated their models on STS tasks (Semantic Textual Similarity) and SR (Semantic Retrieval).
23Finally, Riemenschneider and Krahn (2024) jointly submitted a paper for the SIGTYP conference (Special Interest Group on Typology), where they presented a model that combines the hierarchical tokenization technique suggested by Sun et al. (2023) with a DeBERTa-V3 model (He, Gao, and Chen 2023). The core elements combined here that require to be introduced are two: HLM and DeBERTa-V3. We will begin detailing the latter.
24After the introduction of BERT and RoBERTa, many other models can be numbered: the architecture of DeBERTa suggested by He et al. (2021) improved RoBERTa models adding in the pre-training process the disentangled attention to improve the relative-position understanding in the model. Here, two separate vectors provide respectively the content and the position of the represented word. Still, it is pre-trained with MLM. Proposing ELECTRA, Clark et al. (2020) suggested another pre-training technique: Replaced Token Detection (RTD). Here, ambiguous corruptions in the tokens are automatically generated and the model is pushed to spot them. Therefore, ELECTRA models are trained with two transformer models (Generative Adversarial Network - GAN style), a generator that creates ambiguous tokens (MLM) and a discriminator that has to predict whether the tokens are original or were replaced (RTD). DeBERTa was further enhanced in DeBERTa-V3 with a combination of two techniques as detailed by He, Gao, and Chen (2023). The authors implemented changes to the pre-training process, switching from MLM to RTD (basically adapting it from ELECTRA models) while maintaining the relative position encoding of DeBERTa models. He, Gao, and Chen (2023) introduced a new method called Gradient-Disentangled Embedding Sharing (GDES) that enables the generator (MLM) and the discriminator (RTD) to share embeddings more efficiently. In doing so, they addressed the "tug-of-war" dynamic between generator and discriminator. "Tug-of-war" conveys the idea that, having the generator and the discriminator two different purposes, they end up shaping the embeddings according their own scope, endangering the quality. GDES allows both components to perform their respective task (to generate and detect false tokens) exploiting the same embeddings without pulling them in the opposite way.
25The second element, HLM, introduced for Ancient Greek by Riemenschneider and Krahn (2024) for the SIGTYP 2024 consists in the Hierarchical Language Model (HLM) tokenization technique suggested by Sun et al. (2023) that aims to fix the vocabulary issue in the tokenization phase, i.e. the problem that arises when subword tokenizers—especially in multilingual models—split words inconsistently across languages and scripts, producing a noisy vocabulary. The hierarchical tokenization technique proposed by Sun et al. (2023) addresses this challenge by introducing a two-level approach: it first segments the text into smaller intra-word units, applying self-attention to represent them into initial embeddings, and then refines these representations with further integration of inter-word contextual information. This approach not only enhances the handling of rare words, but also ensures better alignment of tokens across languages, which is crucial for multilingual models aiming to perform well across diverse linguistic contexts.
- 26 In each occurrence, EvaLatin poses shared tasks to scholars and teams to obtain and improve state-o (...)
26English, a widely attested language, has several benchmark datasets available for evaluating PLMs, such as GLUE (Wang et al. 2018) for sentence understanding tasks, SQUAD (Rajpurkar et al. 2016), RACE (Lai et al. 2017), and XNLI (Conneau and Kiela 2018), which is tailored for cross-lingual evaluation. On the other hand, despite the growing number of transformer-based PLMs for Ancient Greek — whether BERT, SBERT, RoBERTa, or multilingual models distilled from larger tools — much work remains to be done in creating one or more reference standards shared by scholars, which would enable a homogeneous and transparent evaluation. It would be desirable that a benchmark campaign to evaluate tools tailored for Ancient Greek started, following the positive example opened by EvaLatin, an evaluation event that has seen, in 2024, its third edition26. This necessity is accentuated by the fact that a more recent model is not necessarily a better performing model, especially since the fine-tuning adapts it to specific downstream tasks. Therefore, digital humanists might invest considerable effort and time testing and implementing models to identify those that best suit their research project. Nonetheless, as mentioned above, some projects, such as Krahn, Tate, and Lamicela (2023), provide their internal standards of self-evaluation. In a recent paper, Giuseppe Celano (Celano 2025) compared six PLMs (including GreBERTa, PhilBERTa, GreTa and PhilTA from Frank (2023a)) to evaluate which performs better regarding automatic prediction of morphosyntax and lemmatization.
27As authors of this contribution, we highlight the lack of a fair and standardized benchmark for statement retrieval across different models, i.e. the task of retrieving the most correlated texts from a corpus given a query sentence, across different transformer-based models. To date, the only attempt to establish a benchmark for PLMs in Ancient Greek is the work of Krahn, Tate, and Lamicela (2023). Their study provides an evaluation of the recall and mean average precision (mAP) for each of the models developed. In their repository27, 100 queries have been paired with corresponding relevant passages. However, we express reservations about whether this repository can be regarded as a gold standard for information retrieval in Ancient Greek. Specifically, the methodology associates each query with only a single passage, which we consider highly unlikely to exhaustively represent all semantically related passages within the corpus.
28Additionally, our efforts to advance the state-of-the-art in benchmarking statement retrieval with transformer-based PLMs for Ancient Greek and Latin are published within the proceedings of a domain-specific conference (Toyin et al. 2026).
29We can now outline the technique that allows for statement retrieval with transformer based PLMs. Scholars can already refer to rule-based tools to query an Ancient Greek corpus to retrieve statements from sources they are focusing on in their work. TLG28 is the most famous one, while Tracer29, despite being more flexible, requires more technical preparation to be used. Besides direct word retrieval, either exact matches or lemmatized forms, both TLG and Tracer offer scholars to use an n-grams retrieval tool which allows them to construct a single query by combining multiple if clauses, specifying in which shape words must appear in sequence within a defined span of n-grams.
30In this review, we focus on semantic retrieval for Ancient Greek sentences performed using PLMs. Kevin Krahn has implemented a transformer-based PLM in a retrieval interface for Ancient Greek30, which, although still in its nascent stage, provides a useful insight of the essential expected outcome of our research project. Therefore, we need to briefly explain how this approach works and how it differs from the rule-based ones.
31As a retrieval method, this approach enables users to extract information from a database which, in this case, is an Ancient Greek corpus of written sources31. Scholars can query this textual material by specifying one or more words or sentences relevant to their research. Both the corpus and the queries are encoded into multidimensional vector representations by the transformer-based model to capture morphosyntactic and semantic features. A program calculates the similarity between the query vector and each sentence vector in the corpus. Among the existing similarity measures, the most common is cosine similarity.
Figure 1
Example of a workflow for retrieving statements semantically similar to the user’s query using a transformer-based model.
32This process generates a similarity score for each sentence that can also be expressed as a percentage. Based on these scores, the sentences in the corpus are sorted in descending order of similarity, allowing researchers to prioritize the most relevant results.
33We may number some key differences between statement retrieval performed through transformer-based and rule-based verbatim retrieval tools: Firstly, corpus resorting. One can easily tell that the higher the n becomes in a n-grams query, the less results the software will retrieve since the number of if rules makes the query always more restrictive and the possible matches will decrease. This does not happen with statement retrieval performed with transformer-based PLMs since these tools do not select verbatim matches within the corpus relying on if-rules. Instead, they perform corpus resorting depending on the (cosine) similarity of the sentences to the query. The corpus is thus reordered in a list of sentences based on similarity decreasing value compared to the sentence in query. Since the user does not select statements across the corpus according to specific rules the number of results will always remain equal to the number of statements in the corpus.
34Secondly, non-restrictive search with neglection of sentence length and word order. Since sentence retrieval performed through transformer-based PLMs allows for a less restrictive and non-verbatim display of results, the length of the query sentence and the order of the words do not really influence the retrieval process, making this kind of query much more flexible and opening up the possibility to surf the corpus in a semantically related way. This is usually achieved through specifically tailored tokens ([CLS] tokens) that represent the entire sentence, regardless of its length.
35Thirdly, representational and semantic retrieval. If n-grams impose an exact word or lemma match, retrieving words and sentences with PLMs offers a much greater range of possibilities, since these tools operate at the distributional representation level of such statements. With the method we outline here, the comparison between statements is performed at the level of the numerical representation, rather than the lexical level. This means that semantic closeness of terms in the vectorial space will significantly influence the retrieval process, resorting the corpus according to similarity and, therefore, considering shared meaning elements.
36However, some downsides must be taken into account.
37Firstly, there is an inherent uncertainty regarding the completeness of the retrieved texts. The scholar will inevitably face the possibility that the retrieval process is not exhaustive. Even if the corpus is fully reordered according to its similarity with the query, it cannot be guaranteed that all the meaningful sentences related to the query will appear among the top-ranked results. While rule-based search returns all the explicit occurrences of the query in the corpus according to the retrieval parameters, semantic retrieval performed through transformer-based PLMs and cosine similarity produces a continuum of decreasing similarity scores across the entire corpus. Consequently, the user must personally decide when to stop examining the ranked list of statements, being aware that relevant items may remain hidden among the lower-ranked results.
38Secondly, unrelated results. The retrieval approach we detail here involves the possibility of having unexpected and unrelated sentences in the resorted results. In fact, verbatim retrieval exhausts all the possibilities offered by the corpus, while sentence retrieval performed confronting numerical representations might suggest unrelated elements. Therefore, the scholar will have to carefully evaluate the resorted sentences to make sure they are really related to his query.
39Thirdly, black box. This likely remains the core issue, that of the "black box": even if, once the training process is complete, the model is crystallized and operates in a (deterministic) fixed way always returning the same representation per word or sentence, we still cannot fully explain the rationale behind the distribution of the meaning of the model. The researcher does not have control over the elaboration performed by the distributional model, this happens entirely at a sub-word level and cannot be oriented through rules since it is completely data-driven. The developer’s space of action remains, in any case, absolutely important and details can radically change the results provided by the model: the workflow that surrounds the embedding process, especially the training phase, must be carefully designed, as minimal nuances, scaled to tens of millions of words, deeply impact the results. The pipeline includes core steps like composition and pre-processing of the training and validation dataset, training or distillation of the model, fine-tuning for specific tasks, exploitation and evaluation. This is the reason why researchers, in the training phase are expected to carefully establish hyperparameters such as the loss function (difference between predicted and actual outputs), the batch size (number of samples processed per iteration), the max sequence length of tokens in a sentence, learning rate (how quickly the optimizer updates the model’s weights), number of warm-up steps (in which the learning rate is gradually increased), mask ratio (for MLM training) and training epochs. This parameters are to be carefully designed and significantly impact the performances, but, in the end, scholars will have to face a margin they cannot deal with, the black box.
40In sum, statement (cosine) similarity retrieval pursued confronting numerical representations, the method we outline here, is a technique that may offer advancements in the scholarship with the caveat of considering the tool as a co-pilot instead of an oracle. Therefore, a good training should be offered with the distribution of this kind of tools.
41We have shown how the application of distributional semantics and transformer-based models has significantly advanced the study of Ancient Greek, transforming both semantic analysis and historical linguistic research. The development of PLMs highlights the growing potential of these methods to uncover semantic changes of words, semantic relationships and intertextual connections within ancient texts. Despite these advances, challenges remain, including the need for comprehensive evaluation benchmarks and the difficulty of training models for an ancient language. About statement retrieval, although PLMs offer more flexible and semantically nuanced retrieval methods compared to traditional rule-based tools, issues such as uncertainty about retrieval completeness, irrelevant results, and the "black box" nature of these models remain significant obstacles. As digital humanists continue to refine these models and explore new applications, it is clear that only careful evaluation and a deep understanding of the limitations of the models can fully unlock their potential. In other words, some caution is necessary to temper some positivistic tendencies that have surrounded the use of AI in language studies over the past 70 years.
42In 1955, McCarthy et al. (1955) submitted a whitepaper containing their proposal for a two-months research program to be held at Dartmouth College the following summer. In this document, they introduced the expression "artificial intelligence" for the first time and suggested laying the groundwork to replicate human intelligence and language, based on the conjecture that
every aspect of learning or any other feature of intelligence can in principle be so precisely described that a machine can be made to simulate it. An attempt will be made to find how to make machines use language, form abstractions and concepts, solve kinds of problem now reserved for humans, and improve themselves.
43A lot has changed since this first step: instead of having human instructors expliciting all the rules that describe a certain reality to train a domain-expert system, nowadays, with language models, information is implicitly and automatically distributed among weights: no single weight corresponds to a specific piece of information. Instead, it is the collective activation of weights that represents complex features and relationships from the input data. This method is effective, but at the same time, it makes it difficult to put into words what kind of linguistic features the model has learned, entangling the challenge of explainability and tenability of scientific gains. Automatically learning from data with sub-symbolic representation of features means, as suggested by , exchanging interpretability for performance or, in other words, giving priority to the results promised by PLMs relieving control over the work process. We should also be cautious about anticipating results: more than half a century later the white paper of McCarthy et al., Turney and Pantel (2010) say that
it seems possible to us that all of the semantics of human language might one day be captured in some kind of VSM
44The two quotations share the hope of the authors for a drastic development of AI’s performance toward and beyond the human level, a hope that all the features of human-level subjects will one day be fully grasped by computers so as to obtain a complete representation of the data. As the authors of this contribution, we suspect that the sub-linguistic distribution of features without its logical understanding will not grant models to properly appreciate meaning. In any case, even if models will always have to asymptotically approximate concepts from the outside, the craft of improving and tailoring them for specific tasks may be a beneficial aid to the community of scholars who daily engage with the challenge of semantics.
45This review was performed in the framework of the Italian Strengthening of the ESFRI RI RESILIENCE (ITSERR) project (cod. Progetto IR0000014 - CUP B53C22001770006) – Piano Nazionale di Ripresa e Resilienza (PNRR) – Missione 4, “Istruzione Ricerca”, Componente 2, “Dalla ricerca all’impresa”, Investimento 3.1, “Fondo per la realizzazione di un sistema integrato di infrastrutture di ricerca e innovazione”, finanziato dall’Unione Europea – NextGenerationEU – rif. Avviso MUR 3264/2021. ITSERR is an interdisciplinary and distributed Research Infrastructure for Religious Studies that aims to strengthen the RESILIENCE RI project in Italy. Elia Scapini and Federico Iezzi are involved in ITSERR as memebers of the fourth Work Package (WP4) that devotes its efforts to study the semantics of the Nicene-Constantinopolitan Symbolo and to build informatic tools for scholars in Religious Studies. Federico Iezzi and Elia Scapini are also members of the Fondazione per le Scienze Religiose (FSCIRE) in Bologna.
RSS feed
