1. Epidemic Intelligence: Objectives and Challenges

3Within this global framework, Member States have developed ways to implement this concept to support situation awareness and evidence-based decision-making in public health.

4Italy started to develop its own national approach to epidemic intelligence in 2007 as part of a project funded by the Italian Ministry of Health coordinated by the Istituto Superiore di Sanità (ISS) (Del Manso et al. 2022). At this time a situation and need assessment was performed in order to assess existing capacities and areas where additional implementation would be needed.

6Following several pilots to design and test this national event-based surveillance component of epidemic intelligence, Italy chose to follow the implementation model contextually developed and sustainably implemented by the Global Health Security Action Group Early Alerting and Reporting project (EAR) (Riccardo et al. 2014). This consisted of a decentralized approach in which participating countries contributed analysts that were operational on a rotation basis.

7In order to apply this to the Italian regionalized health care system, since 2017, Italy has adopted a decentralized method of setting up a network of analysts (Network Italiano di Epidemic Intelligence - Italian Network of Epidemic Intelligence) nominated by regional authorities among subject-matter experts employed within the national health system at the national, regional and local level. Each nominated analyst, before being included in the network, is required to accomplish theoretical and practical training coordinated by the ISS and the Italian Ministry of Health. Since 2023, the Italian Network of Epidemic Intelligence has been part of the community of practice of Epidemic Intelligence from Open Source - EIOS - an initiative of the WHO.

10Analysts of the Italian Network of Epidemic Intelligence work in rotating teams. Each day they screen news items, identifying those that are relevant to the surveillance focus (e.g., cases or clusters of infectious diseases in Italy or/and signs and symptoms in unexpected frequency) that are called signals. Signals are then individually risk assessed by the analysts using a common methodology (Intelligence and Miglietta 2022) to identify those of public health relevance that are called events and that are then reported.

11The event-based surveillance system operates on three activation levels:

• Level 0: situation awareness - signals captured by international and possible national monitoring are shared in a restricted group in real-time through instant messaging. The focus is national (on cases or clusters of any infectious disease in Italy) and on any international signals of interest for Italy. This activity is conducted in collaboration with other epidemic intelligence networks, such as ECDC and GHSAG-EAR.

• Level 1: activated once an event occurring internationally has been detected. The analyst performs the activities related to level 0 and in addition follows the identified event in time. Signals related to this event are shared in a restricted team in real-time through instant messaging.

• Level 2: activated once an event occurring or with the potential to occur in Italy is identified. In addition to activities performed under level 0, and if active level 1, analysts conduct an in depth screening and assessment of terms related to the identified level 2 event and publish an ad hoc thematic bulleting that, depending on the situation, can be daily or weekly. Instant messaging can be also applied if relevant.

12While Level 0 is by default always active, Levels 1 and 2 are subject to activation and subsequent de-activation based on an assessment made by the ISS and the Italian Ministry of Health. Based on the workflow described, at any given time, analysis is required to manually screen thousands of news items, reject irrelevant ones categorize signals, and assess them as events. Especially the screening phase of this work is extremely time consuming and resource intensive and this undermines the long-term sustainability of this surveillance system.

13The application of Natural Language Processing techniques is beneficial in this scenario to optimize information analysis and monitoring. The main goal is to enhance effectiveness in the following ways:

• Increased research capabilities: Thanks to its ability to process large amounts of data, the system can identify hidden infectious diseases and potential biological threats. This helps analysts discover relevant information that might otherwise go unnoticed.

• Reduced monitoring time: Artificial intelligence can automate some monitoring tasks, freeing analysts from repetitive duties or saving time. This allows them to focus on more complex and strategic tasks as the assessment.

• Improved analysis: The system can help analysts make more informed decisions in the evaluation process and quickly spot any anomalies or threats.

14The use of NLP and text mining techniques in order to extract relevant information from vast amounts of text data available on the internet, thus allowing the identification of relevant epidemiological events, has been extensively studied in the previous years (see (O’Shea 2017) for a systematic review of proposals dating a few years ago).

15Text classification is a fundamental approach to the identification of relevant events. After early works applying classical machine learning approaches (Kowsari et al. 2019) (Khan et al. 2010), deep learning architectures introduced a new set of general methods (Minaee et al. 2021), (Luan and Lin 2019) for text and news classification.

16Interest on the topic received a boost with the outbreak of the COVID-19 pandemic (Al-Garadi, Yang, and Sarker 2022), (Raza, Schwartz, and Rosella 2022), (Raza and Schwartz 2023). Moreover, the advent of the attention mechanism in neural networks (Vaswani et al. 2017) and the adoption of transformer-based encoders (Devlin et al. 2019), (Gillioz et al. 2020) made it possible effective information extraction from texts (Gupta et al. 2021), (Choudhary, Alugubelly, and Bhargava 2023) as well as document classification (Li et al. 2022), (D. Lin et al. 2021), (Kaliyar, Goswami, and Narang 2021).

17The use of transformer architectures as a basis for Large Language Models, to news classification is an active research area (see for example (Khosa, Mehmood, and Rizwan 2023), (D. Lin et al. 2021), (Santana, Oliveira, and Nascimento 2022), (Gunes and Florczak 2023)). However, the evaluation of the use of such approaches to the medical, and in particular in the epidemiological, field has been performed only quite recently (Wang et al. 2023), (Adaszewski, Kuner, and Jaeger 2021) and it is still in its infancy.

18This study investigates the impact of automatic event classification in Epidemic Intelligence by leveraging the advanced capabilities of Transformer-based models, specifically exploring the application of these models for automatic news classification relevant to epidemic monitoring. The research is motivated by the need for efficient and accurate tools to navigate the vast amount of information available, aiming to improve the identification of pertinent epidemic signals. This is accomplished by employing robust Transformer models, such as BERT-based models (Devlin et al. 2019), to classify articles from the ISS surveillance of infectious disease events in Italy.

19The methodology employed in this study is grounded in the practical application of the UmBERTo model (Parisi, Francia, and Magnani 2020), a Transformer-based classifier tailored to the Italian language, to analyze and classify articles. The experiment is designed to provide a detailed examination of the model’s performance in classifying the content according to predetermined themes related to epidemic intelligence. The results of this classification process are meticulously documented, providing insights into the effectiveness of Transformer-based models in a real-world application scenario where comprehensive coverage and accuracy are paramount.

20In addition to the classification task, this study also investigates the integration of the classified data into a sophisticated semantic search system. This system is designed to go beyond simple information retrieval, offering dynamic search capabilities that consider various metadata attributes, including date and region. The introduction of such a system represents a significant enhancement in the field of epidemic intelligence, offering a more efficient and focused mechanism for accessing and analyzing news related to epidemics.

21Overall, this research highlights the potential of Transformer-based models to support the field of epidemic intelligence by providing tools that significantly improve the speed and accuracy of event classification. This advancement is crucial for public health officials and analysts who rely on timely and accurate data to make informed decisions in response to epidemic threats. The study’s findings underscore the applicability of machine learning algorithms in enhancing the capabilities of epidemic intelligence systems, thereby contributing to the broader effort to safeguard public health.

22In the rest of the paper, we explore neural AI in language processing (2), demonstrate automating event classification for Epidemic Intelligence (3), and detail integrating this into a semantic search system for epidemiological analysis (4).

2. Language Processing with Neural AI models

23Machine Learning approaches to Language Understanding and Text Processing tasks date back to the 80’s (Lebowitz 1988), and have been revived during the 90’s by the research activities on Statistical Natural Language Processing (Jelinek 1998; Manning and Schütze 2001). The rise of neural approaches to NLP, although very old in its inspiration (Ide and Véronis 1990; Yuan et al. 2016), has inspired important contributions to distributional models of the lexicon (e.g., (Bengio, Ducharme, and Vincent 2000), (Mikolov et al. 2013)) lately applied to sentence (Le and Mikolov 2014) and text encoding tasks (Devlin et al. 2019; Reimers and Gurevych 2019). In this work, we emphasize the adoption of advanced Text Encoding techniques as a basic mechanism to capture event information which is relevant for Epidemic Intelligence.

24Transfer learning, i.e. pre-training a neural network on data available for T1, and then using the model \(\mathit{M}_1\) induced for T1 as the basis for a fine-tuning stage on the data available for the task T2, has been widely applied in the recent years, and has been shown beneficial in many contexts, among others computer vision. For example, the pre-training of convolutional neural networks on the ImageNet dataset is commonly used to support the later fine-tuning stage of the resulting pre-trained network, in order to obtain a new optimized task-specific model, e.g., (Girshick et al. 2013).

25The approach proposed in (Devlin et al. 2019), namely Bidirectional Encoder Representations from Transformers (BERT) embodies exactly this approach, applied in NLP. It provides a very effective model to pre-train a deep and complex neural network over very large corpora of unannotated text. This makes the resulting network suitable for application to a large variety of NLP tasks: it can be simply extended to new tasks by fine-tuning the entire architecture with (possibly small) problem-specific datasets.

26BERT exploits the Transformer architecture, an attention-based mechanism able to learn contextual relations between words (or sub-words, i.e. word pieces, (Schuster and Nakajima 2012)) in a text. In its original form, proposed in (Vaswani et al. 2017), a transformer includes two separate components, an encoder that reads the text input and a decoder that produces a prediction for a targeted rewriting (e.g., Machine Translation) tasks. Notice that the BERT input is handled by incorporating positional embeddings: individual symbols in the input sequence, such as words or wordpieces, are coupled with their specific positional information, through a dedicated embedding based on sinusoidal functionals, designed to represent the position of each word within a sentence. Notice that positional embeddings aim at inducing and expressing the contextual and syntactic information inherently carried out by the word order in a sentence. By encoding the position of each word, BERT can be made sensitive to the role and relationship of words within the entire sentence.

27As BERT may depend on more than 110 million parameters, the key to its success lies in the concept of pre-training. Network’s weights are initialized through several tasks, called pre-training tasks, that, while potentially unrelated to the target network’s primary task, are linguistic in nature and help the model generalize its understanding of language use. According to Wittgenstein (Wittgenstein 1953), language meaning arises as a side-effect of its use by native speakers. Language use is thus the crucial source of information about syntactic and lexical semantics phenomena in natural language. Pre-training in transformers aims at capturing exactly such universal properties of natural languages before attempting the training aimed at specific linguistic inferences (e.g., machine translation or question answering).

28These tasks are thus carried out by applying the network to extensive document collections, often consisting of billions of tokens, supporting a large-scale exposure to diverse linguistic phenomena. BERT is thus allowed to develop a nuanced understanding of natural language, akin to linguistic acquisition among humans.

29As reported in Figure 2 (on the left), during pre-training the Transformer encoder takes in input entire sequences of words at once, in parallel. It acquires a language model by learning to reconstruct an original sentence from its corrupted version: it randomly masks some of the tokens from the input, and the objective is to predict every original masked word based only on its context. The model derived through such pre-training objective is called MLM (Masked Language Model). In addition to the masked language modeling task, BERT also uses a second task, i.e. next sentence prediction, that jointly pre-trains over sentence-pair representations. This last objective is crucial to improve the network capability of modeling relational information between text pairs, which is particularly important in tasks such as QA (Devlin et al. 2019) in order to relate an answer to a question.

30After the language model has been trained over a generic document collection, the BERT architecture allows encoding (i) specific words belonging to a sentence, (ii) the entire sentence, and (iii) sentence pairs with dedicated embeddings. These can be used in input to further deep architectures to solve sentence classification, sequence labeling, or relational learning tasks by simply adding simple layers and fine-tuning the entire architecture (Bouraoui, Camacho-Collados, and Schockaert 2020). On top of such embeddings, fine-tuning is applied by adding task-specific and simple layers on top of the architecture acquiring the language model. In a nutshell, this layer introduces a minimal number of task-specific parameters and is trained on the targeted tasks by simply fine-tuning all pre-trained parameters, optimizing the performance of the specific problem.

31The adoption of BERT encodings in Text Classification. BERT can be considered a sentence-enconding model that takes an input sequence \(s\) and generates a vector representation \(h = M_{BERT}(s)\) for \(s\). In the context of this work, our primary interest is in classification tasks. For a given input sequence \(s\), BERT can be viewed as generating a vector from the first symbol of the sequence. In BERT’s architecture, this first symbol is the artificial token [CLS], and \(h\) is a dense vector of \(d\) dimensions (e.g., \(d=\)768).

32BERT can be employed for classification tasks, as its output \(h\), a 768-dimensional vector, is further processed through a neural classification network, able to map this high-dimensional vector to a space of \(c\) dimensions, where \(c\) is the number of targeted classes. The output of this classifier, \(y\), is a one-hot vector representing the class probabilities. Mathematically: \(y = M_{Class} \circ M_{BERT}(s)\) where \(M_{BERT}(s)\) is the BERT model applied to the input sequence \(s\), and \(M_{Class}\) is a dedicated multilayer perceptron with a given number of hidden layers (usually one) and one output layer with \(c\) neurons. The domain of the last weight matrix is thus \(\mathbb{R}^{(d+1) \times c}\), with \(d=768\) being the dimension of the BERT output. The classifier model \(M_{Class}\) is trained through backpropagation, usually by referring to the cross-entropy loss function \(\text{Loss} = -\sum_{i=1}^{c} y_i \log(p_i)\)

33Here, \(y_i\) is the true class label in one-hot encoded form, and \(p_i\) is the predicted probability of each class. The minimization of this loss function on training data optimizes the model parameters, ensuring the accurate classification of input sequences. The straightforward application of BERT has shown better results than previous state-of-the-art models on a wide spectrum of natural language processing tasks (T. Lin et al. 2022).

3. Automating Event Classification in Epidemic Intelligence: from keywords based retrieval to Automatic Topic Recognition

34Epidemic Intelligence (EI) is a wide process that involves information spread across several sources ranging from governmental health information systems, Clinical Data at the National and International level as well as Web and Media sources such as newspaper articles, technical reports, and social media posts. The involved profiles are analysts who are focused on the collection of evidence about epidemic phenomena as documented by eventualities (i.e. facts, events) and the geographical locations where they occur. Notice that the quality and the timeliness of the collected information are very important for the analysts as they are faced with impressive data volumes to be analyzed, (mostly) filtered out, and then summarized in health-related reports and infographics. The ability to locate as timely as possible the information of interest across the different and heterogeneous sources is crucial for supporting and augmenting the capability of the EI analysts.

35Usually, Web search is central to the above process, as the gathering of the suitable but unknown sources should be able to “scan” at least any publically accessible distributed information source. In this scenario, the Information Retrieval techniques currently employed are based on the Web metasearch engines fed with traditional keyword-based searches. Keywords allow on the one-side the analysts to directly specify the target phenomena through lexical expressions (names or phrases recalling events, virus names, or biological expressions related to potential epidemic phenomena) and guarantee a certain level of precision in the resulting retrieved materials. On the other side, keywords are usually adopted with similar semantics across different engines and sources, in order to make documents derived from different sources and engines semantically coherent and comparable with respect to the target epidemic phenomena.

36It is clear in the above scenario that keyword-based searches cannot always be a guarantee for consistent and precise information gathering so they may increase the risk of gathering large volumes of irrelevant information. This is mainly due to ambiguity phenomena affecting individual keywords. For example, clinical studies may well refer to uses of the word flood that are health-related, as in the case of epidemics, e.g.,

... flood of novel clinical cases are reported across the entire area ...

37However, when other meanings are adopted as in

Forests provide protection against floods and drought.
After you recognize it, real remorse floods your soul..

38Notice that every time a wrong use of a keyword is captured by a search engine the corresponding documents have a very high probability of resulting irrelevant to the experts analysts activity. This burden becomes prohibitive if an improper usage of a keyword is made by analysts who are surely non-expert in Web indexing and searching. They are thus not keen to adopt complex languages to express keyword logical compositions. Moreover, leaving the burden of correct, consistent, and efficient usage of keywords to analysts is a strong limitation of the overall impact of Web search. It makes the result semantically limited and not scalable so that the ability to improve the coverage (recall) of the targeted epidemic phenomena is severely impacted. In fact, the variety of phenomena of interest for analysts is left to its choice of keywords, a step that limits the semantic scope of the entire Web search process: information obtained by the analysts through the retrieved texts is fully enclosed by the phenomena that he contributed to define. No generalization or semantic inference about the targeted topics is made possible and the resulting information novelty is poor. With this impoverished level of technology, any proactive support to the analyst is minimally realized. The consequence is an evident reduction in the potentially reachable recall, i.e. a limitation of the involved Web search technology, as a concrete support in augmenting the scope of the analyst search as well as the novelty of the gathered Web information.

39The above limitations strongly depend on the fact that keyword-based searches, usually close to the bag-of-word approaches, directly represent text semantics just through word sets and their descriptive statistics. No generalization nor linguistic interpretation of the retrieved texts is attempted either during indexing or during retrieval. The entire literature on lexical semantics and distributional methods of induction of word and text semantics from corpora is neglected by keyword-based approaches. Although different semantic approaches to Web search have been proposed and discussed since the 90’s (Guha, McCool, and Miller 2003; Madhu, Govardhan, and Rajinikanth 2011), the Epidemic Intelligence community is not yet able to benefit from any advantage from them. In this work, we will pursue the idea that text interpretation is useful in EI for several dimensions

• Allows the analysts to expand the keywords with semantically related terms in the health domain, useful to augment the overall IR recall and to better constraint the interpretation (and ranking) by the search engine

• Use semantic interpretation of keywords and texts to filter out the truly irrelevant retrieved documents and automatically classify the incoming interesting data

• Support the analysts within an overall knowledge framework able to understand document content and helpful in better formulating target topics, in composing queries about specific phenomena as well as in supporting conceptual rather than textual navigation across the emerging document networks where associations are justified on semantic grounds

40In the experimental section of our paper, we describe the methodology employed for classifying news articles using advanced natural language processing techniques. Data for this study was sourced from the ISS Epidemic Intelligence - event-based surveillance (EBS) activity. During the second phase of the pandemic, an ad hoc EBS was implemented by ISS to support the monitoring of the risk of spread during the COVID-19 pandemic, under the Ministerial Decree of 30 April 2020 (World Health Organization 2008) and (Riccardo et al. 2014). During the surveillance period from February 2020 to September 2022, analysts concentrated on monitoring COVID-19 outbreaks across various epidemiological settings. Within this timeframe, ISS experts manually categorized a total of 3,245 news articles. Of these, 25% were sourced from national newspapers, reflecting the broader public health discourse at a country level. The remaining 75% were derived from local news outlets, providing insights into the pandemic’s impact on specific communities and regions.

41The annotation process was carried out by a team of four analysts at the ISS, comprising two medical doctors and two prevention technicians, all domain experts with specific knowledge on the spread of viral diseases such as COVID-19. These individuals meticulously categorized each article, ensuring that the classification reflected the most significant outbreak settings observed during the pandemic. The focus was particularly on outbreaks involving vulnerable groups such as hospital patients and the elderly, as well as those highlighting high rates of social interactions, including events involving children who, during the pandemic, were identified as having a high number of contacts and, by extension, their families. This detailed categorization aimed to sharpen the analysis on specific outbreak scenarios critical for public health surveillance and response strategies.

42These categories thus reflect the primary theme of each news piece and include: “School Outbreaks”, Family/Friend Outbreaks”, “Nursing Homes/Long-Term Care Facilities”, “Hospital Outbreaks”, “Outbreaks in Other Settings” and “Other Variants”. Notably, the Outbreaks in Other Settings category corresponds to a miscellaneous class, specifically useful in an open-world and exploratory scenario. It corresponds to a less semantically characterized area, whereas different topics (e.g., regional issues, environmental health-related phenomena as well as politics) are mixed. The distribution of examples is reported in Figure 3.

44In the model fine-tuning, a dropout rate of 0.1 was applied to the last layer. We adopted a 5-fold classification schema for our experiments. In each fold, 8 were designated for training data, 1 for development of parameters, and 1 for testing. After repeating these experiments 5 times, results were evaluated in terms of average accuracy, which is the percentage of examples correctly reassigned to their respective categories. To gauge the model’s performance across different categories, metrics such as Precision, Recall, and F1 Score were also calculated for each class.

45Experimental Results. The results are presented in Table 1. The first column lists the different categories, for each of which Precision, Recall, and F1 scores are reported. The last row provides the arithmetic mean across all classes. Overall, the results are quite promising. A reference baseline model that assigns the most frequent class to any incoming document would just achieve an accuracy of 29.3%. The overall accuracy reached by our model is 86%, which confirms its robustness across all classes. The average F1 score obtained is 85%. Some classes are particularly well categorized (e.g., Nursing Homes/Long-Term Care Facilities or School Outbreaks) with F1-scores over 0.90. The category with the lowest results is “Family/Friend Outbreaks”, with an F1 score of 0.63, which is also the least represented, accounting for less than 8% of the data (278 out of 3245 documents).

Table 1: Performance evaluations of the automatic categorization task in terms of Precision, Recall, and F1 measures: the overall average Accuracy is 86%.

Category	Precision	Recall	F1-Score
Nursing Homes/Long-Term Care Facilities	0.88	0.93	0.91
Family/Friend Outbreaks	0.59	0.68	0.63
Outbreaks in Other Settings	0.81	0.84	0.83
Hospital Outbreaks	0.96	0.80	0.87
School Outbreaks	0.91	0.91	0.91
Other Variants	0.96	0.89	0.92
Global Micro Average	0.85	0.84	0.85

Table 2: Comparative Performance Evaluation of Different Models on the Automatic Categorization Task. Accuracy, Precision, Recall, and F1 Scores are reported for the baselines and the Transformer-based model, more comprehensively presented in Table 1.

Model	Accuracy	Precision	Recall	F1-Score
Logistic Regression	0.68	0.84	0.71	0.77
Gradient Boosting Classifier	0.63	0.86	0.68	0.76
Multinomial Naive Bayes	0.66	0.71	0.82	0.76
UmBERTo (this paper)	0.86	0.85	0.84	0.85

47Notice that the nature of the dataset is representative of different complex phenomena, such as news discussing multiple topics or ambiguous texts referring to some borderline events not related to just one class. As, in the target reports, the analysts are asked to decide a unique category for each news item, subjective choices representative of just partial views on some news are well possible. Unfortunately, as no inter-annotator agreement had been measured during the development of the corpus, it has not been possible to have a quantitative estimation of these phenomena.

Table 3: Confusion Matrix of the Classification outcomes of the UmBERTO model: rows represent the gold classes while columns report the system’s assignments.

Confusion Matrix	Nursing Homes	Family Friend	Other Sett.	Hosp.	School	Other Variants
Nursing Homes	183	0	11	0	2	0
Family/Friend	1	51	19	0	4	0
Other Settings	7	25	232	4	6	2
Hospital	16	3	7	111	0	2
School	0	5	9	1	160	0
Other Variants	0	3	7	0	3	100

49In this example, the system’s misclassification can be attributed to the various expressions in the news that typically refer to the school environments. The event is about a soccer school but it also makes a significant reference to a school outbreak. The reference to a “scuola calcio” (soccer school) and the mention of “bambini” (children) and “classi” (classes) in the context of the quarantine are likely misleading information for the model pushing for preferring a “School Outbreak” class. notice how the actual context of the news focuses on the sports activity (soccer school) and this is what matters to the expert as the main aspect of this news item. It is thus correctly annotated as “Outbreaks in Other Settings”.

52The above different classifications illustrate the challenges for the categorization task, especially the limitations of keyword-based approaches. Terms in fact may overlap across categories as well as exhibit senses that are related to the news context in complex manners. This experience highlights the importance of an accurate contextual understanding of incoming news, thus outlining the need for high-quality training with clear definitions for the target classes. Notice that in Epidemic Intelligence classes may arise in an unexpected fashion and the development of large and accurate training sets is often not viable. This emphasizes the role of a collaborative framework where the system’s proactive data gathering and filtering abilities can be made available to the analyst, in order to alleviate the uninteresting part of its effort. This is why an overall integrated Natural Language Processing and Information Retrieval approach, as described in the next section, can be considered a relevant support to trigger proper Machine Learning functionalities for the analyst, through a fully engineered collaborative environment.

4. Integrating News Classification into a Semantic Search System for Enhanced Epidemiological Analysis

53The direct application of the accurate news classification into Epidemic Intelligence (EI) relevant thematic classes concerning COVID, as described in the previous section, allows to devise of a fully integrated Semantic Search system. The system is designed to support EI analysts in thematic event analysis as they can be observed in online documents and news streams, that reflect real-world phenomena.

54The system under discussion aims to enable analysts to search for web news and then focus on themes related to specific epidemiological analyses. In essence, it acts as a filter for Web news, allowing analysts to concentrate on information pertinent to their research or monitoring needs.

55Once the news articles are located, uploaded, and indexed, the system provides search functionalities. Analysts can use keyword-based or natural language queries to retrieve news articles. The thematic classification previously described plays a critical role in this process. It allows for the filtering of results based on the themes of interest to the analyst. For example, a query like “Suspicious death” filtered using the class “Nursing Homes/Long-Term Care Facilities” can enable analysts to delve deeply into this topic in a focused manner.

56The thematic classification aims at refining the search results but also adds contextual information about news for interpreting its relevance to the query. When categorized news articles are retrieved by a query the system ensures that search results can be organized into classes, such as into classes like “School Outbreaks”, “Hospital Outbreaks”, or “Other Settings”. Classes can be used as an enrichment of news and can be used as filters for targeting the specific information needs of epidemiological research.

57News articles are also enriched with other editorial metadata, such as the publication date or the geographical region of origin. In this way, standard Business Intelligence (BI) filters and aggregations can be directly employed to cluster conceptually retrieved data, compute their statistics, and analyze trends. The thematic classification into EI classes of interest can be combined with these metadata elements. This enables dynamic insights and allows to examination of emerging patterns within a broad spectrum of web-based news. For example, a simple query towards a specific class (e.g., “Hospital Outbreaks”) and filtered according to metadata matching a specific region can be used to focus on specific news articles whose statistics are direct evidence of potential warnings, e.g., regional increase in hospital-related COVID cases. This can directly prompt more detailed investigations, or immediate actions, based on other region-specific data. Additionally, tracking changes over time, made possible by temporal metadata, allows for trend analysis across time, and offers vital insights into the progression, or decline of specific events or alerts across different geographical areas.

58The fully automatic data-driven approach suggested here, leveraging natural language understanding and thematic classification provides enriched metadata and transforms the way news is analyzed. It is a concrete way to straightforwardly promote informed decision-making and strategic planning, in public health and epidemiological research.

59In Figure 4, the landing page of the Semantic Search system is showcased. This interface is designed to facilitate advanced search capabilities for users, particularly in the context of epidemiological analysis. Beyond the standard text field for natural language query input, the system provides additional options that allow users to refine their search based on various criteria related to publication time, geographical information, or EI topics related to news articles. The key features include:

• Document Type: This option enables users to select news based on information about their source and technology: here articles (such as news from Google) or tweets are distinguished. It allows filtering the information by its medium, by offering thus diverse perspectives or source perimeters.

• Team: This feature allows users to apply filters corresponding to specific team-defined criteria. For instance, users can filter data based on different analyst teams, as these are focused on independent aspects of epidemiological research or data curation.

• Region: With this feature, users can select a subset of news articles assigned during the download phase to one or more specific geographical districts or regions. This is particularly useful for localizing EI events, e.g., hospital outbreaks, to specific areas like Piedmont or Lombardy.

• Main Topic: Each analysis session addresses a specific set of EI events called “main topic”, such as COVID, MonkeyPox, or Dengue. Articles downloaded during an event campaign are labeled with a specific “main topic” property: articles downloaded within different campaigns may carry multiple labels. When active, this feature allows for support requests related to specific epidemiological analyses (e.g., COVID vs. MonkeyPox) by also enabling a cross-thematic analysis on all downloaded news.

• Category: This pertains to the categories assigned to each main topic, as discussed in the previous section. This functionality helps recognize events and separate them by categories. While COVID related categories are defined on historical data, the collaborative work of analyst teams is currently underway to define more granular categories also for other topics, such as MonkeyPox.

• Time Period: This functionality allows users to specify a time frame for the returned news articles. Users can define this period using a calendar interface, thus enabling the analysis to focus on news from a particular period.

60In Figure 5, we display the response to the query for “contagi” (infections) that also filters specifically news articles about the year 2020 and from the Lombardy region, under the main topic “Coronavirus”. Unlike a traditional Web retrieval engine that might have returned tens of thousands of news items, the current system retrieves only 220 news articles, that precisely meet the analyst’s needs.

62At the bottom of Figure 5, the top two relevant news articles are displayed, with their main metadata (such as the Lombardy region and year of publication), along with a news snippet preview.

1. Query Expansion: Every entered query term triggers the system selection of \(k\) words in the domain lexicon whose embeddings are most similar (i.e. those maximizing the cosine similarity with the vector obtained through the linear combination of already entered query terms).

2. Re-ranking: Once the news articles are retrieved using a SOLR-based search engine, which adopts a BM-25 model for ranking results, the system offers the possibility to reorder the news semantically more connected to the query, i.e., those whose representation in the word space maximizes the cosine similarity with the query representation.

3. Similar News Search: This function can be activated using the magnifying glass icon present for each news article. The vector representation of a news item can be used, similar to modern RAG systems, to retrieve news semantically most related to a query news item.

64Currently, the system is operational under validation by the analysts: as shown in Figure 5, users can judge the quality of the news retrieved by the system (ThumbUp/ThumbDown symbols) to measure the search engine’s quality and provide examples for the application of re-ranking methods (Liu 2009).

5. Conclusions

65This study has explored the application of Natural Language Processing (NLP) and Machine Learning (ML) in the context of Epidemic Intelligence (EI), with a focus on developing a Semantic Search system that integrates advanced NLP techniques. The findings and developments presented aims at demonstrating the utility of these technologies in enhancing epidemiological analysis. The major outcomes of this work include the use of Transformer-based models for classifying web news into thematically relevant categories for EI, showing promising results. This approach suggests the effectiveness of language-specific models in handling complex linguistic data for health-related information processing. Moreover, integrating classified information into a Semantic Search system seems beneficial in streamlining the information retrieval process in EI. This system enables efficient thematic searches, focusing on epidemiological relevance and offering a more targeted approach to data analysis. Finally, the ability of the system to utilize thematic classification and metadata enriches the analysis process. By applying Business Intelligence techniques for data aggregation and trend analysis, the system aims at offering a nuanced understanding of epidemiological patterns and trends. In summary, this research contributes to the field of EI by showcasing the potential of intelligent NLP and ML applications. The development of the Semantic Search system highlights the significance of technology-driven approaches in public health and epidemiological research, particularly in terms of processing and analyzing large-scale, linguistically complex datasets.

66Our ongoing and future work involves a comprehensive approach to enhancing and expanding the Semantic Search system developed in this study, with a focus on its application in Epidemic Intelligence (EI). The immediate ongoing task is the qualitative and quantitative evaluation of the system. This involves in-depth user feedback analysis and usability studies, combined with a quantitative assessment of the system’s accuracy and efficiency in information retrieval and classification. These evaluations are critical for identifying areas that require improvement and ensuring the system’s efficacy and reliability for EI applications.

67Looking toward future developments, a significant area of expansion is the adaptation of the system to cover a broader range of diseases beyond COVID-19. Other types of diseases, such as the dengue that is transmitted by mosquitoes, present specific and new challenges in terms of information needs and contexts. Adapting the system to these challenges will involve refining classification schemas and renewing training and search algorithms to make them seemingly effective to the unique aspects of further diseases. This is why, the exploration the application of unsupervised learning methods, particularly leveraging Large Language Models, for the classification of news. A promising direction in this regard is the implementation of zero-shot learning techniques, where the model is capable of categorizing news into relevant themes without being explicitly trained on those categories. This approach could significantly improve the adaptability and flexibility of the system, allowing it to respond effectively to the detection of threats and risks to public health detect threats and risks to public health. Another innovative direction for future research includes the unsupervised extraction of classes from news data. This method aims at automatically identifying new and emerging themes and categories in epidemiological news, thus enabling the system to stay updated with evolving health threats and trends.

68The project realized with the technical and financial support of the Ministry of Health - CCM

69Special thanks to: Stefania Giannitelli, Angela Ancona, Xanthi Andrianou, Federica Attanasi, Debora Ballarin, Elisa Bernini, Sandro Bonfigli, Bianca Borrini, Marco Cristofori, Silvia Dari, Elisa Di Maggio, Maria Paola Di Sebastiano, Mariapaola Farinelli , Pina Iannelli, Michele Labianca, Aurora Luciani, Dario Macchioni, Valentina Marras, Stefano Marro, Elena Mascia, Noemi Maria Mereu, Alessandro Miglietta, Nadia Olimpi, Daniele Paramatti, Ilaria Pati, Vincenzo Restivo, Alessandra Rossodivita, Michela Sabbatucci, Angelo Salzo, Sarah Samez, Monica Sane Schepisi, Francesca Sanità, Roberto Santoru, Simonetta Santus, Stefania Scaltriti, Irene Schenone, Marco Serale, Matteo Sponga, Francesco Vairo, Donatella Visentin, Francesca Zanella.