1Each author has a discursive identity made up of identifiable lexical and grammatical choices. Therefore, one of the challenges of deep learning on text is to describe these identities.
2Although it was shown in the literature that, in terms of accuracy, CNN based approaches outperform existing classifiers based on statistical key-indicators (e.g. the relative words frequency) or other machine learning techniques, it is still not clear if and how CNNs make use of standard features used in text mining (for instance word co-occurrences). We might also go further and assume that, for text classification, CNNs can rely on other complex linguistic structures that might be of interest for linguists. In the attempt to shed some light on this topic, our approach mainly relies on deconvolution process (i.e. transpose process), allowing us to interpret the CNN features in the input space.
3This paper focuses on linguistic object analysis via a multichannel convolutional architecture. That is, a CNN is trained to associate several parts of transcribed political speeches to their speaker (e.g. E. Macron and D. Trump). Our main contribution is an improvement of an existing measure, the Text Deconvolution Saliency (TDS) (TDS, Vanni et al. 2018), called weighted Text Deconvolution Saliency (wTDS), allowing us to visualize the linguistic markers used by the CNN to perform the classification of a text, but also to make them fully interpretable for the linguists. In order to have a relevant description of a dataset, the wTDS is included in a model that introduce two further contributions i) processing the CNN parameters in order to “rank” text segments assigned to an author from the more to the less representative of that author and ii) introducing a multi-channel CNN architecture in order to exploit additional linguistic information (e.g. lemma or part-of-speech) for each token.
4The next section describes some of the most representative related works. Two of them are discussed in more details in order to motivate and better describe our own main contribution.
5Since the seminal work of Collobert and Weston (2008), adopting CNNs for several NLP tasks (part-of-speech tagging, chunking, named entity recognition and semantic labeling), many researchers have widely used CNNs for similar and other purposes, such as text modeling (e.g. Kalchbrenner et al. 2014) or sentence classification (e.g. Kim 2014). While CNNs are not the only available deep architecture in Text Mining, it has been noticed that they have several advantages with respect to recurrent architectures (RNNs, in particular LSTM and GRU) when performing key-phrase recognition (Yin et al. 2017). This supervised classification task is the one we are interested in this work. In particular, we aim at uncovering linguistic patterns used to highlight similarities and specificities (Feldman & Sanger 2007, Lebart, Salem & Berry 1998) in a corpus. Standard text analysis techniques originally relied on statistical scores, for instance on the relative frequency of words (a.k.a. z-scores, see Lafon 1980). However, these techniques could not exploit more challenging linguistic features, such as syntactical motifs Mellet and Longrée (2009). In order to overcome these limitations and to account for long term dependencies in sentences, CNNs have been recently used. Indeed, being CNNs more robust than RNNs to the vanishing gradient problem, they might be able to detect links between different parts of a sentence (Dauphin et al. 2017, Wen et al. 2017, Adel & Schutze 2017). This property is crucial, since it was shown that long range dependencies emerge in real data (Li et al. 2015). Aiming at inspecting these dependencies as long as other complex linguistic patterns, some tools explaining how CNNs perform the classification task are required. In this regard, a recent crucial contribution is represented by the Local Interpretable Model-agnostic Explanations (LIME Ribeiro et al. 2016) framework. The basic idea of LIME is to approximate any complex classifier (e.g. a CNN) by a simpler one (e.g. sparse linear) in a neighborhood of a training point xi. A simplified representation x̅i of xi is adopted, and N points in a neighborhood of x̅i are sampled uniformly and used to minimize a distance between the original classifier and the simpler one. Once the simpler classifier is trained, it can be used to assess the (positive or negative) contribution of each feature to the classification task as easily as in linear models. This approach provides very interesting results and is generic, since it can provide explanations for any kind classifier. However, for every training point it involves sampling N neighbors and evaluating the classifier for each one of them. This might be computationally prohibitive, especially for high dimension data. In the context of key-phrase recognition, an alternative approach was proposed by Vanni et al. (2018). They considered as input data text segments of fixed size (M tokens). Each data point was represented as an M × D matrix, where D is the word embedding size. After training a CNN for an author recognition task, they used a Deconvolution Network (Zeiler & Fergus 2014) to project the feature map back into the input data space. Thus, the “deconvolution” assigns to the m-th token in the i-th text segment (say dim) a vector xim ∈ ℝD. The sum of its entries defines the Text Deconvolution Saliency (TDS) of dim. Intuitively, the higher (respectively lower) the TDS of dim, the more (less) dim contributed to assign the text segment to its class (i.e. its author). Although this approach returns meaningful results it may suffer from some inconsistencies in the explanation, as it will be shown in Section 2. In order to preserve the computational efficiency of TDS (once the CNN is trained it can be computed at a cost of one model evaluation per data point) we propose an improved version of the TDS (Section 2.2) overcoming the explanation drawbacks.
6This paper is organized as follows: Section 2 describes our CNN architecture as well as our contributions. Section 3 illustrates the framework described in Section 2 on two datasets: a English corpus and a French corpus. Section 4 concludes the paper and outlines some perspectives for future research.
7The first part of this section details our model, a convolutional neural network, trained for author classification tasks. In this work, this task corresponds to an intermediate step but does not represent our final goal. Indeed, the scope is to learn how to exploit a trained CNN to recover linguistic markers, specific to the different authors. Thus, after detailing the architecture, we focus on some original contributions to the linguistic features extraction. Our main contribution, the weighted Text Deconvolution Saliency (wTDS) is described in Section 2.2. Two other contributions, the softmax breakdown ranking and the multi-channel convolutional lemmatization are discussed in Section 2.3.
8Notation. In the following, v ∈ ℝN will denote a real vector v with N entries. If not differently stated, it is intended to be a column vector. The notation A ∈ ℝM×N will be used to define a real matrix with M rows and N columns and the function relu(·) is defined as
relu(x) = max{0, x}.
9The CNN considered takes as input d1,…,dN text segments, each containing a fixed number of tokens M. In the examples that we consider in Section 3 each segment is part of a presidential speech, so that the number of classes K is the number of considered presidents. An embedding layer is used for word representation. Although this layer might rely on different well known models such as fastText (Bojanowski et al. 2017, Joulin et al. 2017), Word2Vec (Mikolov et al. 2013) or Glove (Pennington et al. 2014) as long as a fine tuning of the embedding vectors is allowed during optimization, the choice of the embedding model is not crucial. Once the word feature vectors are obtained, they are concatenated (by row) in such a way to form a matrix with M rows. This resulting matrix can then be input into a convolutional layer applying several filters all having the same width as the dimension of the embedding matrix. One max-pooling layer follows, equipped with a nonlinear activation function. A deconvolutional layer (up-sampling plus convolution with transpose filters) is then introduced to bring the convolutional features back into the word embedding space. Finally, two fully-connected layers and a softmax function output for each segment di a vector ẑ ∈ {0, 1}K, where K is the number of classes/authors. The following multinomial cross-entropy loss function is considered:
10where θ denotes the set of all the network trainable parameters and z ∈ ℝN×K is an observed binary matrix, whose k-th row encodes the class/author of the i-th text segment (thus zik = 1 iff di is affected to the k-th class/author). The above loss function is minimized with respect to θ via an Adam optimizer. In order to avoid overfitting the whole dataset is split into train (80%) and validation (20%) sets and the loss function in Eq. (1) is monitored on the validation set during optimization, allowing us to apply early stopping (Prechelt 1998) (Figure 1). A graphical representation of the model described so far can be seen in Figure 2.
Figure 1. Model loss and accuracy
Figure 2. Three channels convolution/deconvolution for three representation of the input 1) full-forms (words), 2) part-of-speech (POS), 3) lemma
11After the CNN has been trained on the train dataset, it can assign a text segment di (either in the train or in the validation set) to its class/author. We recall that di can be viewed as a real matrix with M rows, where M is the number of tokens of di and D columns, where D is the embedding size. The m-th token of di, corresponding to the m-th row of the matrix, is denoted by dim and it is a vector in ℝD. The deconvolutional layer (see Figure 2) assigns to every dim another vector of the same size denoted by xim ∈ ℝD. Note that, since this representation is the output of two convolutional layers, it is sensitive to the context of dim (neighbor tokens). The Text Deconvolution Saliency (TDS, Vanni et al. 2018) of the token dim is defined as
12where the real number ximd the d-th entry of xim We stress that, although this measure is defined for each token of di it also accounts for the context of di (see also the experiments in Section 3). The authors in Vanni et al. (2018) argue that, the higher the TDS of a token, the more the token (conditionally to its context) plays a crucial role in the classification task, according to the CNN. As a matter of fact, even though TDS can correctly highlight the relevant words/contexts in di being used by the CNN to classify di, it cannot tell us how the network uses them. To illustrate this point in more detail, consider the following extract from a speech by Donald Trump:
13This text is part of a corpus described in Section 3 and collects several part of speeches from the US presidents. Once properly trained for an author recognition task, the CNN detailed in the previous section can correctly recognize this speech as being pronounced by the president Trump.
Figure 3. Comparing the activation boost of the tokens toward the class “Trump” according to TDS and LIME
14In Figure 3a a histogram reports the TDS scores for the tokens of the extract. The higher the bars, the more the corresponding tokens had a key role in the classification task. Now, when comparing these TDSs with the word contributions detected by LIME (Figure 3b) we see that most of the tokens having a high TDS correspond to brown right bars having a positive impact in classifying the speech as “Trump” (e.g. righteous, people). Conversely, according to LIME, the noun “poverty” seems to have a negative boost when performing a binary classification “Trump” or “No Trump”. Indeed, if we additionally compute the z-scores of the tokens of di (Figure 4), with respect to the whole corpus, we see that the noun “poverty” is underused by D. Trump and this is in line with the explanation provided by LIME. However, this noun is very specific to another president in the corpus: L.B. Johnson. Thus, the importance of the word “poverty” was correctly captured by TDS, but we cannot say if that word contributed for “Trump” or against “Trump”.
Figure 4. Z-scores for the noun “poverty” for the US presidents in the analyzed corpus
15This motivated us to improve the TDS score initially proposed by Vanni et al. (2018), with two additional features: i) it should be able to go negative to indicate negative contributions of words to some classes and ii) in case of multi-class classification, for a word dim it should be able to quantify its contribution to each class. In order to build such a measure, note that the last two fully connected layers of the CNN basically map the de-convolved features xi1,…,xiM into a single vector in ℝK, denoted yi (see Figure 2), where K is the number of classes. If we concatenate xi1,…,xiM into a column vector Xi, of size D×M, the map can be specified as
16where A ∈ ℝE×DM, b ∈ ℝE, C ∈ ℝE×K and d ∈ ℝK and E is the size of the penultimate layer. In order to obtain a score that is specific to the token dim we observe that
17where Am ∈ ℝE×D is the sub-matrix of A obtained by selecting all the rows and the D columns form the (D(m − 1) + 1)-th to the (D(m − 1) + D)-th. Thus we define
18Note that, instead of TDS(dim), wTDS(dim) is a vector with K entries. Each entry quantifies the activation boost of word dim (conditionally to its context) for the class K. Moreover, the matrix multiplication Amxim induced K weighted sums of the entries of xim, in contrast with the simple sum defined in Eq. (2). For this reason we call the measure in Eq. (5) weighted Text Deconvolution Saliency (wTDS). Figure 5 shows the wTDSs for the class “Trump” of the tokens in the Trump’s speech reported above. As it can be seen, the word “poverty” now has a small negative contribution when classifying the speech as “Trump”. We notice that, once the CNN is trained, the computation of the wTDS for one token (for all the classes) has the cost of the matrix multiplications in Eq. (5). This is a huge advantage compared to LIME for two reasons: first, no sampling is required.
Figure 5. wTDS for classes “Trump” and “Johnson” for the tokens in the sample speech of D. Trump
19Second, whereas LIME can only provide us with the tokens contribution in the binarized problem (e.g. “Trump” vs. “No Trump”), wTDS computes the tokens contribution to each class in one shot.
20In the previous section, we described how, given an input text segment di, wTDS can be used to assess the contribution of each token in di for the class assignment. Now, we zoom one step out and try to detect the key-segments in the data set, i.e. the segments being the more representative of each author according to the CNN. In particular, it might be of interest to be able to rank d1,…,dN from the most to the least representative for each author.
21A possible way to do that is described in the following. The number of neurons in the last layer of the deep CNN coincides with the number of classes, previously denoted by K. In the previous section yi ∈ ℝK denoted the value of that layer for the text segment di. Thus, yik is the value of the k-th neuron and it is a real number. As usually, a softmax activation function is applied to yi in such a way to obtain K probabilities ẑik (see Figure 2) lying in the K − 1 simplex
22Note that the above ẑik is the very same as in Eq. (1). The highest probability ẑik corresponds to the class assigned by the network to the observation di. However, if one entry of yi is significantly higher than the others, it is mapped to 1 by the softmax transformation and all the other entries are mapped to zero. For instance, consider two de-convolved features yi and yj corresponding to two different documents both assigned to class k. Assume also that yik > yjk, so that the document di is more representative of the class than dj. If yik and yjk are large enough, after applying the softmax function they both will be mapped to one and it will no longer be possible to assess whether di or dj is more representative of class k. Thus, we make unconventional use of the trained deep neural network and observe the activation rate of neurons before applying the softmax transformation. Doing that, allows us to sort the learning data (text segments) based on their activation strengths. This simple but efficient method provides us with the most relevant key-segment in the corpus for each class.
23Often, CNN for images have multiple channels. Indeed, the RGB colors encoding could be considered as three different representations of the input. Each representation corresponds to a data matrix and the convolutional layers apply different filters to each matrix and then later merge the results. Also with texts, it is possible to encode the data in multiple channels that might be used, for instance, to combine different word embedding solutions (skip-gram, cBow or Glove). Apart from word embedding, a pre-tagging process (Collobert & Weston 2008) allows data scientists and linguists to get supplementary material on each word, such as the part-of-speech (POS) and the lemma. Both of them are essential for a linguistic interpretation of the key-segments and to observe complex linguistic patterns (a.k.a syntactical motifs Mellet & Longrée 2009). It is those reasons motivated us to implement a multi-channel CNN to account for the POS and the lemma. However, using a single multi-channel convolutional layer to learn those patterns from each representation is not convenient for our purposes. Indeed, the max pooling operations merge all the information into one channel, thus making it impossible to retrieve which representation (word, POS or lemma) contributed to the classification. Since the aim of our contribution is to interpret the classifier, we split the convolution (and the max pooling) in three parts, one for each channel (see Figure 2). By doing that, the deconvolution mechanism can be applied to the three channels separately and all the linguistic features can be observed right after the deconvolutional layers. Finally, to combine this information, the features are merged into a global vector and the final dense layers use them to perform the class assignment. In more details, the m-th token of the segment di is now represented by three embedding vectors, say dim(w) for the full form, dim(pos) for the POS and dim(l) for the lemma (see Figure 2). After deconvolution, these embedding vectors are mapped to xim(w), xim(pos) and xim(l), respectively. Thus, whereas with a single channel, wTDS(dim) was a vector in ℝK, in a multichannel environment, we can define three wTDS vectors in ℝK for each token. For instance, wTDS(dim(l)) refers to the lemma component of the m-th token and it can be computed as
24where Aim(l) denotes a sub-matrix accounting for the lemma channel (the green one in Figure 2) and the m-th token xim(l).
25Political discourse analysis is one of the major challenges for linguistics in textual data analysis. For many years, statistics have provided tools and results that help linguists to interpret political speeches. We will now see how our deep architecture allows us to describe international political discourses. We propose to test our model by analysing two political discourse corpora in two different languages, English and French. For comparison reasons, these two corpora are made from presidential speeches and respect the same chronological span, from the 1960s to today.
26The first dataset targets American political discourse. It is a corpus of 1.8 millions of words of American presidents from J.F. Kennedy in 1961 to D. Trump in 2019. With 11 presidents, we focus on D. Trump to make a short but profound linguistic analysis of the discourse of the current US president. The second is symmetrical with the speeches of the French presidents under the 5th republic from 1958 to today. It is 8 French presidents from C. De Gaulle to E. Macron with 2.7 millions of words we focus also on current president, E. Macron.
27By default, the accuracy of each model (English and French) exceeds 90%, but the markers displayed by the wTDS seem to be too sensitive to low frequencies (very rare linguistic markers) or on the contrary very frequent but unique to a president (high z-score). The purpose of our architecture being to observe new linguistic markers different from those known by statistics, each corpus has been filtered with precise rules to reduce the weight of these markers. Some words have been replaced: i) proper names ii) dates iii) words only present in a president. These rules reduce model accuracy by about 10% but help to reduce overfitting and extract relevant key segments. The table 1 compare those models, unfiltered (English, French) and filtered (English*, French*).
Table 1. English and French datasets
28Section 2.2 introduce a key-segment of D. Trump detected with the softmax break-down ranking method with a simple model using only one channel for the full-form of words. With the multi-channel convolutional lemmatization (Section 2.3), we have now a wTDS score on each token for each channel and this selected segment become fully interpretable for the linguists due to exploitable features on full-form (blue words), part-of-speech (orange words) and lemma (green words):
29We highlight here the main activation zones having a wTDS higher than a fixed threshold. As it can be seen, there is a redundancy of “righteous people” and “righteous public”, being part of a simple and compassionate vocabulary (e.g. “families”, “mothers”, “children” or simply “good jobs”), which is typical of populist speeches.
30“But” appears as a characteristic of a polemical discourse that defines Trump’s rhetoric. The president rarely makes a consensual speech. Opposition marks, as “But”, allow him to build a speech setting him apart from the mainstream. Being “But” placed at the beginning of the sentence, its full-form wTDS highlights the role of conjunction of opposition rather than of conjunction of coordination.
31We also report that the full-form wTDS for the word “many” is negative (Figure 3a). Since “many” is one of the words more often employed by president Trump (high z-score), a negative wTDS might appear surprising. However in this context, “many” is preceded by “too” which is taken into account by the convolution layer. Thus we checked the z-score of the linguistic pattern “too many”, and we found out that it is higher for B. Obama than D. Trump. This is a very good example of the wTDS capability to capture the linguistic context.
32Finally, the wTDSs of part-of-speech focuses on a simple but essential marker, the dot (encoded as “SENT”). The over use of this marker refers to a fundamental rhetorical choice of D. Trump: short sentences. The reduction of the sentence length is a trend that can be observed in most democracies in Europe or in USA. In the attempt to be accessible to as many people as possible, D. Trump’s speech thus plays on syntactic simplification (Norris & Inglehart 2019). For a long time, political discourse has imitated literature with long sentences and relative or subordinate proposals, but nowadays, political discourse imitates popular language with short sentences that include only one subject, one verb and one complement. On average, in the corpus, Trump’s sentence counts 14.15 words when Obama’s sentence counts 21.51 words (Figure 6). In fact, the end of sentence markers characterize the current president. In 50 words here, Trump seems to take up the linguistic characteristics of populist discourse (Oliver & Rahn 2016) as it is expressed in the United States and Europe at the beginning of the 21st century.
Figure 6. Average sentence size
33This section aims at demonstrating that Deep learning can easily adapt to the subtleties of each language. A French presidential corpus is considered. In this dataset, the segment that the model identifies as being the most characteristic of E. Macron’s speech gathers remarkable features of the current French president language. The wTDSs highlight linguistic markers with multiple interpretations:
34Some main features of the E. Macron’s speech emerge. First, the French president tries to give a non-ideological and pragmatic talk oriented towards action, movement and efficiency (Colen 2019). Thus, the lemmas “construire” (to build) and “transformation” are very meaningful of such a discourse whose main scope is to be dynamic. The word “numérique” (digital) is often at the heart of the speech of a president who talks about changes and who wants to show his technical modernity. Then, from a grammatical and syntactic point of view, most of the time, the “PRP PRP:det” sequence (meaning preposition + contracted article, in French) introduces adverbial phrases. Thus, E. Macron avoids the main topics but he is precise with the modalities of the action. In E. Macron’s speech, both the subject and the object are less important than the way of the proposed reforms. Finally, from a lexical point of view, the CNN seems to focus on “concitoyens” (fellow citizens) which allows E. Macron to avoid the term “compatriots”, considered too nationalist in the 21st century, in the context of the European integration. A high wTDS also corresponds to the “nos” and “notre” (“our” and “ours”) forms as well to the lemma “notre”. Indeed, the construction of a political “we” appears as the main rhetorical objective of a discourse that aims at gathering the people behind its leader.
- 1 http://lstm.seas.harvard.edu/
35We have introduced and tested a new method to extract relevant linguistic objects characterizing the different classes/authors in a multi-class classification context. The main focus of the present work are the hidden layers of a trained CNN. In particular we introduced a measure (wTDS) which, entirely relying on the learned parameters, allowed us to detect the key words that, conditionally to their context, were used by the CNN to assign a text segment to its author. We have proposed a routine to rank the text segments from the most to the least representative for each author providing a new and different view in the author discourse analysis. The way we propose to compute all these features internally to the network leads to a highly reduced computation cost (compared to LIME for instance) and thus allows us to design a multi-channel architecture accounting for part-of-speech and the lemma leading to extract enriched linguistic objects at almost no cost. The linguistic objects that we learn in this multi-class classification framework are those better discriminating one author with respect to the others. In order to extract not only discriminative spatial linguistic objects (using CNNs) but to take into account the sequential generation of the discourse based on these linguistic objects, recurrent networks have to be considered. Some tools already explore the hidden layers of such architectures (e.g. LSTMVis1) and future works might focus on the combination of both approaches, for instance, first extracting spatial patterns then analyzing their sequential organization for an even more in depth discourse analysis.