Enriching Documents by Linking Salient Entities and Lexical-Semantic Expansion

3.1 Document Enrichment

In the following, we discuss the various steps for document enrichments.

3.2 Feature Selection and Ensemble Clustering

We utilize a clustering ensemble method, which combines different clustering results to finally partition documents. Although we exploit the same clustering algorithm to partition the input document corpus, as we can adopt different enrichment and associated vector representations of documents, the final clustering results may differ. The rationale of using ensemble clustering is that each single enrichment strategy may generally work for the whole corpus, but may introduce noise in the representations of a few documents that are eventually clustered badly. Ensemble clustering permits us to exploit many possible document enrichments and finally remove possible noisy results through a consensus method.

A cluster ensemble method consists of two steps: Generation, which creates a set of possible partitions of the input objects (in our case, a document corpus), and Consensus, which computes a new partition by integrating all the partitions obtained in the generation step [21].

In our experiments for the generation step, we adopt a hybrid function ℱ; indeed, many different instances {ℱ⟨}h=1,…,n of this function entail different feature selection methods, thus, generating different subsets of features and vectorial representation of documents. Therefore, ℱ models different possible enrichment strategies.

Hence, we consider the above multisets of words for each document D_i, denoted by SES(Di)={Ori,Sumi,Nai,Mei,Syi,Prei,Suci} and combine them using different instances of function ℱ.

Let ℂ={𝒞∞,𝒞∈,…,𝒞\} be the different clusterings of the document corpus 𝒟, where each clustering 𝒞⟨ is obtained by first applying the instance ℱ⟨ of the feature selection function over the corpus’s documents and, then, by running over them a given clustering algorithm. In our case, we exploit k-means, a well-known algorithm that takes the input document corpus and produces k disjoint clusters. Thus, each 𝒞⟩ is a partition required for the ensemble method. Formally, the enriched bag-of-words representation of D_i, obtained by ℱ⟨, is denoted by Dih, while the instance ℱ⟨ of the combining function is due to the different settings of six integer parameters, namely, αh,βh,γh,εh,δh, and η^h:

where αh,βh,γh,εh,δh,ηh∈{0,1,2,…,v} indicate the number of times we replicate the elements of SES(Di) to generate a new bag-of-words document representation Dih. More formally, αh⋅Sumi=⊎j=1αhSumi, where the operator ⊎ denotes the multiset union, and thus, Dih will contain α^h replicas of the document summary Sum_i. In case a parameter equals zero, for example, αh=0, then αh⋅Sumi is equal to ∅. In our experiments, we varied these parameters and used a different maximum value v for every parameter.

It is worth remarking that by varying the parameter setting to generate a different ℱ⟨, we may change the vocabulary used to identify the dimensions of document vectors, but we may also modify the term frequency and, thus, the tf.idf weights used in the vectors. As a consequence, if we enrich and represent a corpus D according to different ℱ⟨, we produce different partitions of the corpus even if we run the same clustering algorithms.

For the consensus step, we apply the objects co-occurrence approach, which is based on the computation of how many times two objects are assigned to the same cluster by the various clustering instances of the ensemble. Like in the cluster-based similarity partitioning algorithm (CSPA) [20], we, thus, build m × m similarity matrix (the co-association matrix), which can be viewed as the adjacency matrix of a weighted graph, where the nodes are the elements of the document corpus D, and each edge between two objects (documents) is weighted with the number of times the objects appear in the same cluster, for each instance of the clustering ensemble. Then, the graph partitioning algorithm METIS is used for generating the final consensus partition.

3.3 Words Sense Disambiguation

In this section, we discuss our unsupervised WSD method that uses WordNet as a knowledge base. Given a target word to disambiguate, we utilize the four above-mentioned semantic relations of WordNet to identify its best sense (meaning) among all the possible senses in WordNet. The disambiguation WSD strategy takes advantage of the word context, i.e. a portion of document that surrounds each word. The size of word contexts may be different, e.g. Unigram, Bigrams, Trigrams, Sentence, Paragraph, or different size of a window [16]. Determining such word context for a target word is crucially important because wrong relations between the target word and other words in the context may affect the best sense selection.

The novel idea of our approach is to limit the word sense disambiguation task to the terms included in M^(Di), while M^(Di) is also used as the context used by our WSD algorithm. As our word context M^(Di) is extracted by a summary including terms closely related to the main topic of documents, semantically related to each other, this should hopefully favor a fair selection of the appropriate senses for each target word.

Our approach proceeds as follows: given M^(Di)={w1,w2,…,wni} as the word context, where w_i is a word (noun) included in a mention of a salient entity, we create n_i semantic trees, one for each wk∈M^(Di), as illustrated in Figure 1A. The method works as follows:

Figure 1:

The conceptual shapes to visualize the relationships and information in (A) WordNet-Based Trees t₁ and t₂, Created for Words w₁ and w₂. (B) Graph G Build on the WSD Output for M^(Di), with the Cuts by METIS Algorithm.

1. First, for each wk∈M^(Di), associated with the root of a tree, we identify S(wk)={s1,s2,…,smk}, which includes all the possible senses (synsets) of w_k. From the root w_k, the tree is thus grown by adding mk=|S(wk)| children, where each child corresponds to a distinct synset sj∈S(wk);

2. For each sense s_j in S(w_i), currently a leaf of the tree, we denote by G(sj)={wg1,wg2,…,wghj} all the terms within the associated WordNet’s gloss. From each leaf s_j, we further grow the tree, by adding hj=|G(sj)| children, where each child corresponds to a distinct word wgt∈G(sj);

3. For each wgt∈G(sj), and for all sj∈S(wgt), we repeat step 1, and add another level to the tree, where the new leaves are the possible synsets associated with word wg_t.

Finally, our technique creates a forest of n_i indirected trees T(Di)={t1,t2,…,tni}, each of three levels and each associated with a distinct word of context M^(Di).

As explained above, we exploit four semantic relations of the WordNet ontology, namely, hypernym (SY), member meronym (SM), part meronym (SP), and substance meronym (SS). For each of these four relations, we can extract a directed graph (rooted DAG), where the nodes are synsets, and the edges are the semantic relations.

Returning to consider the forest T(D_i), for each pair of synsets s_i and s_j occurring in two distinct trees of T(D_i), if a directed edge between them exists in one of the four semantic DAGs, we add an undirected inter-tree edge between s_i and s_j, labeled as either SY, SP, SM, or SS. In our example in Figure 1A, these new undirected inter-tree edges are represented as dotted (labeled) links between pairs of synsets. These edges indicate that the two connected synsets are semantically related.

To extract the best sense for each word in M^(Di), we proceed through a voting process, using these semantic relations between pairs of synsets as a sort of “mutual vote” between them. The final goal is to rank, for each wk∈M^(Di), the synsets S(wk)={s1,s2,…,smk} occurring at depth 1 of each tree, for finally selecting the synset that obtains the highest vote. The voting mechanism works as follows:

1. First, we assign an initial vote to each synset s_i occurring in the forest of trees. This initial vote is simply the degree of the corresponding node, by only considering the dotted edges, labeled by either SY, SP, SM, or SS. The intuition is that a synset is important if it is related to others synsets occurring in other trees, in turn modeling the context M^(Di).

2. Second, we assign the final vote to each synset in si∈S(wk), by summing up the votes of all the synsets that belong to the subtree rooted at s_i, indeed the leaves at depth 3 of this subtree.

If the voting strategy discussed, so far, is not able to select the best sense for wk∈M^(Di), for example, because all the votes assigned to the synsets in S(w_k) are zero, then, we select the sense that was tagged for the highest number of times in the semantic concordances (Actually, it corresponds to the most common sense of w_k.).

Looking at the example in Figure 1A, the vote of s_m in tree t₂ should be equal to 1 only considering the inter-tree relations, as the degree of s_m is equal to 1 if we only consider its dotted edges. The final vote of s_m becomes 5, by also considering the contribution of the synsets in the subtree rooted at s_m – namely, s₁, s₂, and s_p – which contributes to the final vote by the quantity 2 + 1 + 1 = 4.

The reason for introducing a new WSD technique instead of using other methods, already proposed in the literature, is the different contexts used to disambiguate the senses of the target words, i.e. the words for which we have to identify the best senses. Instead of using the original sentence(s) including these target words, we limit ourselves to the words that are the most relevant with respect to the salient entities mentioned in the text. This context is less noisy, even if we have to pay an initial expensive step to first identify the salient entities and their mentions. Another reason is that our WSD method works on a WordNet semantic network, and the same elements of the WordNet networks, used to disambiguate and identify the best senses of words, are also used to enrich the vector representation of the original documents.

In Section 5.2, we compare our WSD approach with a lesk-based algorithm as a baseline, i.e. the adapted lesk algorithm [1], in order to evaluate the accuracy of our approach dealing with the contexts commonly used for evaluation.

4.1 Preprocessing and Evaluation Measures

Besides stop word removal, lower case conversion, and stemming (http://tartarus.org/martin/PorterStemmer/), we also identify sentences and paragraphs in each document. We also preprocess the WordNet ontology, to create data structures that allow a fast navigation of semantic relations.

For evaluating the clustering results, we use the well-known Purity measure [19]. For evaluating our WSD systems and comparing with baseline, we simply use accuracy, which is the percentage of correctly identified word senses.

5.1 Assessing the quality of the salient entities extracted by NG-rank

For a given document d belonging to a topic within DUC2002, we first create a set including all the words of the mentions related to the entities appearing in d, namely, DP(d). This step, which still leverages the Dandelion API, aims to prepare the ground truth used to assess our method to identify entities that are salient. We then take the five summaries SUiExp(d) of document d created by experts and included in DUC2002. Each SUiExp(d), i={1,…,5}, has a different length, as we use the DUC2002 summaries of 10, 50, 100, and 200 words, and perdocs one, respectively. From the summaries, we produce five corresponding sets of words SPiExp(d), where SPiExp(d) includes those words of SUiExp that are in common with the ones of DP(d). As a consequence, we can also rank the saliency of entities mentioned in d by considering the length of the DUC2002 summary where each word in DP(d) appears. An entity whose mention occurs in a 10-word summary is more salient than the one whose mention only appears in a 50-word summary of d.

To evaluate our summarization method, we created 13 summaries of different sizes by applying NG-rank, where SUjNG−Rank(d) denotes each summary. Afterward, for each document d, we can compare each SUjNG−Rank with the terms included in the various SPiExp.

Figure 2A shows the obtained results, where we plot the measure R to evaluate the covering of the salient entities mentioned in the document. R is maximum (equal to 1.0) when all the entity spots in SPiExp(d) are found in the summary SUjNG−Rank(d). The abscissas of the plot in Figure 2A indicate the summarization fractions of SUjNG−Rank. The ordinates are, indeed, the average R (percentage) for all the documents of the dataset, where each curve of the plot refers to a given SPiExp, i.e. the entity mentions occurring in expert-produced summaries of a given length in DUC2002. Note that the average numbers of words mentioning salient entities in the summaries generated by the NG-rank method are in the range of 7.78 words (summarization fraction = 1/3) to 37.26 words (summarization fraction = 1/15). These words cover the actual salient entities, as identified in the ground truth, for more than 93% in the worst case. For example, the text included in the NG-rank summaries of size 1/10 covers more than 99% of the mentions appearing in the expert-based 100-word summaries.

Figure 2:

The plots of the obtained results described (A) Covering of the Salient Entities Mentioned in the Documents. (B) Percentage of Mentioned Salient Entities in the Sections of DUC Documents.

Because NG-rank, i.e. the summarization method used in this paper, emphasizes the first and last sentences of each paragraph to summarize a document, we also investigated the average percentage of mentions to entities occurring in the different sections of the DUC2002 documents. As you can note from Figure 2B, the first sentences of each paragraph are remarkable, the section of a document that includes most of the mentions to entities present in the document.

5.2 Assessing the Quality of our WSD Method

To assess the quality of our WSD technique, we use a Python implementation of the well-known baseline (Liling Tan. 2014. Pywsd: Python Implementations of Word Sense Disambiguation (WSD) Technologies [software]. Retrieved from https://github.com/alvations/pywsd) adapted lesk algorithm [1], which makes use of the lexical items from semantically related senses within the WordNet hierarchies to generate more lexical items for each sense. Because in our WSD method the context used for disambiguating a target word only consists of nouns (i.e. spot nouns of salient entities within the summary of the document), for all tests operating on Senseval-1 (http://www.senseval.org/), we limit ourselves to only noun words for which there are WordNet mappings in Senseval. We compare our selected senses for the various target words with the “gold standard” senses, which are the sense that the human sense-tagging team considered correct for each corpus instance. The detailed results, concerning the accuracy of our method in comparison with the baseline for each target noun word in the gold standard, are shown in Figure 3A. The average accuracy for all the target nouns is shown in Figure 3B.

Figure 3:

The obtained results of assessing the quality of our WSD method that show (A) the Accuracy Obtained by our WSD Method and the Adapted Lesk Algorithm on 13 Noun Target Words. (B) Average Accuracy for all the Target Words, indicating a considerable improvement in result (shown in bold) compared to the Adapted lesk method.

Using the Senseval contexts, made of one or more sentences including a given target word, the results obtained by our WSD method are very good and are substantially better than the baseline.

5.3 Clustering Results

In this section, we finally evaluate how our overall algorithm (SEED) is able to improve the quality of text clustering. The algorithm adopted for clustering is always the k-means, while the vectorial representation of documents is based on a classical tf-idf weighting of terms, and the measure of similarity between vector pairs is the Cosine one. We utilize RapidMiner (https://rapidminer.cosm/products/studio/), which is an integrated environment for analytics and also providing tools for text mining. As stated in Section 4, for testing clustering quality, we use two subsets 𝒮∞ and 𝒮∈ of BBC NEWS.

In the first experiment on 𝒮∞, whose results are reported in Table 1, we evaluate the quality of clustering when we only use salient entities and the associated categories and spots to enrich the original documents. First, we show in Table 1(a) the results obtained when we exploit the state-of-the-art approach NEKW [3], which uses all the entities mentioned in a document, not only the salient ones as SEED.

From Table 1(b), we observe a slight improvement (1.37%) in the total purity of clustering when we use salient entities and their categories only, instead of using all the entities and their categories as NEKW does. The utility of only using the mentions of salient entities is shown in Table 1(d), in which we observe an improvement (0.9%) over a method that exploits categories of salient entities. However, the utility of using spots of salient entities in improving the quality of clustering is finally reported in Table 1(f), in which we can see a considerable improvement (3.67% in the total purity) over NEWK when we use the mentions of salient entities only or even using the spots of all the entities in Table 1(e) (2.72% in the total purity).

In the second experiment, we evaluate the quality of the document clustering obtained by our full SEED method, which also exploits WordNet to expand the mentions of salient entities. The results are reported in Tables 2 and 3. Table 2 shows 14 distinct partitions of the same collection, indeed, 𝒮∞ or 𝒮∈, obtained by clustering with document representations obtained using different instances of function ℱ, namely, {ℱ⟨}h=1,2,..,14. The first column of Table 2 indicates the ℱ-based representation of documents, whereas the other columns are the results of clustering using the specified representation. The range of parameters α,β,γ,ε,δ,η in ℱ⟨ is bounded by an empirical value v that takes into account the document lengths and the sizes of the various bags of words extracted from (or enriching) the documents, considering a threshold of minimum occurrence of words in the documents in order to limit repetition of the bags. The summaries produced by the NG-rank method are 1/3 of the original documents.

Table 2 also shows the utility of enriching documents by expanding mentions of salient entities. Consider that the best purity measures we can obtain by clustering with the original vectorial representations (plain clustering) of the two subsets 𝒮∞ and 𝒮∈ are 0.858 and 0.656, respectively. We can see the positive effect of using the PYC representation (WordNet-based enrichment) in the final clustering quality by looking at the 11th row: we improved by 8% on 𝒮∞ and by 14.63% on 𝒮∈ over plain clustering. Moreover, the use of document summaries and salient entities (see the EU representation at the 14th row) also improves the clustering quality: we obtained 5.7% and 11.43% of improvements on 𝒮∞ and 𝒮∈, respectively, over plain clustering. In order to improve uniformly the purity of all the clusters obtained, SEED combines the various clustering results by exploiting an ensemble method. The final clustering is, thus, a consensus one, whose results obtained by combining the 14 distinct partitions of 𝒮∞ and 𝒮∈ (shown in Table 2) are reported in Table 3(d) and (f).

For clustering 𝒮∞, as many document representations can be generated by replicating the bags of words in SES, we selected the top-N partitions (with the highest total purity) obtained using these representations. In this way, we obtained the 14 partitions shown in Table 2, where each partition is associated with a different instance of ℱ. We can consider this phase as a sort of learning phase, used to determine the best document enrichments. The same learned ℱ configurations were used to obtain the 14 partitions of 𝒮∈, also shown in Table 2.

SEED finally obtains 9.67% and 18.75% improvements of the overall purity – shown in Table 3(d) and (f) – on subsets 𝒮∞ and 𝒮∈, respectively, over the results obtained using the original document representations – shown in Table 3(a) and (b). We also observed that the results obtained by NEWK on subsets 𝒮∞ and 𝒮∈ improved by 7.8% and 14.4%, shown in Tables 3(c) and (e). It is worth noting that the best improvements were obtained by the consensus algorithm on subset 𝒮∈, where the types of enrichment, used for producing the various partitions to combine, were decided on subset 𝒮∞.