Pruning and repopulating a lexical taxonomy: experiments in Spanish, English and French

2.1 Hypernymy and lexical taxonomies

The concept of hypernymy is fundamental for the semantic description of the vocabulary of a language [13], as it organizes the vocabulary into semantic categories. Entities can be hierarchically organized in categories and connected in a tree-shaped configuration, such that each node in the tree inherits the properties of its parent, e.g. a horse is a type of animal and therefore all properties that are common to animals will also apply to horses. The interest in hypernymy predates modern linguistics, and can be traced to the work of Aristotle, who outlined its basic properties in the Categories.

Hypernymy relations are the basic links organizing lexical taxonomies (where links are drawn between lexical units) as well as ontologies (where links are drawn between concepts). One would use taxonomies or ontologies depending on whether one has a semasiological or onomasiological perspective [45], the first being the traditional choice in linguistics and the second in artificial intelligence. Whereas an ontology requires a tree-like constraint, a lexical taxonomy does not, because nouns sometimes have more than one hypernym due to polysemy or differences in point of view (e.g., we can see a chicken or lettuce as a living organism or as a type of food).

Hypernymy relations,however, are common to both types of structures. Other properties that are common to both are:

1. Their hierarchic structure.

2. The shape of a direct acyclic graph (i.e. not containing cycles because words cannot be used in their own definition).

3. The asymmetry of the relation (1), e.g. if a robin (r) is a kind of bird (b), then a bird is not a kind of robin.

   (1) ∀ r , b ∈ T , r → b ⇒ ¬ b → r

4. The transitivity (2), e.g. if a robin is a bird and a bird is a kind of animal (a), then a robin is a kind of animal.

2.2 Taxonomy induction in NLP

Attempts at automatic extraction of hypernymy linkswere first made on machine readable dictionaries [8, 19]. Later, hypernymy links were induced from corpora using lexical patterns [10, 22, 37, 44, 54]. These patterns were found to have considerable precision but low recall, since both the hyponym and its hypernym must co-occur in the sentence or text span.

This limitation inspired quantitative methods, which were applied as syntagmatic and paradigmatic association measures. Syntagmatic association measures are well-known in linguistics. Mutual Information (MI) [9], for instance, has been extensively used in lexicography to detect collocations, multiword expressions and other types of syntagmatic bonds. The basic idea is to oppose the frequency of co-occurrence of two units against their independent frequency, as done by other measures such as t-test, chi-square and cosine, among others. The limitation for these types of measures is mainly that they are symmetric. Paradigmatic association measures, on the other hand, are derived from the concept of distributional similarity [2, 6, 14, 18, 27, 30, 46, 51].

Unsupervised methods based upon distributional similarity properties dominated the hypernym detection scene for a long period of time [11, 18, 25, 26, 27, 47, 48, 51, 56]. In such methods, the task is to assign a score to a pair of words (x, y) using a measure that exploits the distributional similarity hypothesis from [21], with high scores indicating strong possibility of an hypernymy relation. Recently, values obtained with distributional similarity methods have been incorporated into supervised methods as features, outperforming unsupervised methods, as discussed in [53]. However, it is clear that supervised methods are restricted by the availability of training data, which is costly and may be even unavailable in some cases.

Several measures have been proposed by other researchers, taking into account different distributional aspects of hypernym relations. According to [53], the classification measures can be similarity, inclusion, and informativeness. Some measures are framed around the concept of similarity since similar words appear in similar contexts, even though hypernymy relations are asymmetric. Some examples of similarity measures are: cosine similarity [47]; Lin similarity [27], which calculates the ratio of shared contexts of each word; and APSyn [48], that computes the intersection of contexts of a word pair weighted according to the rank of the shared contexts.

Other measures are inspired by the concept of inclusion, in which the contexts of a hyponym are expected to be included in those of its hypernym: X_c ⊂ Y_c. This idea is exploited by [56], who quantify the inclusion of x’s contexts by the contexts of y. Later, [26] combined cosine and Weeds precision. ClarkeDE [11] measured inclusion by quantifying the weighted coverage of the hyponym’s context by the contexts of the hypernym. Also, [25] designed a metric called balAPinc, that combines Lin similarity with APinc, an average precision measure taken from information retrieval. In [26], there is invCL, as a measure that takes into account the distributional inclusion of x in y and the distributional non-inclusion of y in x. Similarly, other measures are based upon the concept of reversed inclusion, in which the relevant contexts of a hypernym are expected to be included in those of its hyponyms [53].

Finally, some researchers have found inspiration in the concept of informativeness. In this context, hypernyms tend to be less informative than hyponyms, as they are likely to occur in more general contexts than their hyponyms. Cf.: SLQS [49], which measures the informativeness of a word x evaluated as the median entropy of its top N contexts; and RCTC [43], which measures the ratio of change in topic coherence.

According to the comparative study carried-out by [53], it is not possible to single out a preferred measure or parameter, even though the SLQS measure is slightly better than others when tested with English word pairs. However, as already mentioned, it is possible to incorporate scores of distributional similarity-based methods into supervised models as features. For instance, [16] presented a supervised model based on sense embeddings trained on available lexical resources such as Wikipedia, and [50], also using a supervised model, proposed an algorithm that combines different methods, such as word embeddings via word2vec, term substring and Jaccard similarity measures.

As we include in our work on vector-based representations of words, we consider it relevant, as well, to briefly review the three main approaches representing words: (i) Distributional representation; (ii) Cluster-based representation; and (iii) Distributed representations.

Distributional representation methods map each word w to a context word vector C_w, which is constructed directly from co-occurrence counts between w and its context words. The learning methods either store the co-occurrence counts between two words w and i directly in C_wi [23, 55] or project the co-ccurrence counts of words into a lower dimensional space [29, 42], using dimensionality reduction techniques such as SVD [15] or LDA [3].

Cluster-based representation methods build clusters of words by applying either soft or hard clustering algorithms [28]. Some of them also rely on a co-occurrence matrix of words [38]. The Brown clustering algorithm [5] is the best-known method in this category.

Distributed representation methods usually map words into dense, low dimensional, continuous-valued vectors, with x ∈ R^d, where d is referred to as the word dimension. In this category we find word embeddings, which take a text corpus as input and produce word vectors as output [12, 30]. The hypothesis is that similar words would have similar word embedding vectors. The method we propose in this paper is hybrid but mainly belongs to the first category.

3.1 Materials

3.1.2 The Core Ontology

Another resource that was used to conduct these experiments is what we call the Core Ontology (or CoreOnt, for short), a tree-shaped arrangement of ca. 300 of the most general concepts, starting with those such as entity, property, event, group and so on (Figure 1). This structure is used by the top-down modules.

Figure 1

A fragment of the CoreOnt

The CoreOnt is loosely based on the CPA Ontology ^[1], developed by Patrick Hanks for the Pattern Dictionary of English Verbs [20]. We freely adapted it to our purposes, adding or eliminating terms according to their frequency in corpus, and translated it to Spanish and French.

3.2 The pruning algorithm

The input of this algorithm is a list of suspected hypernymy pairs (a → h) and the output is a true or false value for each pair. As explained earlier, the objective for this pruning step is to obtain a reduced number of assertions with a high probability of being correct, from which to extract features that in turn are used to classify new nouns during the repopulation phase. The assertions that are likely to be true are denoted as matrix T_a,h, where a is a member element (a hyponym) and h is the semantic category (a hypernym).

The pruning is conducted according to a measure of syntagmatic association. Thus, the output of this first module for T_a,h is obtained from estimator M_a,h (3), which measures the frequency of co-occurrence in the corpus of the alleged hypernymy pair a → h. It can be said that M_a,h is a measure of the relative importance of h to a, and is therefore appropriate for asymmetric relations like hypernymy.

M_a,h is computed by the analysis of a random sample of contexts of occurrence of each hyponym a in the corpus, denoted as set C = {c₁, ...c_n} (with n <= 5000 to allow for lower processing time). The context window is ten words at each side of the target and each context C_i is represented as a bag of words, with no order and no repetitions. M can be used to compute a ranking of the most reliable assertions or just to return a Boolean if M_a,h > p, with p as an arbitrary parameter (experimentally set to 0.3). Additionally, a true value for a pair M_a,h is only issued if it passes the asymmetry test shown in (4).

3.3 The repopulation algorithm

The repopulation algorithm accepts one or more words as input and tries to classify them by adding them to the now cleaner version of the Legacy Taxonomy. It is composed of various modules, each of which is identified with a code name (e.g. asymmetric, paradigmatic, morfeo) and described in detail in the following subsections.

As already anticipated, much of the classification is done by extracting features from the cleaned up Legacy Taxonomy. Different modules use different types of features (e.g. distributional, morphological) but all of them are subjected to a process of weighting in order to select the most discriminant ones. We score the features according to a given semantic category, as shown in Equation 5.

Let F _i be some feature (e.g. a morphological one, like the final letter-sequence -itis); T_j a given semantic category (e.g. disease); f(F_i|T_j) the frequency of feature F_i in the category T_j (the number of hyponyms bearing this trait within the category) and D(F_i) the dispersion of said feature F_i, i.e. the number of semantic categories in which it is present. This score thus promotes features that are concentrated in few categories and that, at the same time, are productive in each category. To continue with the same example, this would be the case of finding a great proportion of words ending with the segment -itis among the population of words in the category of diseases.

Best features are kept in a matrix FM_m,h,f as shown in 6, where m stands for the name of the module, h for the hypernym (the semantic category), f for some feature and u is an empirically defined parameter, set to 0.001 in our experiments.

3.4 The central processor

In the repopulation process, a final decision is needed for the construction of a taxonomy chain for any given noun. This is where the central processor comes into play. It controls the calls to each of the modules explained earlier. The general structure of this module is represented as a flow diagram in Figure 2.

Figure 2

A flow-diagram of the central processor

According to the flow chart, given one or more input terms, it starts by taking and analyzing them one-by-one. The symbol a represents thus the input term. At this stage, one could argue that it is necessary to check, in advance, whether the input term a is not already in the Taxonomy T, but that would prevent us from discovering new meanings of an already known term. We will assume, however, that a ∉T.

Following the chart, we see it takes the syntactic head c of term a and if c ∈ T, then a result is found and the system moves to the next input term. Otherwise, then the td() function is called, short for top-down. This function, as well as its counterpart bu() (bottom-up), are explained in detail in what follows, but they are nothing more than a particular arrangement of the already presented modules. The symbol h represents a set of hypernym candidates that are obtained as the result of these operations, conducted upon the input term a as well as its head c and also to the hypernyms h_i that have already started to appear. At each time of the process, we check if any of the obtained hypernyms are known entities (i.e., h_i ∈ T). The final decision is taken by a ranking method function best(h), which selects the best hypernym-candidate chain according to how many modules voted for them. We compute those values in a matrix R to manage the contribution of each module, and obtain the final score of pairs a and h(Z(a,h)) as a weighted sum of each contribution (Equation 12).

Finally, and before deciding to promote the candidacy of a given hypernym, the system calls function prune(a, b), which takes as arguments the input term and the best hypernym candidate. This function returns a Boolean by calling the pruning algorithm already explained in Section 3.2.

If the input consists of a number of terms, including multi-word expressions, and there are reasons to believe that these terms belong the same specialized domain, there is then room for the application of some forms of pre-processing instead of working with the input terms one-by-one. For instance, the repetition, among this set, of certain head units would be a way to obtain relevant domain-specific semantic categories (e.g. disorder in a medical domain). This possibility is left, however, for future work.

4.1 Datasets for evaluation

We used three different datasets in our experiments: two developed by us and the other by other researchers. We will refer to these datasets by number: 1, 2 and 3.

As already discussed, different datasets have been proposed recently and used in the Semeval-2016 and SemEval-2018 tasks [4, 7]. However, we opted for the generation of a new evaluation dataset because we found a number of inconsistencies in the gold standards that are currently in use. Some of these seem to occur randomly, such as animal as a kind of airplane or lexicology as a sport. Others, in contrast, seem to be more systematic, such as the confusion between hypernymy and instance-of relations, (e.g. Nina Simone as a hyponym of person or Green Day as hyponym of rock band). These errors were more widespread during the construction of the original WordNet until, later, its developers acknowledged the problem and the need to distinguish between the two types of relations [32]. By then, however, the resource was already populated with assertions such as Aristotle is a hyponym of philosopher.

Thus, in order to obtain a high-quality dataset, we embarked on the task of manually curating a new, parallel trilingual dataset (Spanish, English and French), with 925 terms evenly distributed among the semantic categories of fish, machine, cheese and weapon. This is Dataset 1. In this dataset, each hyponym-hypernym pair is annotated with a Boolean value indicating its real status. This is because the sample also contains a number of false assertions with elements that are semantically related to the hypernym, but not in a hypernymy relation, like pizza and cheese or hatchery and fish. We added this extra difficulty to assess the performance of modules based on distributional properties. We also artificially inflated the number of false assertions by mixing hyponyms and hypernyms randomly and assigning zero values to their matches. This way, from the original set of assertions we obtain a new set of thousands where the false ones outnumber the correct ones by 3 to 1. Of course, this also makes it a harder task, as many of the categories are also semantically related: cheese and fish are types of food and weapons and machines are types of artifacts.

As units of the general vocabulary, this first dataset is restricted to single words only. We did this for simplicity and also to control the effect of hypernyms being the syntactic head of the hyponyms in the case of multiword expressions, as in applied linguistics and linguistics. The proposed algorithm does have, however, the capability of dealing with multiword expressions; therefore we also crafted a second dataset of 1767 terms from a specialized domain (psychiatric drugs). All this data was obtained from various sources such as dictionaries and encyclopedias, and each link was revised by pairs. We will refer to this as Dataset 2.

Apart from these two datasets, we compiled a third one from the gold standards shared in the literature. This third dataset is needed in order to compare our results with those of other researchers. It consists of materials taken from multiple sources, including WordNet and the Eurovoc database ^[3]. We will refer to this as Dataset 3.

4.2 Pruning and repopulation on general vocabulary

Figures of evaluation are estimated on the basis of the results of the automatic contrast against the reference value of Dataset 1, which indicates the real true/false value of each assertion. This allows us to represent the result of each trial in the standard terms of precision, recall and F1. The script that produces the comparison with the gold standard includes what we call a “flexibility table” (flextab, for short), to account for synonyms or otherwise acceptable answers from the algorithm. For instance, in Spanish there are two words for fish: pescado and pez. The first is used to to refer to fish that has been caught and is intended for eating, while the other one is used the denote the animal in general. In the case of edible fish, sometimes the algorithm proposes the hypernym pescado, and consequently the term is classified as a type of food. This is a problem of regular polysemy [1] but, with the flextab, the evaluation script will accept both forms as correct, even when it is only the form pez that is attested in the gold standard.

Table 1 shows the performance of the pruning and repopulation methods in the case of Dataset 1, the trilingual general vocabulary set. Overall, the algorithm achieved competitive F1 scores across languages and the four semantic classes with an average F1 of 0.82 and 0.83, respectively (the lowest F1 being 0.76 and the highest 0.90). Cross-language performance seems stable, although in French and Spanish the algorithm tends to perform slightly better than in English.

Figure 3 shows a plot comparing the precision and recall of the pruning algorithm for each language. There, it can be seen that an error within the first positions of the ranking is severely penalized, as is usual in this type of graph. This is the case, in English, of hatchery, incorrectly classified as a kind of fish. After that, however, precision seems to recover in the three languages and then gracefully decay, in this case after the .40 recall mark. From that point onward, it deteriorates quicker in the case of French, but by a relatively short margin (differences are between 0.9 and 0.8). To facilitate comparison, Figure 4 shows the result of the repopulation algorithm with a bar diagram.

Figure 3

Precision and recall plot of the pruning algorithm across languages using Dataset 1

Figure 4

Evaluation of the repopulation algorithm across languages using Dataset 1

As regards to error analysis, we found that incorrect results fell into three main categories of relatively equal importance in terms of proportion. The first category is made up of cases in which the algorithm proposes as hypernyms elements that bear a semantic relation to the input term without it being a hypernymy type of relation, such as quesadilla, a Mexican dish, incorrectly assigned to the category of cheese, or cases of meronymy, like in scale which is part of a fish rather than a hyponym of such category.

The second category of error consists of cases that are tagged as errors according to the gold standard and yet could be considered correct, such as artillery and bazooka, which are classified as machines in disagreement with the gold standard, where they are classified as weapons. The same occurs in French, for instance with arquebuse (arquebus) and mitrailleuse (machine-gun) and in Spanish with ametralladora (machine-gun) among others. One can argue, however, that these are indeed incorrect due to underspecification (the hypernym is not specific enough).

The third type of error consist of those that are attributable to polysemy, that is, when the algorithm produced acceptable results that, however, did not match those in the gold standard. Figure 5 shows a few such cases. Naturally, the algorithm is attracted to the most common meanings, as for example in the case of horse, which appears a kind of fish in the gold standard, same as dory. The word tomahawk appears as a weapon in the gold standard because of the missile. And we also see again cases of regular polysemy, as in the case of fax, which appears as a type of machine in the gold standard, but it can also be the document produced by this machine.

Figure 5

Some examples of results of the algorithm that are in disagreement with the gold standard due to polysemy

4.3 Individual and collective evaluation of the modules

We were interested in measuring the contribution of each individual module to the overall result; therefore, we tested the performance of the algorithm by repeating experiments and switching off one module at a time. Figure 6 shows an evaluation of each module using the leave-one-out method with Dataset 2, the psychotropics gold standard. As already discussed, specialized terminology is better for evaluating the performance of the different modules, since some of them are especially designed to work with multiword expressions. This figure shows how the team performs when one member is not working, and as it can be seen, the overall output seems to be remarkably robust in such scenario. The test shows the relative importance of module Morfrules and Palex, if any, but none appears to be indispensable.

Figure 6

Evaluating the contribution of each module to the overall output by leaving out one at a time, using Dataset 2

In Figure 7, in turn, we test the performance of the whole system by switching off more than one module at a time. Here again, we see the system is stable even after the loss of four modules. Excluding more modules results in dramatic loss of recall, but precision remains unaffected.

Figure 7

Evaluation of performance with different number of activate modules, using Dataset 2

4.4 Evaluation with Dataset 3

As already explained, in addition to the dataset prepared by ourselves, we experimented with material compiled by other researchers, in order to obtain a comparative evaluation, and this is what we called Dataset 3. Table 2 shows the results of the repopulation algorithm with various subsets in the three languages. We include again Dataset 2, the psychopharmacology dataset used in Subsection 4.3, to facilitate comparison. Dataset 3 was originally divided into training and test data. In our case, however, as we work with an unsupervised algorithm, we renamed them as corpus 1 and corpus 2 in order to avoid confusion (Medical and Music).

A qualitative analysis of the results reveals that here too the cases of errors fall in the same categories as in Subsection 4.2 Again we encounter confusion between hypernymy and other relations like meronoymy (airplane → engine) or others (airfield → air). We also encounter multiple instance of polysemy-related problems, such as classic rock → rock → mineral, etc. or, in French, guppy → transport while the gold standard says it is a type of fish. Finally, we also find here a large number of cases in which our algorithm is correct but the result is tagged as incorrect because of the aforementioned errors in the gold standard. In our results, precision is most affected in the subsets where incorrect links are concentrated, such as General and Food.

Despite the large number of incorrect relations, we see that results still compare well with those reported by other authors. The TAXI system [36], which ranked best in the Semeval-2016 shared task, performs with 0.26 precision and 0.34 recall (0.33 F1) in the Food dataset and with 0.37 precision and 0.38 recall (0.37F1) in the Science dataset in the case of English. The rest of the numbers fall within this range, and their average precision across domains is 0.33 precision and 0.32 recall (0.32 F1). For French, they report an average precision of 0.33 and 0.24 recall (0.28 F1).

The baseline used to compare results is the overlap between hyponym and hypernym. In the best case, this baseline achieved 0.62 precision and 0.28 recall (0.20 F1) in the Science domain in English and 0.87 precision and 0.26 recall (0.40 F1) in the Science domain of Eurovoc in French. The baseline shows a similar pattern across languages and domains: higher precision (0.57 on average in English and 0.58 in French) than recall (0.24 and 0.23 respectively).

With respect to other sets, the CRIM system [7] performed best in English in the SemEval-2018 shared Task, reporting .30 precision in the general corpus, .49 precision in the Medical 2 corpus and .48 precision in Music 2. Our results compare favourably here too, though not by a very large margin. In the case of the Spanish General set, the NLP_HZ system [40] performed best in this task reporting 0.21 precision, a result we improved by a larger margin.