1 Introduction

The naive Bayes classifier is still a popular method for classification, especially in text classification [23]. One advantage of the naive Bayes classifier is that it has the interpretation of a generative model that can be easily extended to model more complex relations [3] and can easily be extended for semisupervised learning [21].

The work in Ref. [20] showed that a background distribution (word uni-gram distribution) estimated from large unlabeled data can help to improve the naive Bayes classifier. However, as their method does not explicitly allow to control how much the background distribution can influence the estimation of naive Bayes parameters, their usage of the background distribution might be suboptimal.

In this work, we propose a different approach that explicitly models the background distribution by extending the generative model of naive Bayes. The proposed model assumes that any word in the document is sampled either from a class word distribution θ_z or from a background distribution γ. To decide whether a word is sampled from the distribution θ_z or from the distribution γ, we introduce a binary indicator variable d, one for each word in the document. Despite the simplicity of our model, our experiments show that our method can statistically significantly outperform previous methods such as [20] when the training data are small.

Furthermore, we theoretically interpret and analyze several properties of our model. One property of our proposed model suggests that our model’s usage of the background distribution has the effect of down-weighting high-frequent like “a” or “the” (stop-words) and other irrelevant words (see Section 5.1). Especially, if the number of training data instances is small, these words might by chance occur more often in one class than the other. As a consequence, irrelevant words are not spread evenly over all classes z and can degenerate a classifier’s performance. Another property of our model shows that the background distribution becomes more necessary when the number of labeled data is small (see Section 5.2).^[1]

The remaining of the paper is structured as follows. The next chapter describes the related work, followed by Section 3, briefly reviewing naive Bayes. Next, in Section 4 and Section 4.1, we explain our proposed model and explain how the hyperparameter can be learned using empirical Bayes. In Section 5, we proof some theoretic properties of our model. Finally, in Section 6, we present our experiments, and we conclude our work in Section 7.

2 Related Work

Our work is related to Jelinek-Mercer smoothing [as we will show in Equation (5)], which was proposed in the context of information retrieval [29]. In contrast, we theoretically analyze and experimentally evaluate the smoothing in the context of text classification. Furthermore, the work in Ref. [29] did not exploit the underlying probabilistic model for finding parameter estimates.

Feature weighting for naive Bayes can also be achieved by replacing the maximum likelihood (ML) estimate of the class conditional word probabilities by weighted frequency counts (tf-idf) and correlation-based feature selection [14, 16, 26]. Such ad hoc weighting schemes can be incorporated in our framework by using a word prior that is proportional to the desired weight.

The work in Ref. [21] proposed an extension of naive Bayes that models each class by a mixture of several word distributions (components) rather than only one word distribution θ_z. They assign to each class exclusively a fixed number of components. This way, the assumption that the documents of one class are generated from only one multinomial distribution is relaxed, and it is possible to model subtopics within one class. However, as one component is assigned only to one class, their model does not allow to share word distributions across different classes. As a consequence, their model is not able to model words that occur frequently in documents independently of their class. The method in Ref. [21] forms one of our baselines.

The idea of sharing word distributions across different classes (topics) is realized in latent Dirichlet allocation (LDA) [1]. However, it is known that this standard topic model is negatively influenced by high-frequent words (common words) and therefore is filtered out manually or weighted with the use of priors [25].

Several works proposed to extend the standard topic model to explicitly model common words. Therefore, as in our model, they suggest to use binary “switching variables” to include a background distribution. The extended model is applied to information retrieval [2], topic-aspect modeling [22], summarization [5], template extraction [19], and also text classification with labeled words instead of labeled documents [9]. However, to the best of our knowledge, we are the first to analyze the effects when combining this idea with a naive Bayes classifier and comparing it to other proposed extensions of naive Bayes.

The work in Ref. [8] proposed a different extension of naive Bayes that also uses additionally a background distribution. Instead of modeling the word occurrence probability using a multinomial distribution, they model it by applying the soft-max function (exponential normalization) to adjusted log-word frequencies. For each class c, they estimate the log-frequency deviation from the log-background frequency distribution. Furthermore, to enforce sparsity, they place a Laplace prior on the log-frequency deviations. In Section 6, we will compare their method to our proposed method.

More recently, Su et al. [24] and Lucas and Downey [20] also proposed two semisupervised naive Bayes methods that use the background distribution. Both approaches reestimate the conditional word probabilities p(w|c) for a class c by using the background distribution p(w) estimated from unlabeled documents. Su et al. [24] estimated p(w|c) by using p(c|w) estimated only from the labeled documents and p(w) estimated from the unlabeled documents. In contrast, Lucas and Downey [20] optimized p(w|c) subject to the constraint that p(w) should equal p(w|c₁)·p(c₁)+p(w|c₂)·p(c₂), where p(w|c₁) and p(w|c₂) are the probabilities that a word comes from a document that has label c₁ and label c₂, respectively. Their experiments showed that their method is superior to the method in Ref. [24]. We therefore adapt the method in Ref. [20] as another baseline for our experiments in Section 6.

The naive Bayes classifier is a generative model (a Bayesian network) that makes the strong assumption that the features (words) are conditionally independent given the class of the document. A natural direction for improvement is therefore to relax these independence assumptions by considering also class conditional bi-gram probabilities as, for example, in Ref. [15] or learning a binary decision tree [27]. In contrast, our focus here is on extending the generative model to allow the incorporation of a background distribution estimated from unlabeled documents.

Departing from the generative model of the naive Bayes classifier, several other improvements to naive Bayes classifier have been proposed: sample weighting [13], feature selection [30], and feature weighting [12, 31]. The latter is most relevant, as our proposed generative model also has the effect of feature weighting. In particular, in Ref. [31], the authors proposed the gain ratio weighted naive Bayes text classifier (GRWNB). ^[2] Their experiments show that GRWNB (combined with the multinomial naive Bayes classifier) is statistically significantly better than other state-of-the art feature weighting approaches such as χ² feature weighting. We therefore choose the GRWNB (with the binomial naive Bayes classifier) as another baseline in our experiments (see Section 6). However, we emphasize that, in contrast to all other baselines and our proposed method, GRWNB is not a valid generative model anymore.

3 Naive Bayes Model

Given the class z of a document, the naive Bayes model assumes that each word w in the document is independently generated from a distribution θ_w|z . A popular choice for this distribution is the categorical distribution. ^[3] Let us represent a text document t as (w₁, …, w_k) where w_j is the word in the jth position of the document. Under this model, the joint probability of the document with given class z is

where θ_·|z is the parameter vector of the categorical distribution, with ∑wθw|z=1.

Let us denote by θ all parameters {θ_w|z }_w,_z, for all words w and all classes z. Given a collection of texts with known classes D={(t₁,z₁), …, (t_n,Z_n)}, we can estimate the parameters θ_w|z by

argmaxθp(θ|D)=argmaxθp(θ)⋅p(D|θ)=argmaxθp(θ)⋅∏i=1np(ti, zi|θ)=argmaxθp(θ)⋅∏i=1np(ti|zi, θ),

using the usual iid-assumption, and that z_i is independent from θ. Furthermore, using Equation (1), we get ^[4]

argmaxθp(θ|D)=argmaxθp(θ)⋅∏i=1n∏j=1kθwj|zi.

For simplicity, let us assume that p(θ) is constant, and then the above expression is maximized by

where we define N_w,_z as the number of times word w occurs in documents belonging to topic z, and V denotes the set of words in the corpus. Finally, assuming some class probability p(z), a new document is classified using argmax_zp(z|w₁, …, w_k). The class probability p(z) is set using the ML estimate.

4 Proposed Extension of Naive Bayes Generative Model

We now describe our extension of the naive Bayes model as displayed in Figure 1. Under the proposed model, the joint probability of the text document with words w₁, …, w_k, hidden variables d₁, …, d_k and class z is

where the word probability p(w|z,d,γ,θ) is defined as follows:

p(w|z, d, γ, θ)={θw|zif d=1,γwif d=0.

Figure 1:

Proposed Model in Plate Notation.

The variables d_j are binary random variables that indicate whether the word w_j is drawn from the class word distribution θ_w|z or from the background distribution γ. The distributions θ_w|z and γ are two categorical distributions with ∑w∈Vθw|z=1, and ∑w∈Vγw=1. The variables d_j are hidden variables that cannot be observed from the training documents. To acquire the probability of a training document (w₁, …, w_k, z), we sum over all values for d₁, …, d_k, leading to

p(w1, …, wk, z)=∑d1,….,dkp(z)⋅∏j=1kp(wj|z, dj)⋅p(dj)=p(z)⋅∏j=1k∑djp(wj|z, dj)⋅p(dj),

where, to make the notation more compact, we skipped the conditioning of p(w_j|z,d_j) on γ and θ and p(d_j) on δ, respectively.

We assume that the prior probability p(d_j) is independent from the class of the document and independent from the word position j. Therefore, we define δ:=p(d_j=1), which is constant for all words. This way, the joint probability of the document with class z, i.e. p(w₁, …, w_k, z), can be expressed as follows:

For a class z, we estimate the word distribution θ_w|z as before using Equation (2). The class prior p(z) is also estimated in the same way as for the naive Bayes classifier.

Finally, to classify a new document w₁, …, w_k, we use

p(z|w1, …, wk)∝p(z)⋅∏j=1k((1−δ)⋅γwj+δ⋅θwj|z).

We set the background distribution γ to the word frequency distribution of the whole corpus (labeled+unlabeled documents).

We note that, with this choice of the background distribution, the result is actually identical to a naive Bayes model with the class-dependent word probabilities θ′w|z set to

which corresponds to Jelinek-Mercer smoothing [29].

In the next section, we will discuss how the hyperparameter δ can be learned.

5 Theoretical Analysis

In this section, we investigate the effect of the background distribution γ (see Section 5.1), the divergence of the background distribution estimated from the labeled and unlabeled data (see Section 5.2), and the effect of our model on the (labeled) training data likelihood (see Section 5.3).

6 Experiments

For our experiments, we used four standard corpora, 20 Newsgroups, Ohsumed, RCV1, and Reuters-21578, written in English. For the corpus 20 Newsgroups, we used the standard split of training and test data as suggested in http://people.csail.mit.edu/jrennie/20newsgroups/ with a total of 18,846 documents. For the corpus Ohsumed [10], we also used the standard split of training and test data with a total of 13,929 documents. RCV1 refers to the RCV1 corpus, with a total of 806,422 documents, as described in Ref. [18]. The Reuters-21578, described in Ref. [28], contains a total of 10,369 documents. ^[6] For 20 Newsgroups, we used all 20 classes, with the class proportions ranging from 3.3% to 5.3%. For Ohsumed, we used all 23 classes, with the class proportions ranging from 1.0% to 28.6%. For RCV1 and Reuters-21578, we used the top 5 classes as shown in Table 1.

We preprocessed the corpora using tokenization and stemming with Senna [4].

For 20 Newsgroups and Ohsumed, we used the whole test data. For RCV1 and Reuters-21578, we used as test data a random sample of 4000 documents of the whole corpus. For all four corpora, we used as training data a random sample of 50, 100, 300, 500, and 1000 documents of the remaining corpus.^[7] We ensure that the randomly sampled set of training data contains at least one document of each class.

We did not use a stop-word list, as it is partly domain dependent and therefore needs to be adjusted manually. As baseline methods, we used five methods: feature marginals-naive Bayes (FM-NB) [20], SAGE-naive Bayes (SAGE-NB) [8], EM-naive Bayes (EM-NB) [21], naive Bayes, which is identical to our method when δ is set to 1.0, and the GRWNB [31].

For GRWNB, we used Algorithm 1 of Ref. [31] with the binomial naive Bayes classifier.

For EM-NB, the hyperparameter is determined using two-fold cross-validation of the training data. For FM-NB, we also tried to change the Laplace smoothing with the Dirichlet prior but did not get any improvement. ^[8] For SAGE-NB, we used the original implementation. ^[9]

For our experimental setting, we assume that we have only access to the class label information from the training data and know all the test instances but without class label information. This experimental setting is the same as the one used in the original work of FM-NB [20]. ^[10] The estimation of the background distribution γ_w, which is used by our method and also by the baseline methods FM-NB and SAGE-NB, is calculated using the whole corpus. For our proposed method and the naive Bayes baseline, we placed a uniform Dirichlet prior over θ to prevent 0 word probabilities. Prior counts for each word, corresponding to the Dirichlet prior, are set to 1|V′|, where V′ is the set of vocabulary in the whole corpus. For our proposed method, we learn the hyperparameter δ using EM as described in Section 4.1 and stop if the change of δ is less than 0.0001.

For evaluation, we used the break-even point of precision and recall as defined in Ref. [17].^[11] The results of the (macro-averaged) break-even points are shown in Table 2.

Furthermore, we performed a pairwise comparison between all methods using the micro sign-test (see, for example, Ref. [28]) with the decision boundary induced by the break-even point. If method A was better than each other method B with p<0.01, we consider the result as statistically significant and marked the result with an asterisk in Table 2.

As we can see in Table 2, our proposed method statistically significantly improves over other proposed semisupervised naive Bayes methods when the training data are small (≤100).

In addition to the micro sign-test for one classification task, we also test for statistical significance across all classification tasks (i.e. different class labels and different corpora). We follow the guidelines in Ref. [7] to test whether the proposed method performs statistically significantly better than other NB methods across different classification tasks. First, we use the variation of the Friedman test proposed by Iman and Davenport [11] to check the null hypotheses that all classifiers perform equal. We use the Friedman test to compare the average ranking with respect to the break-even points, which are listed in Table 3 for different training data sizes. The resulting test statistics for training data sizes 50, 100, 300, 500, and 1000 are 36.4, 51.3, 49.1, 36.6, and 25.8, respectively. The test statistic is F-distributed with 6−1=5 and (6−1)·(53−1)=260 degrees of freedom. For all training data sizes (50, 100, 300, 500, and 1000), the test statistics is larger than the critical value of the corresponding F-distribution at significance level 0.05. We can therefore reject the null hypotheses. For post hoc test, we proceed with the Bonferroni-Dunn test with one control. This can be tested by comparing the average rank difference as explained in Ref. [7]. The critical difference (CD) for significance of at least p<0.05 is 0.94 and for p<0.10 is 0.85. Overall, we can conclude that our proposed method improves classification performance for small training data sizes (50–300). In particular, our proposed method improves statistically significant at p<0.05 over all other generative naive Bayes methods that use the background distribution (i.e. FM-NB and SAGE-NB).

Finally, we inspect some of the δ’s that were actually learned by our proposed method. Recall that the hyperparameter δ denotes the probability that a word is generated from a class word distribution θ_w|z rather than from the background distribution γ_w. In general, we should expect that, for small labeled training data, the distribution p˜(w) that is induced by θ_w|z gives a bad estimate of γ_w. Consequently, as we have proven in Section 5.2, we expect that, for small labeled training data, the use of the background distribution γ becomes more important, leading to a small value for δ. Indeed, inspecting some of the values for δ in Table 4, we can experimentally confirm this.

7 Conclusions

We analyzed a generative model for classification that models a word w being generated either from a class word distribution θ_w|z or from a background distribution γ_w. The decision whether a word is sampled from γ_w or θ_w|z is performed individually for each word by a Bernoulli trial with parameter δ. We explained that the hyperparameter δ can be learned using empirical Bayes (see Section 4.1).

Although similar models have been proposed in the past (see Section 2), we are the first to analyze and evaluate its usages in the context of text classification. Our theoretical and experimental analysis suggests that γ_w helps to model the misfitting of the unlabeled data to each class word distribution and can also be interpreted as a kind of feature weighting (see Section 5.1).

Furthermore, our experiments, in Section 6, using three standard text classification corpora showed that the proposed method can statistically significantly outperform previous NB extensions such as [20] and [8] that also incorporate the background distribution. Based on these results, we therefore think that there is strong evidence that a generative model for text classification should incorporate the background distribution as suggested in this paper.