Contextualized Latent Semantic Indexing: A New Approach to Automated Chinese Essay Scoring

1.1 Background

AES originated from related research on English [29]. Earlier in the 1970s, Ellis Page developed the Project Essay Grader (PEG) for large-scale essay scoring upon the request of the College Board, which heralded the dawn of the field of scoring by intelligent systems and computers [24]. Since then, many studies have been conducted to improve the effectiveness and accuracy of AES, along with the emergence of commercial software in the education market and applied to industry. AES not only assists testers in assessing the proficiency of test takers, but also helps faculties and teachers to get knowledge about students.

PEG utilizes surface information to score essays, such as the lengths of passages, the number of words and misspellings in an essay, etc., but ignores deeper semantic structures, which makes PEG vulnerable: students can use more complex words, and write longer passages without any logical contents in essays to trick this system and get high scores. Intelligent Essay Assessor utilizes latent semantic indexing (LSI) to avoid the deficiencies of PEG and focuses on semantics instead of surface features [18, 19]. Educational Testing Service applies its E-rater to the writing section in the Test of English as a Foreign Language and analytical writing in Graduate Record Examination [1, 3, 27]. IntelliMetric, the first automated scoring system based on artificial intelligence combining natural language processing (NLP) with statistics, was developed by Vantage Learning [11, 31, 32]. Based on statistics and the Bayesian theorem, Lawrence M. Rudner developed the Bayesian Essay Test Scoring sYstem (BETSY) [28]. Similarly, BETSY uses lots of samples to train models, and evaluates database statistics and determines the classification accuracy.

Because automated English scoring emerged earlier and it is relatively easier to process English texts than Chinese ones, automated scoring systems for English have been applied widely. ACES, which started later, however, is left behind.

1.2 Related Work

Recently, ACES has attracted much more attention. Especially, when the scale of the MHK test grows larger, it is more necessary to replace conventional human scoring by AES. Unfortunately, there still lacks such a mature architecture caused by difficulties in processing Chinese texts.

For Chinese text processing, the first hurdle is segmentation. Unlike English or other Western languages, Chinese does not provide inter-word delimiters within a sentence, leading to one sentence possibly having different meanings according to different segmentations. Currently, there are several methods focusing on Chinese word segmentation, among which the integrated generative and discriminative character-based model proposed in Ref. [33] is the most comprehensive one. The methodology, however, needs to be adapted to ACES.

The second hurdle for ACES is feature extraction, which also exists in AES for any other language. Inspired by the studies for English, similar methods have been researched from various angles [16, 20, 26]. For instance, topic feature extraction methods using topic modeling strategies like regularized LSI (RLSI) [14, 34] and latent Dirichlet allocation (LDA) [2, 14] are novel ones. Other methods have also been tested; however, their results are relatively rare [4]. Semantic feature extraction using LSI [5, 36] explores the contents reflected by words, which is the most classical method. Currently, there are several LSI-based methods applied to AES [5, 15], which are as follows:

1. Conventional LSI

   LSI was proposed by Scott Deerwester et al. in Ref. [9], which starts with a term-document matrix D where every row d_i,: is a word and every column d_:,j is an essay. Typically, D is weighted by the term frequency-inverse document frequency (TF-IDF) method [23]. Next, LSI uses a mathematical strategy called singular value decomposition (SVD) to construct a semantic space by

   D ≈ D∗ = U∗Σ∗V⊤∗,

   where Σ^* is a diagonal matrix with σ_1,1≥σ_2,2≥…≥σ_k,k as the diagonal elements, and D^* is the least square best-fit approximation of D. Finally, after transformation, we can train a scoring model.

   According to the essays in MHK tests, sometimes there are few distinct differences between a higher-score essay and a lower-score essay from the perspective of word usage and occurrence merely. The reason may be that the essay with a higher score uses the words in an appropriate order and context, while the essay with a lower score does not. If two essays have identical words with different orders, they will get the same features when applying LSI to them, as Figure 2 shows. Consequently, they will get the same automated scores, which causes the so-called local overrating problem when low-score essays cannot be recognized, or the local underrating problem when high-score essays cannot be recognized.

2. Generalized latent semantic analysis

   Considering the lacking context information in conventional LSI, a method called generalized latent semantic analysis has been proposed [15]. By using n-gram as the atomic element of each row, it produces a very huge and sparse matrix D. For flexible languages such as Chinese, it is computationally prohibitive.

3. RLSI

   RLSI is a recently proposed method in Ref. [34] in topic modeling and information retrieval, including the batch version and the online version. The first study of scoring Chinese essays from the topic perspective has been used in Ref. [14], with the batch version of RLSI. The target of RLSI is to decompose a term-document matrix D into a term-topic matrix U and a topic-document matrix V, minimizing the Frobenius norm ||D − UV||F2 with ℓ₁-norm regularization on U and ℓ₂-norm regularization on V, which can be described as

   (1)minU,V(||D − UV||F2 + λ1||U||1 + λ2||V||F2),

   where U reflects the relations between documents and latent topics. In Ref. [14], RLSI is combined with support vector machine (SVM) to score Chinese essays.

   RLSI is a useful method to discover the topics in a collection of test essays and is effective to score essays from the topic perspective; however, a comprehensive scoring system requires more perspective.

Figure 2:

An Example Showing That a Meaningless Sentence (Sentence₂) Shares the Same Occurrence Information (Vectors S₁ and S₂) with a Normal Sentence (Sentence₁).

1.3 Main Contributions

Faced with the difficulties of ACES and the limitation of LSI, we propose contextualized latent semantic indexing (CLSI), which has these contributions: integrating Chinese text segmentation and context information extraction, scoring essays from the perspective of language fluency [25], and addressing the overrating and underrating problems. As mentioned earlier, LSI may cause overrating and underrating problems. In language tests, low-score essays weigh much more than ordinary essays, because they can be used to discover the latent learning problems of students and test takers. If a system cannot recognize low-score essays, it will lose important test information, and hurt the fairness of the tests to some extent.

Instead of the traditional term-document matrix recording the occurrence information, CLSI uses a probabilistic matrix reflecting the context information from an n-gram language model, solving the problems of both “What are these words?” and “How these words are used?.” More specifically, we propose two versions of CLSI: Genuine CLSI, which uses originally written essays, and Modified CLSI, which modifies possibly erroneous characters in essays. Experimental results show that both the Genuine and Modified versions are effective in feature extraction and AES.

The rest of the paper is organized as follows. In Section 2, we outline our methods and explain the terms used in this paper. Then, we thoroughly introduce all parts from Sections 3–5, including the language model construction, CLSI, and the SVM-based scoring model. Section 6 shows the experimental results and makes some discussions. Finally, in Section 7, we conclude this paper and point to our future work.

3.1 n-Gram Language Model

In NLP, an n-gram language model is a representative statistical inference strategy. Assume that a sentence S is a sequence of words:

Then, predicting the word w_n can be formularized as a probability:

Consequently, predicting a sentence S can be formularized as

A more common method is to use the logarithm of P(S), denoted as LP(S).

Formally, it is called a uni-gram when there is no dependency between the current word and other words. When the first-order Markov assumption is applied, namely, that the current word depends on one previous word, it is called a bi-gram. Similarly, we call it a tri-gram when the second-order Markov assumption is applied where the current word depends on two previous words. In this paper, we use the tri-gram language model, including tri-grams, bi-grams, and uni-grams. This model can be built on any large-scale corpus. In our study, we use a word list including 47,500 words, which are used as uni-grams to train an n-gram language model by the SRI Language Modeling Toolkit (SRILM) (http://www.speech.sri.com/projects/srilm/) [30], a very successful tool in speech recognition. During the training, we use common smoothing methods, including the linear interpolation and Kneser–Ney discount [7]. According to different corpora, some words are pruned, so the sizes of uni-grams are slightly different. As it is not the focus of this paper, we omit these details.

3.2 Conversion from an n-Gram Language Model to WFST

A finite-state transducer (FST) is a finite automaton whose transitions are presented as arcs with input and output labels on them [22]. When designated with probabilities on arcs, it becomes a WFST. In this paper, for language processing, WFST is adapted to a directed graph G=(S, A_forward, A_backoff), where S, A_forward, and A_backoff denote States, Forward Arcs, and Backoff Arcs, respectively.

1. We use ϵ (the epsilon state) as the very start of the proposed WFST. Each state S_i is defined as follows:

   Si = {arc0, arc1, arc2 … arcn},

   where these arcs are out-edges of S_i (except arc₀). In practice, because every state represents an n-gram with its lower order (n–1)-gram, we use arc₀ as the backoff arc whose probability is the backoff coefficient from the high order n-gram to its lower order (uni-gram to ϵ). ϵ is the starting state of a complete WFST, from which the decoding process starts. Additionally, each state can be regarded as a breakpoint for a segmentation and a start point for the remaining segmentation when decoding in a WFST.

2. Each forward arc A_i represents a word connecting two n-grams with a conditional probability of the word. That is,

   Ai = {Sin, Sout, word, probability},

   where S_in records the previous state and S_out indicates the next state. The word corresponds to the word of a term in an n-gram language model and the probability is the probability from the n-gram language model.

3. Each state has one backoff arc, namely arc₀. By passing through this arc, the decoding process moves from a higher-order n-gram state to a lower-order (n–1)-gram state. The structure of the backoff arc is similar to the forward arc, except that the word is empty and the probability is the backoff coefficient.

Figure 4 shows an example of a conversion from an n-gram language model to WFST. We see that a forward arc can start from the state {

Figure 4:

Conversion from an n-Gram Language Model to WFST.

We only show a small part of the whole WFST. In this figure, solid lines are forward arcs and dash lines are backoff ones. ϵ is the starting state.

4.1 Genuine CLSI

Given an essay collection 𝒟, WFST is used to recognize all essay word lists, and present each essay using a term-based essay vector d. After that, CLSI is used to extract each word with its context information, and thus finish the feature extraction.

CLSI has an advantage that it extracts context information along with segmenting. As we know, a sentence may have different segmentations, and hence different context information. Our objective is to recognize the most reasonable sequence in an essay and extract its context information as accurately as possible, that is, to find out a word list 𝒱={w_i} in the n-gram language model with the maximum probability, which can be formulated as

This task is a search problem in WFST whose goal is to find an optimal path from the starting state ϵ to a certain state. The intuition behind this method is obvious; that is, the more reasonable a sentence S is, the higher its probability LP(S) is.

Given a specific context, context information is used to decide whether a word appears in a reasonable way. The n-gram language model provides a general standard, constructed from a large corpus with probabilities, and judging whether a sentence is understandable or not, which is helpful for detecting fluency and coherence among words. In Figure 2, there is no explicit information about how these words are organized in the two sentences. However, if we consider the processing in WFST and the n-gram language model, and replace term frequency or TF-IDF with probabilities, we will get a different result as Figure 5 shows. A simple calculation, like summation, can make the difference: PL(S1)>PL(S2), which makes sense because Sentence 2 is meaningless with an unreasonable sequence of words.

Figure 5:

Context Information Applied to the Example in Figure 2.

When we consider context information extraction using WFST and the n-gram language model, we see that the two sentences are not identical anymore.

Context information provides an important indicator of scoring essays. According to our assumption about test essays, Sentence 1 may appear in good or excellent essays, whereas Sentence 2 may appear in poor essays. Thus, CLSI is better than LSI because it can make more obvious difference between poor essays and good ones. Figure 6 illustrates how to recognize a word list in a sentence using WFST, and Algorithms 1–4 show the details of Genuine CLSI.

Figure 6:

An Example Showing How to Recognize a Word List in a Sentence Using WFST.

For clarity, we separate the WFST into three parts, though actually they are in the same WFST.

In WFST, each possible solution is represented as a candidate, consisting of a segmented string (segmented), an unsegmented string (unsegmented), a log probability of the segmented string (probability), a current state (state), a word list (𝒱), and an essay vector (d). In Algorithm 1, to recognize a word in a sentence will pass a forward arc according to the word on the arc (arc.word), which is a step of context information extraction. A candidate set is used to define the set of promising solutions. As there is more than one candidate caused by different paths, we use a beam width to maintain the size of the candidate set to avoid unnecessary expansions. We sort the members in the candidate set in descending order, and prune the set during each step of segmenting, according to the members’ probabilities.

Algorithm 1

Context Information Extraction Using WFST in CLSI.

In each iteration, we first pass each candidate S through all of its backoff arcs, arriving at its corresponding lower-order states (Algorithm 2). For example, if one candidate is stopping on a tri-gram state, we pass it to its corresponding bi-gram state, and then to the uni-gram state, and finally arrive at ϵ. Thus, three new candidates Si∗(i = 0, 1, 2) will be generated whose current states are the bi-gram state, the uni-gram state, and ϵ, respectively, and corresponding log probabilities on the backoff arcs will be added. Although no segmentation happens, we still extract some important context information by using the statement “Apply a certain Punishment Rule,” in Algorithm 2, which will be explained later.

Next, in each iteration, we check every arc starting from the current state of the candidate S (Algorithm 3). Note that in line 16 in Algorithm 1, we use the statement “Judge Passable Forward Arcs in Genuine/Modified CLSI,” which means in the Genuine version, it will call the function in Algorithm 3. If the unsegmented part of S starts with the word of the forward arc arc.word, then we pass this arc, and transduce the state to its arriving state (Algorithm 4). The word on the arc will be added to the word set 𝒱, and the corresponding word in the unsegmented part will be eliminated. Simultaneously, we extract the context information arc_i.probability from the arc and append it to the tail of the essay vector d (note that we use ∪ to denote the appending).

In WFST, backoff arcs are crucial, because usually when a sequence w_i, w_i+1 is unreasonable, it needs to backtrack from the higher-order state {w_i} to the lower-order state. Therefore, adding a backoff coefficient lowers the probability, meaning that this sequence may be meaningless. However, a problem is encountered, that is, of which word, w_i or w_i+1, it is supposed to add the backoff coefficient to the context information. There exist three assumptions. First, we assume that w_i is correct but it is unreasonable to observe w_i+1 after w_i. Second, we assume that w_i+1 is correct but it is unreasonable to observe w_i before w_i+1. Third, we do not introduce backoff coefficients to the essay vector d. Based on these assumptions, we give three Punishment Rules, and Figure 7 can help understand this procedure.

• Punishment Rule 1: Add the backoff coefficient to the first word, meaning that the first word is not appropriate while the second is.

• Punishment Rule 2: Add the backoff coefficient to the second word, meaning that the second word is not appropriate while the first is.

• Punishment Rule 3: Do not add any backoff coefficient.

Figure 7:

Three Punishment Rules.

In this figure, p1 and p2 stands for the probabilities, and pb for the backoff coefficient. LP(word_i) is the context information of the i^th word. Note that the plus sign in this figure may have different meanings in the experiments, which we will explain later.

In practice, we will choose a Punishment Rule that performs best. In Algorithm 2, if we use Punishment Rule 1, we will add the backoff coefficient to the last dimension of the essay vector d; if we use Punishment Rule 2, we will append the backoff coefficient to d in advance. Thus, when the next word is recognized, its arc.probability will be added to this dimension; applying Punishment Rule 3 means skipping the statement “Apply a certain Punishment Rule.”

4.2 Modified CLSI

Essays with erroneous characters may interfere with both a human rater’s and a machine’s scoring. Moreover, automated essay correcting can help students learn languages. As for erroneous characters, for example, “

The difference between Modified CLSI and Genuine CLSI is that Modified CLSI detects and corrects erroneous characters before extracting context features. For instance, if we apply Modified CLSI, the erroneous character “

Generally speaking, there are four kinds of errors in essays, which are substitution, deletion, insertion, and transposition. By manual statistics from 200 essays from the MHK tests, we find that the most common kind of error is substitution, as Figure 8 shows. Therefore, we focus on substitution.

Figure 8:

Examples of Four Kinds of Errors with Their Proportions.

A key component of Modified CLSI is the confusing-character table, including approximate and homophonic characters, as shown in Figure 9. We do not separate these two kinds of characters, because those approximate characters are usually homophonic with different tones in Chinese.

Figure 9:

Examples of Approximate and Homophonic Characters.

The two homophonic characters are pronounced “tóng” and the three approximate characters are pronounced “fen” with different tones.

Figure 10 is a part of the confusing-character table. Characters in the same line are easily confused ones. Thus, when tentatively extracting a word and its context information in WFST, we can detect probable erroneous characters, and replace them with characters in the same line. For a certain character, its confused characters have the same chance to replace it. Figure 11 shows how this table helps detect and correct erroneous characters.

Figure 10:

A Part of the Confusing-Character Table.

Figure 11:

An Example to Show the Use of the Confusing-Character Table.

In this example, the second character “

(field)” of the input sentence is misused. If we use Genuine CLSI, we keep it unchanged, and thus, pass only one forward arc. When we use Modified CLSI, if we replace the erroneous character with its corresponding characters in the confusing-character table, then there will be another forward arc to pass, and thus, the erroneous character is corrected. “

” has several confusing characters in the table, like “

(sky).” As the word “

(today)” includes one identical character “

” to the input sentence, and a confusing character “

,” this arc is passable. Therefore, using Modified CLSI will have two possible word sets 𝒱₁ and 𝒱₂, whereas using Genuine CLSI has only 𝒱₁.

The only difference between the algorithm of Genuine CLSI and that of Modified CLSI is the judgment of whether an arc is passable or not. Therefore, we still use Algorithm 1 as the framework of Modified CLSI. However, in line 16, Algorithm 5 is used to replace Algorithm 3.

4.3 Constructing Contextualized-Semantic Space

The first step to construct a contextualized-semantic space is to convert essay vectors into an essay matrix. After that, SVD is used to reduce the dimension and construct a less noisy contextualized-semantic space. Thus, any term-based essay vector can be transformed to that space, which provides a uniform metric under the same space for automated scoring.

4.4 Presenting the Essays Under the Space

Any essay vector is supposed to be projected to the contextualized-semantic space, in order to gain a uniform representation under the space for the scoring model. Therefore, given a term-based essay vector d=[d₁, d₂, …, d_M]^⊤, it can be projected as follows:

where d^*∈ℝ^K. So far, we complete the whole processing of CLSI methods.

6.1 Data Sources

1. Language models (LM): We have three kinds of corpora for training n-gram language models: Literature, Mixture, and News. The Literature corpus includes all the articles awarded the Mao Dun Literature Prize, the highest honor for Chinese writers. The Mixture corpus includes various sources like micro-blog posts, magazines, web pages, and so forth. The News corpus selects from People’s Daily, the bestseller newspaper in mainland China. These corpora are very representative, which stand for daily use of Chinese, mixture of language usage, and standard Chinese, respectively.

2. The confusing-character table: The confusing-character table comes from Modern Chinese Dictionary, including the most confusing homophonic and approximate characters. Additionally, according to previous research about the MHK tests, we manually add some frequently confusing characters. Thus, the table includes 6674 characters in total.

3. The sample set: We use the written essays with their human scores from the MHK test assigned by a same topic as the sample set. It is easy to see that the high-score essay usually contains fluent sentences, whereas the poor essay presents incoherent sentences. Omitting the empty essays and those without human scores, we obtain two training sets with the sizes of 10,000 and 1400, a validation set with the size of 600 and a testing set with the size of 600. The training set is used to train a model; the validation set is to determine details of models due to various settings; the testing set is to test and make comparison among different versions of CLSI, conventional LSI, and other methods. Figure 13 shows the human score distributions of these sets.

Figure 13:

The Human Score Distributions of Training Set 1 (A), Training Set 2 (B), the Validation set (C), and the Testing Set (D).

6.2 Performance Criteria

There are several criteria from different scopes to evaluate the performance of an AES system [35]. In this paper, we use Rating Agreement, Fairness Coefficient, Quadratic Weighted Kappa, Spearman’s Correlation, and Pearson’s Correlation.

1. Rating agreement: Rating agreement is an overall evaluation criterion, including exact agreement and adjacent agreement [10]. The exact agreement calculates the exact equal scoring results between human scores and predicted scores, whereas the adjacent agreement is used to count the results with an acceptable bias. In our experiments, we use adjacent agreement (R.A.):

   (14)Rating Agreement(bias) = ∑i = 1N1{|hsi − psi| ≤ bias}N,

   where N is the size of the testing set; hs_i and ps_i are the human score and predicted score of the i^th essay, respectively; and bias can be set to an acceptable value according to various tests. In the experiments, we set bias to 1 according to the scoring criteria of the MHK test. The indicator function is 1{TRUE}=1 and 1{FALSE}=0.

2. Fairness coefficient: Fairness coefficient looks into each class. From the human score distribution in Figure 13, we see that the majority lies in the score of 3.5. If the classifier scores all essays 3.5, R.A.(1) can rise up to 83.5%, which apparently violates the fairness of the test. Some poorly written essays may get higher scores than they should have, whereas some excellent ones may get lower scores. Therefore, we design a criterion named fairness coefficient (F.C.):

   (15)Fairness Coefficient(bias) = ∑i = 1C(log(niN) ⋅ R.A(bias)i)∑i = 1Clog(niN),

   where C is the number of classes, N is the size of the validation or testing set, n_i is the size of class i, and R.A.(bias)_i is the Rating Agreement of class i with the bias. The reason for using this coefficient is because low- and high-score essays weigh more but have less number than ordinary essays in the sampling set. We increase their weights by using the logarithm of their proportions, focusing on the minorities.

3. Quadratic weighted kappa: Quadratic weighted kappa κ takes chance agreement into account because there exists the risk that a system just uses a baseline classifier (which will be explained later) and guesses all the scores by pure chance. To calculate this value, we can construct a confusion matrix, showing the human scores and the predicted scores.

4. Spearman’s correlation: As in evaluations of other scoring systems, we calculate Spearman’s correlation (Spearman).

5. Pearson’s correlation: For the sake of completeness, we introduce the last criterion, Pearson correlation (Pearson).

6.3 Experimental Results with Different Models

6.3.4 Sensitivity to Dimension

In previous experiments, dimension K in truncated SVD is set to an empirical value 100. This value is important because it will produce much noise if it is too high, or lose information if too low. In order to test the sensitivities of conventional LSI, Genuine CLSI, and Modified CLSI to this value, we run experiments from K=25–200 at intervals of 25, and plot them in Figure 14.

Figure 14:

Comparisons of Conventional LSI, Genuine CLSI, and Modified CLSI with Different Dimensions (K).

(A) Rating Agreement; (B) Fairness Coefficient; (C) Quadratic Weighted Kappa; (D) Spearman’s Correlation; (E) Pearson’s Correlation.

In Figure 14, we see that for R.A.(1), the performance of conventional LSI decreases sharply as K increases, whereas those of CLSIs decrease more smoothly and are better than the conventional method. For F.C.(1), the performance of conventional LSI fluctuates continually, whereas those of CLSIs remain relatively stable and are better, too. Hence, we conclude that CLSIs are more stable considering the dimension K. For the other three criteria, κ, ρ(Spearman), and ρ(Pearson), we see clearly that CLSIs stay relatively stable, but conventional LSI fluctuates and is worse than CLSIs.

These experiments justify that CLSIs are less sensitive to K, meaning that K performs well no matter how much latent semantics is required, which is of great importance, because it is often hard to find the appropriate dimension K, and less K means less memory usage.

6.4 Further Experiments on Selected Models