Neural Machine Translation System for English to Indian Language Translation Using MTIL Parallel Corpus

1 Introduction

Machine translation (MT) is the process of translating text in one language to another language with the use of software by incorporating both computational and linguistic knowledge. Initially, the MT system finds the translation of a text in the source language by simply associating meaning of the words in the source language to the target language with the help of linguistic rules. However, such methods did not yield good results because of its inability to capture various sentence structures in the language. This translation process is highly time consuming and also required people who have proficiency in both languages. Then, corpus-based translation approaches like statistical machine translation (SMT) [21] and neural machine translation (NMT) [4], [8], [19], [41] were introduced with a vision to rectify the drawbacks of rule-based methods.

NMT is a recently formulated method for automatic translation with the help of deep neural networks. NMT has already shown promising results in translations of several language pairs. Unlike SMT, which requires separately trained sub-components for translation, NMT uses a single large neural network for training. This structure comprised of encoder and decoder networks where the encoder consumes the input sentences to produce a vector representation, and the decoder takes this vector and outputs the target language words. Generally, both encoder and decoder networks are designed using the recurrent neural networks (RNN) [14] or long short-term memory (LSTM) or gated recurrent unit (GRU) [9] or bidirectional RNN [32], which are the alternatives to RNN. Even though RNN, especially LSTM, is theoretically proven for handling long-term dependencies in the sentences, practically, translation of long sentences still remains an unsolved problem. This issue was tackled with a technique called attention mechanism [4], [24]. The basic working principle of attention mechanism is to look into different parts of the source sentences, which contributes maximum to the prediction of each target word and translate, instead of taking the entire sentence and translate. This, in turn, helps to align the source and target language words. However, the performance of an NMT system relied greatly on the size and quality of parallel corpus because unlike the SMT system, which has a separate language model, the NMT directly finds the probability of a target sentence for a particular source language sentence [19].

Unfortunately, automatic translations in several language pairs suffer less improvement mainly due to the unavailability of good-quality parallel sentences in abundance. This is the primary problem for the machine translation research in Indian languages not to take a big step. For this reason, the first objective of this work was to create a parallel corpus, which covers sentences from diverse areas and of good quality and large in number and model an NMT system with these sentences in four language pairs (English–Malayalam, English–Tamil, English–Hindi, and English–Punjabi). The motivation behind this was to support machine translation research in resource-poor languages such as Malayalam, Tamil, Hindi, and Punjabi. There are a few parallel corpora available in these language pairs. However, they are not readily usable for research because of the amount of impurities such as spelling mistakes, presence of unknown characters, and presence of bilingual sentences, which do not convey proper meaning in those corpus. To use such data for research, the researchers have to clean it or prepare a separate corpus, both of which are highly time consuming. Creating a parallel corpus is a herculean task especially for language pairs where the availability of bilingual sentences in digital form is very less. In order to avoid such unnecessary time delays, we prepared parallel corpus for four different language pairs by collecting numerous sentences from many resources, which spread across several domains and cleaned them to make good quality corpus. The second objective was to implement NMT for the above-mentioned language pairs by training the system with the prepared corpus. Not many research have been reported on the application of deep neural networks for the translation of language pairs containing Indian languages, and particularly, no other NMT system has been reported for English–Malayalam translation besides Google’s NMT system [18]. The third goal of this research was to study the impact of attention mechanism in dealing with long sentences (sentences with the number of words more than 50). Though neural network architectures like the RNN and LSTM networks are theoretically proven capable of remembering long-term contexts; still, the deep learning architectures find it difficult to translate lengthy and complex sentences. Attention mechanism was introduced to address this difficulty in translation. However, still, there are problems in translating lengthy sentences. For this purpose, from the collected sentences, we keep longer sentences along with shorter ones without breaking them into short sentences. Analysis showed that when the number of words in the sentence is very large, the algorithm will not be able to capture the dependencies properly. This will affect the quality of the entire system. The results showed that the presence of a longer sentence affected the accuracy of translation in the English–Malayalam and English–Hindi systems. However, in English–Tamil and English–Punjabi, the number of sentences with a word length of more than 50 was very few. This is one of the reasons why those two systems achieved state-of-the-art results. In the English–Malayalam and English–Hindi corpus, there were a lot of lengthy sentences. To achieve a NMT system that produces quality translations, one of the important things to be taken into consideration is the size of the sentences in the corpus. So, in the corpus, the number of longer and shorter (word length less than four) sentences should be as minimum as possible.

All the four systems were trained with the same architecture and different corpus, and these systems were evaluated using the automatic evaluation method, BLEU (bilingual evaluation understudy) score [26] as well as the human evaluation metrics [11].

The structure of the paper is as follows: in the next section, related work in the field of machine translation in the English–Indian languages is discussed. The statistics and description about the parallel corpus prepared for machine translation research are given in Section 3, and the methodology behind NMT is described in Section 4. The details about the experiments and he analysis of the results are discussed in Section 5, and finally the paper concludes in Section 6.

2 Related Work

Hindi, Malayalam, Tamil, and Punjabi are among the most widely used languages in India [44]. Also, Hindi and English are the official languages in India [30]. Most of the government documents are available in English and Hindi, and they should be translated into other widely spoken languages to reach the common people. Similarly, an English–Hindi or Hindi–English translation system can help government officials to prepare documents in either English or in Hindi and translate into other languages. Thus, translation from English into these four languages has very much importance in our country.

Much research works have been done on machine translation from English to Indian languages, mostly focusing on ruled-based methods due to the unavailability of good parallel corpora. In spite of the unavailability of sufficient parallel corpora, significant works were done using statistical as well as hybrid approaches to translate text from English to Indian languages. In this section, we will discuss the notable works done in English to Hindi, English to Malayalam, English to Tamil, and English to Punjabi machine translation.

Anglabharti [37] (IIT, Kanpur, India) is one of the oldest machine translation systems in India. It was a multilingual translation system that translates English sentences into Indian languages. This work followed a pseudo-target approach in which the source sentence was converted into an intermediate representation, and the target sentences were generated from this intermediate representation using a text generator. AnglaHindi [36] is an extension to AnglaBharti. In addition to the techniques used in Ref. [37], AnglaHindi (IIT, Kanpur, India) used an example-based approach to translate frequently occurring noun and verb phrases. Anusaaraka [5], [7] was another initiative put forward by the Indian Institute of Technology, Kanpur, which used the principles of Paninian Grammar for translation. Anusaaraka had two modules: the first module does the language-based analysis of the text, whereas the second module performs the statistical analysis.

MANTRA-Rajyasabha [6] developed by CDAC-Pune, India is a machine-aided translation tool primarily designed to translate government documents from English–Indian languages and vice versa. In this approach, both source and target language grammars were represented using the lexicalized tree-adjoining grammar (LTAG). CDAC-Mumbai, India developed a machine translation system called MaTra [25], which relied on a transfer-based approach for translation.

Dave et al. [10] proposed a machine translation approach based on interlingua. The method was designed in such a way that information extracted from sentences in source language are converted into a universal networking language (UNL), and target language sentences are generated from this UNL representation.

Two systems called Shiva [16] and Shakti [15] were developed by the Indian Institute of Science, Bangalore, India, and the International Institute of Information Technology, Hyderabad, India, in association with Carnegie Mellon University. The Shiva system is based on the example-based approach, while the principle behind Shakti is a hybrid approach, which incorporated rule based as well as SMT techniques.

Super Infosoft Pvt Ltd., Delhi, developed an English–Hindi machine translation system named Anuvaadak [3], which used built-in dictionaries for specific domains. This system had a provision to handle out-of-vocabulary (OOV) words by transliterating them. IBM India Research Lab, New Delhi, developed an SMT system [42] that used IBM models 1, 2, and 3. Other notable works in English–Hindi translation were done by Jadavpur University, IIT Delhi [38], and most of them followed the example-based statistical approach.

Some rule-based approaches toward the English–Malayalam automatic translations were proposed by Refs. [13] and [29] and Sunil et al. [40]. Apart from rule-based approaches, various studies were done on SMT in the English–Malayalam translation. In Ref. [28], Rahul et al. proposed a rule-based approach to modify the SMT output. The source language syntax was changed according to the syntax of the target language using some transformation rules and also using a morph analyzer for separating the suffixes from the root words in both languages. Sebastian et al. [33] used parts of speech (POS) information along with bilingual corpus to improve the alignment model of an SMT system. Unnikrishnan et al. [43] incorporated morphological and syntactic knowledge to the SMT system to improve the quality of translation. They claimed that usage of morphological information in an SMT will reduce the size of parallel corpus to train the model substantially.

One of the initial researches in the English–Tamil machine translation was done by an NLP group [2], which was based on a hash array mapped trie (HAMT) approach. This translation system was constituted by three modules – an English morphological analyzer, a mapping unit, and a Tamil morphological generator. A web version of the English–Tamil translation system was developed by Renganathan [31]. It made use of a morphological tagger for translation. Anand Kumar et al. [1] proposed a factored approach for the English–Tamil translation, which used POS-tagged information and morphological information extracted from both languages to assist in the translation. In Ref. [46], authors applied certain rules to separate suffixes in both source and target languages and fed these information to a SMT system. Sridhar et al. [39] proposed a methodology, which used universal networking language as an intermediate representation in the English–Tamil translation. Ref. [12] is one of the latest research in English–Tamil machine translation in which the authors incorporated deep neural network for translation and further improved the system by segmenting the words morphologically. This improved the translation quality as well as reduced the target language vocabulary size by a factor of 8.

For the English–Punjabi translation, Gurleen Kaur Sidhu and Navjot Kaur proposed a hybrid approach [34] in which they used a word sense disambiguation module to identify the correct sense of words, which will, in turn, improve the translation. Parteek Kumar, in his PhD thesis, [23] proposed the UNL-based approach for the English–Punjabi translation. Another notable work in this area is Ref. [35], which is a statistical approach toward translation and the Anussaraka system [5].

Recently, the application of deep neural networks is making tremendous improvement in machine translation. However, not much studies have been done in English to Indian language translation, specifically in Malayalam. This is because the NMT requires relatively massive amount of good quality bilingual corpora. However, Google came up with a very good translation system called Google Neural Machine Translation (GNMT) system [18], [45], which used millions of samples for translation. The architecture of the Google NMT system consists of LSTM networks in which both the encoder and decoder network contains eight layers each. They also implemented attention mechanism to understand the contextual dependencies in the text and also divided words into a finite set of common word pieces to address the problem of occurrence of unknown words.

3 Parallel Corpus

The current research in machine translation demands huge parallel corpus because SMT and NMT systems are based on the probabilistic models generated using the features extracted from the parallel corpus. In order to derive better features, sentences in the corpus should be of good quality. Therefore, parallel sentences should convey the same meaning, and the presence of spelling mistakes and grammatical mistakes should be avoided. However, the SMT and NMT require a large number of good quality sentence pairs to deduce good feature representations. Unfortunately, such a corpus is not available for the English–Indian languages.

This inadequacy of bilingual corpora affected the machine translation research in Indian languages. Keeping this idea in mind, we decided to collect and clean parallel sentences from various sources for language pairs such as English–Malayalam, English–Hindi, English–Tamil, and English–Punjabi. The number of sentences collected and cleaned is listed in Table 1 and the vocabulary size in each language pair is shown in Table 2.

It is important to collect sentences from various domains while preparing a parallel corpus so that frequently used words in all those domains can be added to the vocabulary. This will further reduce the possibility of occurring out of vocabulary words when the system will be tested. On account of this reason, parallel corpora were prepared by collecting sentences from sources such as websites where bilingual texts are available (e.g. vikaspedia.in), story books, new websites, film subtitles, Bhagavat Gita Bible, Quran, and freely available encyclopedias. The coverage of these sentences spread across various fields like film, sports, politics, short stories, agriculture, religion, health, education, and language.

Sentences that were collected from the above-mentioned sources, especially online resources, contained a lot of impurities. The objective of cleaning was to remove foreign characters and incomplete sentences. The sentences that did not convey the meaning properly were omitted, and foreign characters were removed using regular expression. In addition to online resources, parallel sentences were also collected from bilingual books. These books were scanned, and sentences were prepared using the approach described in Ref. [27]. In both cases, long sentences (sentences that contain more than 50 words) were not removed from the corpus with a view to study the performance of neural networks in handling long-term dependencies in the sentence.

This corpus was used in Shared task and workshop on Machine Translation in Indian languages (MTIL-2017) along with the corpus provided by the Technology Development for Indian Languages Programme (TDIL), India [17].

Analysis showed that English–Malayalam and English–Hindi contains a number of lengthy sentences (number of words in the sentence are large), whereas English–Tamil and English–Punjabi do not have sentences with a word length of more than 100. In the English–Tamil and English–Punjabi corpus, the percentage of sentences with a word length <20 is more than 85%, whereas that in the English–Malayalam and English–Hindi corpora is around 75%.

4 Methodology

The fundamental idea behind an NMT system is to predict a sequence of words Y=(y1,…,yt) in the target language for a given source language sentence X=(x1,…,xs). This conditional probability distribution is modeled using the RNN-based encoder–decoder architecture. The encoder takes the variable length source language sentence and converts it into a fixed length vector. This fixed vector length vector (sentence embedding) contains the meaning of the input sentence. The decoder then takes this sentence embedding as the input and starts predicting the output words by taking the context of each word into consideration. Mathematically, this thought can be represented as,

where P(y|x) is the probability of obtaining a target language word y for a given source language word x, and c is the context of that particular word.

4.1 Encoder

Basically, an encoder transforms a sentence into a vector form, which represents the meaning of that sentence. Initially, word representations for both source and target language words are obtained. This word embedding is then fed into the encoder–decoder network. The encoder network transforms these word representations into a sentence embedding. This task is performed using a bi-directional recurrent neural network (Bi-RNN) [32] which contains two RNNs for computing rightward and leftward hidden state sequences. This allows us to capture both rightward and leftward contexts of each word.

(2)hi→=f(hi−1→,Wexi)

(3)hi←=f(hi+1→,Wexi)

where hi→ and hi← represents the rightward and leftward hidden state sequences, W_e is the word embedding matrix, x_i is the input (source) word, and f is a nonlinear function.

Now, each source language word, x_i can be represented using both rightward and leftward hidden state information, i.e. hi=(hi→,hi←). This helps us to obtain more information about each word by incorporating the details of the surrounding words. This information is then fed into the input layer of the decoder.

For example, while translating an English sentence “you always work”, it is first represented using a word-embedding mechanism and then fed into the encoder as input. The encoder is built with the LSTM/Bi-RNN networks and a zero vector as the starting state. The last hidden layer of the encoder network generates a vector, which carries the meaning of the input sentence.

The basic block diagrams of the neural network and NMT system (encoder–decoder architecture) are given in Figures 1 and 2, respectively.

Figure 1:

Block Diagram of Neural Network.

Figure 2:

Neural Machine Translation Encoder–Decoder Architecture.

5 Results and Discussion

We used the same NMT architecture for the English–Malayalam, English–Hindi, English–Tamil, and English–Punjabi language pairs. The parameters for the system architecture are shown in Table 3, and the configuration of the machine on which the NMT system was trained is shown in Table 4. The models were trained with a corpus described in this paper and bilingual sentences obtained from the TDIL database for training in the NMT system for all the four language pairs, which are listed in Table 5. A baseline system was implemented with the aforementioned architecture [20] using the data set of the size given in Table 5.

Before feeding into the network, sentences were tokenized using the tokenizer module given in the Moses tool kit [22]. The system was trained with different parameters such as the number of hidden layer units (200 and 500) and the neural network algorithm (LSTM and Bi-RNN) of which the set of parameters listed in Table 3 was found to be optimal. Details of the average speed taken by the machine while training is presented in the Table 6.

In order to capture the long-term dependencies in the sentences, we used attention mechanism explained in Ref. [24]. However, this mechanism failed when the source language sentences are too long. That is, contexts in the sentences were captured well when the number of sentences is less. From Table 7, it is understood that a number of sentences with more than 50 words are very high in English–Malayalam and English–Hindi corpus compared to English–Tamil and English–Punjabi. This is one of the reasons why the systems in Malayalam and Hindi show poor results even when Tamil and Punjabi show relatively good results. Another problem encountered while translating the test sentences was the occurrence of OOV words. Initially, the system was modeled in such a way that all OOV words in the test data set are replaced with a special character <UNK>. However, when the number of words in the source language sentence is large, the translated output will have many <UNK> characters in it. This problem was rectified using a transliteration module. When an OOV word is met during the decoding phase, instead of replacing it with <UNK>, a transliterated version of that particular word is used. This helps the translated sentence to be like a proper translation.

Evaluation of machine translation output is extremely important as it is the key factor that determines the quality of the translation, required level of post-editing, etc. Generally, MT outputs are evaluated both automatically and manually. One of the most common automatic evaluation metric is BLEU [26]. In automatic evaluation, metrics use gold-standard reference translations to which the translation outputs are compared. In human evaluation, annotators are asked to rate the translations on a 1–5 scale based on the source sentence and a gold-standard reference translation, where 1 is the lowest score, and 5 is the highest. Usually, three measures – adequacy, fluency, and rating – are used for human evaluation purposes [11]. Three annotators were assigned to evaluate the translation in each of the four languages. The final score was computed by taking the average of adequacy and fluency and converting into percentages, and the rating score was also converted into percentages. Human evaluation scores are shown in Table 8.

Automatic evaluation of machine translation results has equal importance as that of human evaluation. In this paper, we used the BLEU score to evaluate the translation. The BLEU score was computed for each of the eight architectures and are listed in Table 9. The LSTM and Bi-RNN algorithms were used to design the encoder network and trained the model with 200 and 500 hidden layer units in each of the hidden layers. It is evident from Table 9 that the Bi-RNN algorithm with 500 hidden layer units produces good translations compared to other models. Therefore, we can infer that, an increase in the hidden layer units will improve the translation. Similarly, the use of the Bi-RNN algorithm along with attention mechanism can capture the dependencies in the text well compared to the LSTM with attention mechanism.

6 Conclusion

Machine translation from English to Indian languages are always a difficult task due to the unavailability of a good quality corpus and morphological richness in the Indian languages. For an NMT system to produce better translations, the size of the corpus should be huge. In addition to that, the parallel sentences should convey similar meanings, and the sentences should cover different domains. Modeling the system with such a corpus can assure good translations while testing the model. This corpus is our contribution to the machine translation research community. Apart from the size and coverage of the corpus, the length of the sentences also plays a significant role in determining the quality of translation. The length should not be too short or too long because deep learning architectures cannot extract very long dependencies present in the sentences. As morphological richness in the English and Indian languages are in two extremes of the spectrum, adding linguistic features along with the sentences can improve the translation.