Sentiment Polarity Detection in Bengali Tweets Using Deep Convolutional Neural Networks

1 Introduction

Over the past decade, there has been an exponential growth in the use of micro-blogging services such as Twitter and access to mobile phones worldwide. In order to enhance customer satisfaction, online merchants enable their customers to express their opinions about the products or the services they have purchased. A vast amount of blog posts, tweets and comments from the internet users are also available in various domains such as politics, crickets, sports, movies and music. This huge amount of online reviews, tweets and other opinion-related information can be used for various applications such as deriving market intelligence and evaluation of public opinions and views.

“Opinion mining” and “sentiment analysis (SA)” are two terms used interchangeably. Sentiment is defined as a feeling or an opinion, especially one based on emotions [4]. It also refers to an attitude, subjective impression, thought or judgment, which are prompted by feelings. Opinions describe people’s sentiments, appraisals or feelings towards entities and events and their properties. SA is a challenging task because detecting feeling or opinion expressed in textual documents requires deeper natural language processing (NLP) techniques to be combined with data mining tasks. Although there are many issues related to different kinds of SA tasks such as subjectivity detection, opinion extraction and irony detection, the most widely studied task is perhaps sentiment polarity classification which aims to determine the overall sentiment orientation (positive, neutral and negative) of the opinions contained within a given document.

Nowadays, as more and more users express their political and religious views on Twitter, tweets become valuable sources of people’s opinions. Tweets data can be efficiently used to infer people’s opinions for marketing or social studies [12, 38]. So SA of tweets is also an urgent need. SA of short texts such as Twitter messages is challenging because they contain limited contextual information.

In recent years, many studies have been conducted in the particular area of sentiment polarity classification. Two approaches are commonly used for sentiment polarity classification: task-lexicon-based approach and machine-learning-based approach. In lexicon-based approach [24, 29], polarity information about the words in the input text is retrieved from a sentiment lexicon which consists of collection of words where each word is stored along with its polarity information. Using this lexicon information, for a review document or a tweet, the probability of being in a class – either positive class, neutral class or negative class – is computed. In the machine-learning-based approach, machine learning algorithms [5, 7, 16, 27, 30] are trained with hand-crafted features for classifying sentiments of text. The supervised machine learning approach is a relatively popular approach in the area of sentiment polarity classification. The supervised method of sentiment polarity classification uses machine learning algorithms by training a machine learning classifier on a sentiment polarity labeled corpus (where each review or tweet is manually labeled as positive/negative/neutral), and then it is applied to classify new unseen texts. Each textual document is turned into a feature vector. The commonly used features are words, n-grams, surrounding words, punctuation, etc. To apply machine leaning algorithms such as support vector machine (SVM), multinomial NB and decision trees to a sentiment polarity classification task, we need to represent each input text as a numeric vector. It is also mandatory to translate each document or tweet into a numeric vector of equal length. This requirement makes the vector size equal to the vocabulary size and hence the vectors become very sparse. This problem becomes worse when we represent short texts like tweets.

Most recently, neural networks have been a most popular topic in the field of NLP, including SA. When more computing power and a large amount of training data are available, the deep-learning-based approach to sentiment polarity classification [11, 17, 35] can give more accurate results than the traditional sentiment classification methods that use SVM, NB, etc. The strength of the deep learning model is that it does not require any human intervention for feature engineering for designing a set of hand-crafted features. Instead of using a set of hand-crafted features, a deep learning model takes an input-dense vector representation of words called word embedding [25], which contains contextual information, and the intermediate layers of the deep neural network learn the features during the training phase itself.

Most of the previous research works on SA involve the analysis of sentiments of text units written in English language [19, 34, 39]. But since India is a multilingual country, many Indians share their opinions or write on the twitter using their mother language or mixture of several languages. So, there is also a need for developing a system that can do SA of texts written in Indian languages. A recent commendable attempt to promote research on SA in Indian language tweets has been made through conducting a shared task on Sentiment Analysis in Indian Languages (SAIL) Tweets, co-located with MIKE 2015 conference held at IIIT Hyderabad, India [28].

Bengali is the most spoken language in Bangladesh, West Bengal, Assam and Tripura. Bengali is the second most spoken language in India. With about 230 million Bengali speakers spread all over the world, the Bangla language is also one of the most spoken languages in the world. To our knowledge, a few number of attempts have been made by the researchers for SA of Bengali tweets [20, 32]. Our research focus is in the area of the sentiment polarity classification of Bengali tweets, and we use deep neural networks for this task. We have also incorporated in our proposed sentiment polarity detection system some lexical knowledge extracted from Bengali SentiWordNet ^[1] (lexical resource for opinion mining) supplied for SAIL contest [28].

2 Related Work

Although most of the research papers on SA present approaches to SA of English texts, work on other languages is still growing. There are some previous research works that have dealt with SA of non-English texts such as German, Chinese, Arabic, Bengali and Hindi.

Zhang et al. [40] used a rule-based approach to Chinese SA which works in two phases: (1) determining each sentence’s sentiment based on word dependency, and (2) aggregating sentences in order to predict the document sentiment. Wan (2009) [36] made use of bilingual knowledge including both Chinese resources and English resources for sentiment polarity identification of Chinese product reviews. In this work, Wan (2009) [36] translated a corpus of Chinese documents into English by using Google Translate and Yahoo Babel Fish. In addition, he applied ensemble methods to combine the individual results over Chinese and English datasets. The results for the combination methods improved the performance of individual results.

Ghorbel and Jacot [13] used a corpus with movie reviews in French. They applied a supervised classification combined with SentiWordNet in order to determine the polarity of the reviews.

Rushdi-Saleh et al. [31] presented a corpus of movies reviews in Arabic annotated with polarity and performed several experiments using machine learning techniques. Al-Ayyoub et al. [1] presented Lexicon-based SA of Arabic tweets.

Banea et al. [3] proposed several approaches to cross-lingual subjectivity analysis by directly applying the translations of opinion corpus in English to training an opinion classifier in Romanian and Spanish. This work showed that automatic translation is a viable alternative for the construction of resources and tools for subjectivity analysis in a new target language. A comparative study of NLP techniques applied to SA and topic detection of Spanish tweets has been presented in [2]. Martínez-Cámara et al. [23] applied the supervised approach to SA of Spanish movie reviews using different machine learning algorithms such as SVM, naive Bayes, Bayesian based logistic regression, K-nearest neighbor (KNN) and C4.5.

Das and Bandyopadhyay [9] presented an approach to opinion polarity classification on news texts for Bengali using SVM. Their system identifies semantic orientation of an opinionated phrase as either positive or negative. They also developed a subjectivity classifier for performing sentence level subjectivity classification. Chowdhury and Chowdhury [6] used a semi-supervised bootstrapping approach for the development of the training corpus which avoids the need for labor-intensive manual annotation. For classification, they used SVM and maximum entropy with a combination of various sets of features. SA of Bengali tweets using multinomial NB with n-gram and SentiWordNet features has been presented in [32]. An approach presented in [20] uses a more sophisticated system that uses an unsupervised approach for expanding a (small) Indian sentiment lexicon, leveraging distributional thesauri, sentence level co-occurrence statistics and SVM.

Joshi et al. [15] presented a study on SA for Hindi. They studied three approaches to perform SA in Hindi: the first approach involves training a classifier on this annotated Hindi corpus and using it to classify a new Hindi document, the second approach translates the given document into English and uses a classifier trained on standard English movie reviews to classify the document and third approach uses a lexical resource called Hindi-SentiWordNet and implements a majority-score-based strategy to classify the given document. They proposed a fall-back strategy for SA in Hindi which applies any one of the above-stated three strategies based on the availability of in-language training data or translation system from Hindi to a resource-rich language or SentiWordNet. Sharma et al. [33] presented unsupervised lexicon-based SA of Hindi tweets. Mittal et al. [26] presented SA of Hindi review based on a negation and discourse relation.

A recent commendable attempt to promote research on SA in Indian language tweets has been made through conducting a shared task on SAIL Tweets, co-located with MIKE 2015 conference held at IIIT Hyderabad, India. In this task, the organizers provided tweets for three Indian languages, namely, Bengali, Hindi and Tamil, annotated with positive, negative and neural polarity. The main objective of this task was to classify the tweets into positive, negative and neutral categories [28]. Most teams participated in the contest used machine-learning-based approaches such as SVM [20], multinomial NB [32] and decision trees. Sarkar and Chakraborty [32] applied multinomial NB for SA for Indian languages, including Bengali and Hindi. Kumar et al. [20] used SVM for SA for Bengali and Hindi tweets.

Most of the previous approaches to SA have used supervised machine learning algorithms such as SVM, multinomial NB, decision tree and KNN for sentiment polarity detection. Recently neural networks have been the most popular topic in the field of NLP, including SA. Deep-learning-based approach to SA [11, 17, 35] has received more attention of the researchers conducting research on various SA tasks.

Kim [17] used a simple convolutional neural networks (CNN) with one layer of convolution on top of word vectors obtained from an unsupervised neural language model. These vectors were trained on 100 billion words of Google News. This work showed that a simple CNN with little hyper-parameter tuning and static vectors achieved excellent results on multiple benchmarks. Socher et al. [35] introduced Sentiment Treebank which includes fine-grained sentiment labels for 215,154 phrases in the parse trees of 11,855 sentences. They used a recursive neural tensor network (RNTN) for predicting fine-grained sentiment for all nodes in the parse trees of sentences. Sentiment label assigned at the root of the parse tree of a sentence is considered as the label of the sentence. They showed that the RNTN performs better than state-of-the art sentiment classification approaches when it is tested on a test set of positive and negative sentences. Dos Santos and Gatti [11] presented an approach that uses deep convolutional neural networks for the sentiment analysis of short texts. Wang et al. [37] combined a convolutional and recurrent neural network for the sentiment analysis of short texts.

3 CNN-Based Model for Sentiment Polarity Detection

The model architecture that we have used for sentiment polarity detection of Bengali tweets is shown in Figure 1. This architecture is a slight variation of the architecture presented in [8, 17]. As we can see from Figure 1, a tweet is represented as an n×k matrix where n is the number of words in the tweet and k is the dimension of the word vector. According to this architecture, the input is the concatenated vector of the word vectors corresponding to the words in a tweet (padding zeros is necessary for equalizing the length). The input is submitted to the convolutional layer that performs convolutional operation involving a filter W of size h×k where h is the size of the window of words and k is the dimension of the word vector. A new feature f_i, which is produced after applying a convolutional operation using a filter W applied to the window of h words, can be expressed as follows:

Figure 1:

Model Architecture for an Example Tweet.

fi=f(WX(i:i+h −1)+b)

where X_(i:i+h−1) is the concatenation of vectors of words within the window spanning from the ith word to the (i+h−1)th word in a sentence, b is the bias weight and f is the non-linear function which may be a hyperbolic tangent or rectilinear function.

The filter is applied to each possible window of words obtained by sliding right with stride equal to 1. Thus a feature map f is obtained as follows:

F=[f1f2, …, f(n−h+1)],

where n is the length of the tweet and h is the window size.

After obtaining the feature maps (number of feature maps depends on the number of filters), we apply max pooling [22] which selects the most important feature from each feature map by finding the maximum value from each feature map f. The pooling can also deal with issues related to the variable sentence length.

If we use m filters (each with varying window size), for each filter, we can get one convoluted feature. Thus, m filters give rise to a feature vector of length m.

After the convolution layer, we add a fully connected layer which accepts the feature vector of length m. The output of this layer is passed to another fully connected layer which is basically the softmax layer whose output is the probability distribution over labels.

4 Our Proposed Sentiment Polarity Detection Method

The various components of our proposed sentiment polarity detection system are shown in Figure 2. Our proposed system has four major components: preprocessing, data representation, training a CNN model and saving the model for future prediction.

Figure 2:

Block Diagram of Our Proposed System.

Since we deal with the tweet data which are usually noisy data, we need to apply preprocessing on the training data as well as test data. We preprocessed the entire data collection to remove irrelevant characters from the data. For each tweet in our corpus, we remove irrelevant characters such as [, -, _, =, :, +, $, @, ~, !, ,, ;, /, ^, ), (, ], {, }, <, >. Words which have occurred only once in the corpus are also removed from the tweets as irrelevant features. In the preprocessing step, we also incorporate some lexical knowledge to enhance the sentiment information contained in the tweets. We use SentiWordNet [10] for finding the sentiment polarity of the individual words contained in the tweets. SentiWordNet that we used here is basically containing three sets of words: a set of positive words, a set of negative words and a set of neutral words. Each tweet word is searched into SentiWordNet. If the tweet word is found in SentiWordNet, the sentiment tag of the word is retrieved from SentiWordNet and the tweet word is augmented with the retrieved sentiment tag. If the word is not found, the tweet word will be left as it is. For example, the dataset consists of the following two tweets:

Ram is a good boy.

She hates Cockroach.

After tagging with sentiment tags, we have:

Ram is a good (positive) boy.

She hates (negative) cockroach.

Here, “positive” and “negative” are sentiment tags of the words “good” and “hates,” respectively, which are retrieved from SentiWordNet.

For data presentation, we need to create word vectors for all tokens in the vocabulary. We have used our tweet corpus for creating the word vectors. There are 8483 tokens in our vocabulary. A tweet is represented as the vector of size M×k where M is the maximum size of the tweet in a corpus and k is the dimension of the word vector. The tweet whose size is smaller than the maximum size is padded with zeros. Since sentiment tags obtained from SentiWordNet are assigned to the tweet words at the preprocessing step, they are also considered as the tokens in the vocabulary.

After completion of preprocessing and data representation steps, we train our proposed CNN-based model with our training dataset. After obtaining the trained model, we apply it to our test data for sentiment polarity classification for detecting polarity of sentiment expressed in each test Bengali tweet. Before submission to the trained CNN model, test tweets are also processed using the method similar to that applied to the training data.

5 Evaluation and Results

We have used Bengali datasets released for a shared task on SAIL Tweets, co-located with MIKE 2015 conference held at IIIT Hyderabad, India [28]. The description of the Bengali sentiment dataset is shown in Table 1.

The training data released for SAIL 2015 contest were distributed in three different files containing negative, neutral and positive tweets. We have conducted the following two experiments:

• Experiment 1: In this experiment, we train, tune and test our proposed CNN-based model presented earlier in this paper.

• Experiment 2: This experiment is conducted to compare our proposed CNN-based sentiment polarity detection model with the traditional deep belief networks (DBN)-based model presented in [14]. Here we have used deep belief neural networks which are basically deep multilayer perceptron neural networks having three hidden layers. It does not include any convolutional layer. To train DBN, we have used a two-step training process similar to the approach proposed in [14]:

  • Layer-wise pretraining in unsupervised mode

  • Initializing the whole DBN with the weights learnt in unsupervised mode and then fine-tuning weights in the DBN in supervised mode using labeled data.

The training phase of the DBN-based SA system has several steps: (1) preprocessing of data, (2) data representation, (3) layer-wise pretraining of the DBN, (4) assembling all trained layers to create a deep multi-layer perceptron and (5) fine-tuning the DBN in supervised mode using labeled data.

Data preprocessing part is similar to the CNN-based model presented in this paper, but for data representation, both the training data and the test data are represented in the form of a numeric feature vectors using the bag-of-words model which represents each tweet as a vector of size V where V is the size of the vocabulary and the ith component of the vector corresponds to the weight of the ith vocabulary word when the word is present in the tweet. If the word is not present in the tweet, 0 is taken. Weight of a word is calculated using the TF*IDF method.

We have used the DBN with three hidden layers, one input layer and a softmax output layer. The size of the input layer is 6225 which is equal to the vocabulary size for our training data. The sizes of three hidden layer sizes are set to 300, 50 and 50, respectively. In the pretraining phase, the greedy algorithm was used to train each layer of weights separately, starting at the bottom. Each layer was pretrained for 30 epochs using the training set with the learning rate set to 0.05. The final network is trained up to 1000 epochs with the learning rate set to 0.1 and batch size set to 10.

6 Conclusions

In this paper, we have presented an approach to sentiment detection of Bengali tweets using deep convolutional neural networks. We have compared the performance of our proposed CNN-based Bengali sentiment polarity detection model with the DBN-based model. One of the critical problems that we faced for our experimentations is scarcity of benchmark datasets for SA in Indian languages. Although we have used the datasets released for SAIL contest 2015 [28], it is insufficient and noisy. In future, we have planned to rectify errors in the training data and create my own data for developing a more accurate sentiment classification model for Bengali tweets.

Another future plan is exploitation of unlabeled data in the training process of DBN- as well as CNN-based models. For improving system performance, some other kinds of deep neural networks such as deep recursive neural networks [35] can also be used.

The model proposed in this paper can be easily extended to other Indian languages such as Hindi and Tamil if training data for the corresponding language are available.