Introduction

Text classification (TC) is a supervised learning task that assigns natural language text documents to one (the typical case) or more predefined categories [1]. Classification algorithms typically use a supervised machine learning (ML) algorithm or a combination of several ML algorithms [2].

TC is an important component in many research domains including information extraction, information retrieval, text indexing, text mining, and word sense disambiguation [3]. There are two main types of TC: topic-based classification and stylistic classification. An example of a topic-based classification application is classifying news articles as Business-Finance, Lifestyle-Leisure, Science-Technology, and Sports [4]. An example of a stylistic classification application is classification based on different literary genres, e.g., action, comedy, crime, fantasy, historical, political, saga, and science fiction [5]. Whereas stylistic classification is typically performed using linguistic features such as quantitative features, orthographic features, part of speech (POS) tags, function words, and vocabulary richness features, topic-based classification is typically performed using unigrams and/or n-grams (for n > 2) [6–8].

The traditional model for topic-based TC is based on the bag-of-words (BOW) representation, which associates a text with a vector indicating the number of occurrences of each chosen word in the training corpus [2]. In a topic-based classification, ML methods such as Maximum Entropy (ME, Jaynes [9]), support vector machines (SVMs, Cortes and Vapnik [10]), Naive Bayes (NB, Heckerman [11]), and C4.5 decision tree induction [12,13] have been reported to use a BOW representation of hundreds or thousands of unigram words to achieve accuracies of 90% and greater for particular categories [1].

This study addresses topic-based classification based on the BOW representation. We hypothesize that the application of a selection of preprocessing methods can improve the accuracy results of different TC tasks. We investigate the impact of all possible combinations of five/six basic preprocessing methods on TC. We validate the proposed model on four benchmark corpora to determine what preprocessing combination(s) is (are) best suited to classification tasks.

The key contributions and novelties of this paper are (1) performing an extensive and systematic set of TC experiments using all possible combinations of five/six basic preprocessing methods on four benchmark text corpora (and not samples of them) using three ML methods and training and test sets; (2) presenting a general important conclusion about the importance of performing an extensive variety of preprocessing methods combined with TC experiments because it contributes to improve TC accuracy; and (3) presenting the specific findings that demonstrate what preprocessing combinations for what benchmark datasets significantly improve the classification results when using a BOW representation.

This paper is structured as follows. The Material and Methods section outlines the datasets, applied platform, applied supervised machine learning methods, preprocessing methods, and the chosen metric. In the Preprocessing for Text Classification section, we provide a detailed background for various common text preprocessing methods related to text classification. The Examined Text Corpora section introduces the four examined benchmark datasets. In the Model section, we introduce our classification Model. The Experimental Results section presents a wide variety of experimental results and their analysis. The Summary, Conclusions, and Future Work section summarizes the results and the main conclusion of this study and suggests a few future research proposals. Finally, the S1 Appendix presents examples of two full files for each of the four discussed datasets.

Materials and methods

The four examined corpora in this study are benchmark datasets: WebKB, R8, SMS Spam Collection, and Sentiment Labelled Sentences. Details about them are found in the Examined Text Corpora section. In this study, we applied three supervised ML methods: BN (Bayes Networks), SMO (a variant of SVM), and Random Forest (RF) using a bag of word unigrams. We used the WEKA platform with their default parameters. For each classification task, we used the experimenter mode in WEKA Version 3.9.1 with the following settings: Train (67%) / Test (33%) (data randomized) and the number of repetitions of each experiment was 10. We applied/implemented six basic preprocessing methods: C–spelling correction, L–converting uppercase letters into lowercase letters, H–HTML tag removal (relevant only for the WebKB dataset), P–punctuation mark removal, S–stopwords removal, and R–reduction of repeated characters. The chosen metric to evaluate the experimental results is the accuracy measure.

Preprocessing for TC

Applying preprocessing on a certain dataset can improve its quality in general and for TC in particular. The preprocessing process can "clean" the dataset from "noise" (e.g., correction of spelling errors, reduction of replicated characters, and disambiguation of ambiguous acronyms). Furthermore, in some cases, application of preprocessing methods such as stopword removal, punctuation mark removal, word stemming, and word lemmatization can improve the dataset's quality for TC tasks.

There is a widespread variety of text preprocessing methods. Examples of basic types are conversion of uppercase letters into lowercase letters, correction of common misspelled words, HTML tag removal, punctuation mark removal, reduction of different sets of emoticon labels to a reduced set of wildcard characters, reduction of replicated characters, replacement of HTTP links to wildcard characters, stopword removal, and word stemming. Examples of more sophisticated preprocessing methods are word lemmatization, translation of common slang words into phrases that express the same ideas without using slang, and expansion of abbreviations.

A relatively small number of simple preprocessing methods (e.g., conversion of all the uppercase letters into lowercase letters and stopword removal) are performed by many TC systems. Nevertheless, most preprocessing methods are not performed by most TC systems. Furthermore, not all of the preprocessing methods are considered effective by all TC researchers. Some of them might even harm the classification results. For instance, Forman [14], in his study on feature selection metrics for TC, claimed that stopwords are ambiguous and occur sufficiently frequently as to not be discriminating for any particular class. However, HaCohen-Kerner et al. [15] demonstrated that the use of word unigrams including stopwords in the domain of Jewish law documents written in Hebrew–Aramaic lead to improved classification results compared to the results obtained using word unigrams excluding stopwords.

One of the explanations for the phenomenon that stopword removal harm the TC results is that the TF/TF-IDF values of the stopwords distribute differently in different categories improve style-based TC and in some cases even content-based TC. In addition, the contribution of use/non-use of stopwords depend on the corpus (even within the same language). For some corpora, various preprocessing methods might be irrelevant. For instance, HTML tag removal is not necessary on many SMS/Twitter corpora because of the relatively low prevalence of HTML tags.

Several studies analyzed the influence of various preprocessing methods on TC. A brief summary of many of these studies follows. Song et al. [16] examined 32 combinations of five preprocessing methods: stopword removal, word stemming, indexing with term frequency (TF), weighting with inverse document frequency (IDF), and normalization of each document feature vector to unit length. These combinations were applied to two benchmark datasets: Reuters-21578 and 20 Newsgroups using a linear SVM and different lengths of a BOW representation. Their experimental results demonstrated that normalization to unit length can always significantly improve the effectiveness of text classifiers. Conversely, the impact of the other factors, e.g., stopword removal, word stemming, indexing, and weighting are rather minimal.

Lemmatization, stemming, and stopword removal were examined by [17] using only the multinomial NB classifier on two datasets: 8000 documents in English selected from Reuters Corpus Volume 1 dataset divided into six categories and 8000 Czech documents provided by Czech News Agency divided into five categories. They concluded that the best preprocessing approach for TC is to only apply stopword removal. Their experiments indicated that stopword removal improved the classification accuracy in most of the cases, although the results were not statistically significant. Further, lemmatization and stemming were more negative than positive for both languages.

The use of stopword removal, stemming, and different tokenization schemes on spam email filtering for two email corpora were analyzed in Méndez et al. [18]. They used three ML methods: NB, boosting trees, and SVM. Their main conclusion was that the performance of SVM is surprisingly effective when stemming and stopword removal are not used. One of the reasons is that some stopwords are rare in spam messages and should not be removed to improve the performance of spam filtering. Pomikálek and Rehurek [19] explored stopword removal, tokenization, and stemming by applying eight ML methods: SVM, NB, k-nearest neighbor (KNN, Fix and Hodges [20]‏), C4.5, neural networks, simple linear regression, voted perceptron, and RepTree [21] on limited versions of three datasets: Reuters 21578 (1800 documents), Newsgroups (6000 documents), and Springer (2700 documents). The authors concluded that stemming and stopword removal have minimal impact on the classification results.

Stopword removal, stemming, WordNet, and pruning were used for classification of MEDLINE documents in [22] using six ML methods: Sequential Minimal Optimization (SMO, Platt [23], Keerthi et al. [24]), Random Forest (RF, [25]), Bayes Networks (BayesNet, BN) [26], KNN, J48 (an improved variant of the C4.5 decision tree induction implemented in WEKA), and decision table [27]. Lowercase conversion, stopword removal, and tokenization were applied to all experiments without any comparison. Their experiments demonstrated that the application of pruning, stemming, and WordNet significantly reduces the number of attributes and improves the classification accuracy.

Srividhya and Anitha [28] evaluated four preprocessing methods: stopword removal, stemming, TF-IDF weighting [29], and document frequency on the Reuters 21578 dataset. Their main conclusions were as follows: (1) removal of stopwords can expand words and enhance the discrimination degree between documents and improve the classification performance, and (2) TF-IDF is required to create the index file from the resulting terms.

Clark and Araki [30] presented their system, called Casual English Conversion System (CECS). CECS deals with correction of errors and irregular words (which occur in social data) divided into eight categories as follows: (1) Abbreviation (shortform), (2) Abbreviation (acronym), (3) Typing error/ misspelling, (4) Punctuation omission/error, (5) Non-dictionary slang, (6) Wordplay, (7) Censor avoidance, and (8) Emoticons. CECS uses a manually compiled and verified database, which contained 1,043 entries. Each entry contain four columns: “error word” (the casual English item), “regular word” (the corresponding dictionary English item), “category” (the item’s category), and “notes”. The authors evaluated the performance of two open source English spell checkers on Twitter messages and measured the extent to which the accuracy of the spell checkers is improved by firstly preprocessing the texts with their system. The results showed that average errors per sentence decreased substantially, from roughly 15% to less than 5%.

Haddi et al. [31] investigated the role of text pre-processing in sentiment analysis of two online data sets of movie reviews (Dat-1400 and Dat-2000). They used a combination of different pre-processing methods (HTML tags removal, non-alphabetic signs removal, white space removal, abbreviation expansion, stemming, stop words removal, and negation handling) to reduce the noise in the text in addition to using chi-squared method to remove irrelevant features. They reported that sentiment analysis can be significantly improved using SVM, thousands of word unigrams, and appropriate pre-processing methods and feature selection. Their accuracy results are comparable to the accuracy results that can be achieved in topic classification, a much easier problem. On the Dat-2000 dataset they obtained an accuracy of 93.5% using 9,058 word unigrams.

Uysal and Gunal [32] studied the impact of preprocessing on TC using four preprocessing methods: tokenization, stopword removal, lowercase conversion, and stemming. Their examination was performed using all possible combinations of the preprocessing methods on four datasets: e-mails in Turkish, e-mails in English, news in Turkish, and news in English. They applied only the SVM ML method using feature sizes of 10-, 20-, 50-, 100-, 200-, 500-, 1,000-, and 2,000-word unigrams. Their main conclusion was that appropriate combinations of preprocessing tasks, depending on the domain and language, can provide a significant improvement in classification accuracy whereas inappropriate combinations can also degrade the accuracy. According to their experiments, lowercase conversion improves classification success in terms of accuracy and dimension reduction regardless of the domain and language. However, there is no unique combination of preprocessing tasks that provides successful classification results for every domain and language studied. Another finding is the importance of stopwords in contrast to many TC studies, which assume that stopwords are irrelevant.

Ayedh et al. [33] investigated the effect of three preprocessing methods (stopword removal, word stemming, and normalization of certain Arabic letters that have different forms in the same word to one form) on TC for an in-house corpus containing 32,620 news documents divided into ten categories downloaded from different Arabic news websites. In this study, three ML methods were applied: NB, kNN, and SVM. Experimental analysis revealed that preprocessing has a significant impact on the classification accuracy, especially with the complicated morphological structure of the Arabic language. Choosing appropriate combinations of preprocessing tasks provides significant improvement in the accuracy of TC depending on the feature size and the ML methods. The best result (a 96.74% micro-F1 value) was achieved by the SVM method using the combination of normalization and stemming.

Krouska et al. [34] performed various classification experiments using four ML methods (NB, SVM, KNN, and C4.5) with four preprocessing methods (TF-IDF weighting scheme, Stemming, Stop-words removal, and Tokenization). They did not applied all possible combinations of the preprocessing methods. The experiments were applied on three different datasets of tweets, one with no specific domain (the Stanford Twitter Sentiment Gold Standard (STS-Gold) dataset) and two datasets with specific topics (the Obama-McCain Debate (OMD) dataset and the Health Care Reform (HCR) dataset). The TF-IDF weighting scheme, stemming, and stop-words removal were applied as fixed options, while the experiments were with the tokenization and feature selection. The best accuracy results (92.67%, 92.59%, 91.94%) for the three datasets STS-Gold, OMD and HCR, respectively, have been obtained by NB using 720, 1074, and 1280 word 1-to-3-grams with InfoGain>0.

Jianqiang and Xiaolin [35] discussed the effects of text pre-processing methods on sentiment classification performance in two types of classification tasks, and summed up the classification performances of six pre-processing methods (replacing negative mentions (e.g., "won't'' into "will not"), removing URL links, reverting words that contain repeated letters to their original English form, removing numbers, removing stop words, expanding acronyms to their long forms) using two feature models (the word n-grams features model and the prior polarity score feature model) and four classifiers (SVM, NB, LR, RF) on five Twitter datasets (the Stanford Twitter Sentiment Test (STS-Test) dataset, SemEval2014-Task9 dataset, The Stanford Twitter Sentiment Gold (STS-Gold) dataset, The Sentiment Strength Twitter Dataset (SS-Twitter), The Sentiment Evaluation Dataset (SE-Twitter)). Their experimental results showed that the best preprocessing method was the replacement of negative mentions in the N-grams model. This method leads to a significant increase in the accuracy and F1-measure of almost all classifiers on all five datasets (the maximum improvement of accuracy is 8.23% using SVM and the maximum improvement of the F1-measure is 10.21% using RF on the SemEval2014 dataset). Expansion of acronyms showed significant improvement only for NB in the N-grams model (the accuracy and F1-measure obtained improvements of 6.85% and 6.08%, respectively). The other preprocessing methods (removal of URLs, the removal of stop words, reverting words, and the removal of numbers) minimally improve the classification accuracy. The NB and RF classifiers were found as more sensitive than LR and SVM classifiers when applying the pre-processing methods.

HaCohen-Kerner et al. [36] in their poster paper explored the impact of preprocessing methods on TC of three benchmark mental disorders datasets. They checked all possible combinations of the following six basic preprocessing methods (spelling correction, HTML tag removal, converting uppercase letters into lowercase letters, punctuation mark removal, reduction of repeated characters, and stopword removal) that were applied also in this research (Section 4). In the first dataset, the best result showed a significant improvement of about 28% over the baseline result using all the six preprocessing methods. In the two other datasets, several combinations of preprocessing methods showed only minimal improvement rates over the baseline results.To summarize the studies that have been presented above, both literally and as a summary in Table 1, there is no uniformity in the examined datasets, languages, preprocessing methods, ML methods, results, or conclusions. Most of the studies use a relatively small number of the following components: datasets, ML methods, preprocessing methods, and combinations of these. Moreover, portions of the conclusions of these studies seemingly contradict each other (e.g., stopword removal improves classification accuracy [15,17,21] or does not improve classification accuracy [16,31,37]).

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Table 1. Attributes of studies addressing preprocessing for text classification.

https://doi.org/10.1371/journal.pone.0232525.t001

Table 1 summarizes the attributes of twelve studies described in this section that addressed preprocessing for TC. The presented attributes are the dataset(s), ML methods, preprocessing methods, and best results and conclusions for each study.

The conclusions are not contradictory because they were derived from experiments on different datasets, possibly different languages, with different stopword lists, and different sizes of BOW representation.

Thus, we decided in this research to explore the influence of all possible combinations (in contrast to many of the previous studies) of five/six (more than the majority of the previous studies) basic preprocessing methods on TC for four benchmark corpora (and not samples of them) to determine what preprocessing combination(s) is (are) best suited to TC, if any. Our experiments are applied using training and test sets.

Examined text corpora

The four examined corpora in this study are benchmark datasets: WebKB, R8, SMS Spam Collection, and Sentiment Labelled Sentences. Table 2 introduces general information about these four datasets. More details are provided after Table 2. Examples of two full files from each of these four datasets are given in the S1 Appendix.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Table 2. General information about the examined datasets.

https://doi.org/10.1371/journal.pone.0232525.t002

The WebKB dataset (The 4 Universities Data Set) [32,38,39]) contains 4,199 documents. Each document contains around 10,358 characters that are divided into around 2,918 words. The documents are webpages collected by the World-Wide Knowledge Base project of the CMU text-learning group. These pages were manually classified into seven different classes: student, faculty, staff, department, course, project, and other. We worked with the original WebKB dataset and not with its preprocessed version where all the terms were converted to stems.

The R8 dataset is a single-labeled dataset derived from the multi-labeled Reuters-21578 dataset; its documents were selected from the Reuters newswire in 1987 and manually classified by personnel from Reuters Ltd. The documents contained in R8 are selected from 8 of the 10 most frequent classes of Reuters-21578. These R8 classes contain 7,674 documents with a single topic and all classes have at least one training and one test document. Each document contains around 587 characters that are divided into around 102 words.

The SMS Spam Collection v.1 [40] dataset (http://www.dt.fee.unicamp.br/~tiago//smsspamcollection/) is a public set of SMS labeled messages that have been collected for mobile phone spam research. This dataset contains 5,574 English, real, and non-encoded messages, tagged as being legitimate (ham) or spam. Each SMS message contains around 82 characters that are divided into around 19 words. The dataset is composed of four subsets: (1) 425 SMS spam messages manually extracted from the Grumbletext website; (2) 3,375 SMS randomly chosen ham messages of the NUS SMS Corpus (NSC), which is a dataset of approximately 10,000 legitimate messages collected for research at the Department of Computer Science at the National University of Singapore; (3) 450 SMS ham messages collected from Caroline Tagg's PhD Thesis [41]; and (4) the SMS Spam Corpus v.0.1 Big, which contains 1,002 SMS ham messages and 322 spam messages.

The Sentiment Labelled Sentences dataset (https://archive.ics.uci.edu/ml/datasets/Sentiment+Labelled+Sentences#) is provided by the University of California, Irvine. This dataset contains 3000 reviews/sentences composed of an equal number of reviews (1,000) from three websites (imdb.com, amazon.com, and yelp.com). Each document contains around 66 characters that are divided into around 14 words. These are reviews of products, movies, and restaurants. For each website, there exist 500 positive and 500 negative sentences. Each review in the dataset is tagged as positive or negative. This dataset was created for the study presented in [42].

Model

Our approach is to compare the accuracy results obtained using the original files without any preprocessing to the results achieved using combinations of five/six preprocessing methods: C–spelling correction, H–HTML tag removal (for relevant datasets), L–converting uppercase letters into lowercase letters, P–punctuation mark removal, R–reduction of repeated characters, and S–stopword removal. These six preprocessing methods are relatively basic and/or common. Due to this, many TC systems applied at least part of these methods. Therefore, we decided to apply them. Furthermore, we consider all possible combinations of these five/six basic preprocessing methods, i.e., 31 (2⁵–1) / 63 (2⁶–1) nonempty combinations. The spelling correction is achieved using the Python 3 Spelling Corrector (https://github.com/phatpiglet/autocorrect/). The application of the S preprocessing method deletes all instances of 423 stopwords for English text (421 stopwords from Fox [43] plus the letters “x” and “z” that are not found in Fox [41], yet are included in many other stopword lists).

Various studies explored the selection of appropriate ML methods. Kotsiantis et al. [44] presented a review of classification techniques. Their main conclusion was that the key question when dealing with supervised ML classification is not whether a certain ML method is superior to others, but under which conditions a certain ML method can significantly outperform others for a given task.

Fernández-Delgado et al. [45] evaluated 179 ML methods arising from 17 ML method families on 121 data sets. The best results were achieved by (1) Parallel RF (implemented in R with caret), (2) RF in R tuned with caret, and (3) The LibSVM implementation of SVM in C with Gaussian kernel. Six RFs and five SVMs are included among the 20 best classifiers.

Krouska et al. [46] introduced a comparative analysis of five well-known ML methods: NB, SVM, KNN, Logistic Regression (LR, [47]), and C4.5 and a lexicon-based approach called SentiStrength (Available: http://sentistrength.wlv.ac.uk, last accessed on 2019 December 11.). These five ML methods were chosen as the most representative of ML and lexicon-based methods and were tested using three datasets and two test models (percentage split and cross validation). The best results were obtained by NB and SVM regardless of datasets and test methods.

In this research, we applied three supervised ML methods: BN, SMO, and RF. SMO (a variant of SVM) and RF were among the best ML methods found by the studies mentioned above. BN was found by us as the best ML method out of four ML methods including RF for a topic-based classification task [14]. We used the WEKA platform with their default parameters [20,48]. For each TC task, we used the experimenter mode in WEKA Version 3.9.1 with the following settings: Train (67%) / Test (33%) (data randomized) and the number of repetitions of the experiment set to 10.

A brief description of these three selected ML methods follows: BN is a variant of a probabilistic statistical classification model that represents a set of random variables and their conditional dependencies via a directed acyclic graph [25]. SMO [22,23] is a variant of the SVM ML method [9]. The SMO method is an iterative method created to solve the optimization problem frequently found in SVM methods. SMO divides this problem into a series of the smallest possible sub-problems, which are then resolved analytically. RF is an ensemble learning method for classification and regression [24]. Ensemble methods use multiple learning algorithms to obtain improved predictive performance compared to what can be obtained from any of the constituent learning algorithms. RF operates by constructing a multitude of decision trees at training time and outputting classification for the case at hand. RF combines Breiman's “bagging” (Bootstrap aggregating) idea in [49] and random selection of features introduced by Ho [50] to construct a forest of decision trees.

As mentioned in the Introduction section, the traditional model for topic-based TC is based on the bag-of-words (BOW) representation and many ML methods have been reported to use a BOW representation of hundreds or thousands of unigram words to achieve accuracies of 90% and greater for various TC tasks. Therefore, we decided to apply the traditional and successful BOW model of 1,000 to 5,000 (in steps of 1,000) unigram words in order to check our hypothesis that the application of different combinations of preprocessing methods can improve TC results. In Fig 1, we present a flowchart of our classification model.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 1. Flowchart of the classification model.

https://doi.org/10.1371/journal.pone.0232525.g001

Experimental results

To determine a reasonable number of word unigrams for the BOW presentation that we intend to apply, we performed TC experiments for each pair of dataset and ML method, using five different sets, containing the TF values of 1,000-, 2,000-, 3,000-, 4,000-, and 5,000-word unigrams. Tables 3–6 present the TC accuracy results using the TF values of 1,000-, 2,000-, 3,000-, 4,000-, and 5,000-word unigrams for the four examined benchmark datasets: WebKB, R8, SMS Spam Collection, and Sentiment Labelled Sentences. These tables refer to the experimental results that were performed without any normalization/preprocessing methods. These tables contain different annotations and emphases. The annotation v or * indicates that a specific result in a certain column is statistically better (v) or worse (*) than the baseline result (the result using 1,000-word unigrams). To compare the different results, we performed statistical tests using a corrected paired two-sided t-test with a confidence level of 95%. A number in italics represents the best accuracy result for each ML method (one per column) and a number in bold represents the best accuracy result in the table (i.e., the best accuracy result for the discussed dataset).

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Table 3. TC results for different numbers of word unigrams on the WebKB dataset.

https://doi.org/10.1371/journal.pone.0232525.t003

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Table 4. TC results for different numbers of word unigrams on the R8 dataset.

https://doi.org/10.1371/journal.pone.0232525.t004

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Table 5. TC results for different numbers of word unigrams on the SMS spam collection dataset.

https://doi.org/10.1371/journal.pone.0232525.t005

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Table 6. TC results for different numbers of word unigrams on the sentiment labelled sentences dataset.

https://doi.org/10.1371/journal.pone.0232525.t006

The main findings presented in Tables 3–6 are as follows: (1) the best accuracy results were achieved by the SMO method (1st place in three datasets and 2nd place in one dataset), and the second best accuracy results were obtained by the RF method (1st place in one dataset and 2nd place in three datasets); (2) one best accuracy result was obtained using 1,000-word unigrams for the WebKB dataset; two best accuracy results were obtained using 2,000-word unigrams for the R8 and SMS Spam Collection datasets; and one best accuracy result was obtained using 3,000-word unigrams for the Sentiment Labelled Sentences dataset (higher in 0.01 than the result for 2,000-word unigrams); and (3) for all datasets, neither the 4,000- nor 5,000-word unigrams achieved the best accuracy result.

In two datasets (R8 and SMS Spam Collection), the best accuracy result was achieved using 2,000-word unigrams; it was only marginally and not significantly better than the compatible accuracy results using 1,000-word unigrams. The use of 1,000-word unigrams instead of 2,000 saved a significant amount of time in our extensive set of experiments (64 preprocessing combinations for each pair of dataset and ML method). Thus, we decided to perform the TC experiments using the top 1,000 frequent word unigrams for each dataset.

Tables 7–10 present the TC accuracy results for the four examined datasets: WebKB, R8, SMS Spam Collection, and Sentiment Labelled Sentences using the top 1,000 frequent word unigrams for each dataset separately. As mentioned previously, we compared the results obtained using the original files without any preprocessing to the results achieved using all possible combinations of the following five/six basic preprocessing methods: C–spelling correction, L–lowercase, H–html tags (relevant only for the WebKB dataset), P–punctuation, S–stopwords, and R–repeated characters. We considered all 31 (2⁵–1) / 63 (2⁶–1) nonempty combinations of the five/six preprocessing methods in addition to the empty combination without the use of any preprocessing method.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Table 7. TC results for all possible combinations on the WebKB dataset.

https://doi.org/10.1371/journal.pone.0232525.t007

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Table 8. TC results for all possible combinations on the R8 dataset.

https://doi.org/10.1371/journal.pone.0232525.t008

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Table 9. TC results for all possible combinations on the SMS spam collection dataset.

https://doi.org/10.1371/journal.pone.0232525.t009

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Table 10. TC results for all possible combinations on the sentiment labelled sentences dataset.

https://doi.org/10.1371/journal.pone.0232525.t010

Also in these tables, there are different annotations and emphases. The annotation v or * indicates that a specific result in a certain column is statistically better (v) or worse (*) than the baseline result (the result using 1,000-word unigrams) at the significance level of 0.05. A number in italics represents the best accuracy result for each ML method (one per column) and a number in bold represents the best accuracy result in the table (i.e., the best accuracy result for the discussed dataset).

Table 7 (TC for the WebKB dataset) indicates that for each one of the ML methods, there were several significant improvements. Regarding the impact of each single preprocessing method on the WebKB dataset, the S preprocessing (i.e., Stopword removal) was the only single preprocessing method that enabled significant improvements compared to the baseline result according to the accuracy measure. This was done using two ML methods: SMO (95.61% vs. 94.1%) and RF (95.04% vs. 93.99%) while the BN ML method showed an insignificant improvement. The C and H (spelling Correction, Html object removal) preprocessing methods enabled insignificant improvement for all three ML methods. The R (Reduction of repeated characters) preprocessing method enabled an insignificant improvement for two ML methods (BN and RF) and an insignificant decline for the SMO ML method. The L (i.e., converting uppercase letters into Lowercase letters) preprocessing methods showed insignificant declines for all three ML methods while the P (i.e., Punctuation mark removal) preprocessing method showed significant declines for all three ML methods.

Better results that are significant improvements compared to the baseline result have been obtained by various combinations. The best result 95.74% was obtained by the SMO ML method using the CHS combination (spelling correction, html object removal, and stopword removal), a combination of the only three preprocessing methods that showed improvements for all three ML methods. The S preprocessing method was a part of the most combinations that achieved significant improvements. This finding is not new and was also observed by [15,17,21]. It also fits the claim of Forman [13] that stopwords are ambiguous and occur sufficiently frequently as to not be discriminating for any particular class for TC. Secondary findings are the significant contributions of the H and C preprocessing methods to the S preprocessing method that enabled achieving the best results using SMO and RF.

Table 8 (TC for the R8 dataset) demonstrates similar findings to those of Table 7. Regarding the impact of each single preprocessing method on the R8 dataset, again, for each one of the ML methods, there are several significant improvements and again the S preprocessing was the only single preprocessing method that enabled a significant improvement compared to the baseline result; this time, for all three ML methods: SMO (95.75% vs. 94.98%), RF (94.46% vs. 91.73%), and BN (90.40% vs. 81.99%). The L and P preprocessing methods showed no change for all three ML methods while the C and R preprocessing methods showed insignificant declines for two ML methods and only one insignificant improvement for one ML method.

There were marginal significant improvements such as 95.75% using many combinations such as S (mentioned in the previous paragraph), LP, and PS. The S preprocessing method was again a part of the most combinations achieving significant improvements.

Table 9 (TC for the SMS Spam Collection dataset) demonstrates similar findings to those of Tables 7 and 8. Regarding the impact of each single preprocessing method on the SMS Spam Collection dataset, once more, the S preprocessing was the only single preprocessing method that enabled a significant improvement compared to the baseline result; this time, only for the RF method: 98.34% (the best result in Table 9) vs. 97.62% (the best baseline result). The R and L preprocessing methods showed two insignificant improvements and one insignificant decline. The C preprocessing method showed one insignificant improvement, one insignificant decline, and one no change. The P preprocessing method showed significant declines for all three ML methods (similar to the results in Table 7).

For this dataset (the SMS Spam Collection dataset), there were marginal improvements for each ML method; however, only two were significant: both were 98.34% for the RF method using the S and LP combinations.

Table 10 (TC for the Sentiment Labelled Sentences dataset) demonstrates totally different findings compared to those of the other datasets (Tables 7–9). Indeed, each one of the ML methods presents various significant improvements. Regarding the impact of each single preprocessing method on the Sentiment Labelled Sentences dataset, the only single preprocessing method that enabled a significant improvement compared to the baseline result was L (converting uppercase letters into lowercase letters). The L type presents significant improvements by RF (78.22% vs. 75.99%) and BN (69.26% vs. 65.76%) and an insignificant improvement by SMO. The C preprocessing method enabled insignificant improvement for all three ML methods. The R preprocessing method showed two insignificant improvements and one insignificant decline. The P preprocessing method showed insignificant declines for all three ML methods. Quite inversely to the three previous datasets, the S preprocessing method was the worst preprocessing method. The S preprocessing method presented significant declines for two ML methods (i.e., for BN 54.37% vs. a baseline of 65.76% and RF 73.19% vs. a baseline of 75.99%) and one insignificant decline for SMO.

The best result in Table 10, 78.78%v (a significant improvement of 2.79% from the baseline) was obtained by the RF method using CL. The majority of the improvement is because of the L preprocessing, which presents a significant improvement of 2.23% from the baseline; SMO, the second best ML method for the Sentiment Labelled Sentences dataset, demonstrates similar results to those obtained by the RF method. The best result, 75.81% was obtained using CL. The S preprocessing method did not appear in any of the combinations of all three ML methods that achieved significant improvements compared to the baseline results.

Possible explanations for the relatively low TC accuracy results and the surprising preprocessing results for the Sentiment Labelled Sentences dataset could be the fact that many reviews are implicit or indirect or complicated by positive and negative parts intertwined. Another possible explanation is the one presented in Toman et al. [16] that some stopwords are rare in these sentiment sentences (as in spam messages because they are both relatively short) and should not be removed to improve the classification accuracy.

Table 11 presents a summary of the three best accuracy results in descending order for each dataset.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Table 11. Summary of the three best accuracy results in descending order for each dataset.

https://doi.org/10.1371/journal.pone.0232525.t011

Summary, conclusions, and future work

Our main contribution and novelty is performing an extensive and systematic set of TC experiments using all possible combinations of five/six basic preprocessing methods on four benchmark text corpora (and not samples of them) using three ML methods and training and test sets. Specifically, we investigated the influence of 31 (2⁵–1) / 63 (2⁶–1) nonempty combinations of the following five/six basic preprocessing methods (correction of common misspelled words, conversion of uppercase letters into lowercase letters, html object removal, punctuation mark removal, stopword removal, and reduction of replicated characters). That is, we explored 31/63 combinations of preprocessing methods for each combination of dataset and ML method. We validated the proposed model on four benchmark corpora using training and test sets to determine what preprocessing combination(s) was (were) the best suited to classification tasks, if any.

In all four datasets, we have found significant improvements compared to the baseline results. The S (stopword removal) preprocessing method was the best single preprocessing method; it was the only single preprocessing method that enabled a significant improvement for three datasets (WebKB, R8, and SMS Spam Collection). This finding is not surprising because it is not new and many TC systems apply this preprocessing method.

Regarding the best combinations for the different datasets, the C preprocessing method was a part of the best combination for three datasets, while the S (!), P, and L preprocessing methods were part of the best combination for two datasets.

The fourth dataset (Sentiment Labelled Sentences) demonstrated completely different findings to the findings of the other three datasets. The S preprocessing method presented for two ML methods significant declines. Moreover, the S preprocessing method did not appear in any of the combinations for the three ML methods that obtained significant improvements. The L preprocessing method (converting uppercase letters into lowercase letters) was the only single preprocessing method that enabled a significant improvement compared to the baseline result. The best result for this dataset was obtained using the CL combination.

The general conclusion (at least for the datasets verified) is that it is always recommended to perform an extensive and systymatic variety of preprocessing methods combined with TC experiments because it contributes to improve TC accuracy. For each of the verified datasets there was (were) different combination(s) of basic preprocessing methods that could be recommended to significantly improve the TC using a BOW representation. Therefore, the suggested advice is to experiment all possible preprocessing method combinations rather than choosing a specific combination because different combinations can be useful or harmful depending on the selected domain, dataset, ML method, and size of the BOW representation.

Future research proposals include (1) implementing TC experiments using additional feature types, e.g., word/character n-grams, skip word/character n-grams, and TF-IDF values; (2) implementing TC experiments for combinations that include more complex types of preprocessing such as lemmatization of words, expansion of common abbreviations, and using summaries of documents instead of the original documents (HaCohen-Kerner et al. [51]); (3) applying other ML methods such as deep learning methods, and (4) conducting experiments on additional benchmark corpora written in English such as the full 20 Newsgroups data, Reuters-21578, and RCV1, and corpora written in other languages.

Supporting information