Abstract
A stop list, or negative dictionary is a device used in automatic indexing to filter out words that would make poor index terms. Traditionally stop lists are supposed to have included only the most frequently occurring words. In practice, however, stop lists have tended to include infrequently occurring words, and have not included many frequently occurring words. Infrequently occurring words seem to have been included because stop list compilers have not, for whatever reason, consulted empirical studies of word frequencies. Frequently occurring words seem to have been left out for the same reason, and also because many of them might still be important as index terms.This paper reports an exercise in generating a stop list for general text based on the Brown corpus of 1,014,000 words drawn from a broad range of literature in English. We start with a list of tokens occurring more than 300 times in the Brown corpus. From this list of 278 words, 32 are culled on the grounds that they are too important as potential index terms. Twenty-six words are then added to the list in the belief that they may occur very frequently in certain kinds of literature. Finally, 149 words are added to the list because the finite state machine based filter in which this list is intended to be used is able to filter them at almost no cost. The final product is a list of 421 stop words that should be maximally efficient and effective in filtering the most frequently occurring and semantically neutral words in general literature in English.
- van Rijsbergen, C. J., Information Retrieval, Butterworths, 1975. Google ScholarDigital Library
- Luhn, H. P., "A Statistical Approach to Mechanized Encoding and Searching of Literary Information," IBM Journal of Research and Development 1(4), October, 1957.Google ScholarDigital Library
- Francis, W. Nelson, and Henry, Kucera, Frequency Analysis of English Usage, Houghton Mifflin, 1982.Google Scholar
- Aho, Alfred, Ravi Sethi, and Jeffrey Ullman, Compilers: Principles, Techniques, and Tools, Addison-Wesley, 1986. Google ScholarDigital Library
Recommendations
Automatic Construction of Generic Stop Words List for Hindi Text
AbstractIn this technological world, Hindi text is speedily increasing on web and attracting many users and researchers to retrieve useful information from this data. In Information Retrieval (IR) process, user comes across many words of least or no ...
Automatic construction of Chinese stop word list
ACOS'06: Proceedings of the 5th WSEAS international conference on Applied computer scienceIn modern information retrieval systems, effective indexing can be achieved by removal of stop words. Till now many stop word lists have been developed for English language. However, no standard stop word list has been constructed for Chinese language ...
Creation of a Russian Stop Word List
Abstract—This article describes three identifying characteristics of stop words—statistical, semantic, and morphological—and postulates new principles for the creation of stop word lists based on these characteristics. The application of the principles is ...
Comments