Hai Zhao
Skip Abstract Section
Abstract
Chinese word segmentation is an active area in Chinese language processing though it is suffering from the argument about what precisely is a word in Chinese. Based on corpus-based segmentation standard, we launched this study. In detail, we regard Chinese word segmentation as a character-based tagging problem. We show that there has been a potent trend of using a character-based tagging approach in this field. In particular, learning from segmented corpus with or without additional linguistic resources is treated in a unified way in which the only difference depends on how the feature template set is selected. It differs from existing work in that both feature template selection and tag set selection are considered in our approach, instead of the previous feature template focus only technique. We show that there is a significant performance difference as different tag sets are selected. This is especially applied to a six-tag set, which is good enough for most current segmented corpora. The linguistic meaning of a tag set is also discussed. Our results show that a simple learning system with six n-gram feature templates and a six-tag set can obtain competitive performance in the cases of learning only from a training corpus. In cases when additional linguistic resources are available, an ensemble learning technique, assistant segmenter, is proposed and its effectiveness is verified. Assistant segmenter is also proven to be an effective method as segmentation standard adaptation that outperforms existing ones. Based on the proposed approach, our system provides state-of-the-art performance in all 12 corpora of three international Chinese word segmentation bakeoffs.
- Asahara, M., Goh, C. L., Wang, X., and Matsumoto, Y. 2003. Combining segmenter and chunker for Chinese word segmentation. In Proceedings of the 2nd SIGHAN Workshop on Chinese Language Processing (SIGHAN’03). 144--147. Google Scholar
Digital Library
- Brill, E. 1995. Transformation-based error-driven learning and natural language processing: A case study in part-of-speech tagging. Comput. Linguist. 21, 4, 543--565. Google ScholarDigital Library
- Chen, A. 2003. Chinese word segmentation using minimal linguistic knowledge. In Proceedings of the 2nd SIGHAN Workshop on Chinese Language Processing (SIGHAN’03). 148--151. Google ScholarDigital Library
- Cohn, T., Smith, A., and Osborne, M. 2005. Scaling conditional random fields using error-correcting codes. In Proceedings of the 43rd Annual Meeting of the Association for Computational Linguistics (ACL’05). 10--17. Google ScholarDigital Library
- Emerson, T. 2005. The second international Chinese word segmentation bakeoff. In Proceedings of the 4th SIGHAN Workshop on Chinese Language Processing (SIGHAN’05). 123--133.Google Scholar
- Fan, C.-K. and Tsai, W.-H. 1988. Automatic word identification in Chinese sentences by the relaxation technique. Comput. Proc. Chinese Oriental Lang. 4, 1, 33--56.Google Scholar
- Fu, G.-H. and Wang, X.-L. 1999. Unsupervised Chinese word segmentation and unknown word identification. In Proceedings of the 5th Natural Language Processing Pacific Rim Symposium (NLPRS’99). 32--37.Google Scholar
- Gao, J., Li, M., Wu, A., and Huang, C.-N. 2005. Chinese word segmentation and named entity recognition: A pragmatic approach. Comput. Linguist. 31, 4, 531--574. Google ScholarDigital Library
- Gao, J., Wu, A., Li, M., Huang, C.-N., Li, H., Xia, X., and Qin, H. 2004. Adaptive Chinese word segmentation. In Proceedings of the 42nd Annual Meeting of the Association for Computational Linguistics (ACL’04). 462--469. Google ScholarDigital Library
- Goh, C.-L., Asahara, M., and Matsumoto, Y. 2005. Chinese word segmentatin by classification of characters. Comput. Linguist. Chinese Lang. Proc. 10, 3, 381--396.Google Scholar
- Grinstead, C. and Snell, J. L. 1997. Introduction to Probability. American Mathematical Society, Providence, RI.Google Scholar
- Hockenmaier, J. and Brew, C. 1998. Error driven segmentation of Chinese. Comm. COLIPS 8, 1, 69--84.Google Scholar
- Kuncheva, L. I. and Whitaker, C. J. 2003. Measures of diversity in classifier ensembles and their relationship with the ensemble accuracy. Mach. Learn. 51, 2, 181--207. Google ScholarDigital Library
- Lafferty, J. D., McCallum, A., and Pereira, F. C. N. 2001. Conditional random fields: Probabilistic models for segmenting and labeling sequence data. In Proceedings of the 18th International Conference on Machine Learning (ICML’01). 282--289. Google ScholarDigital Library
- Lau, T. P. and King, I. 2005. Two-phase lmr-rc tagging for Chinese word segmentation. In Proceedings of the 4th SIGHAN Workshop on Chinese Language Processing (SIGHAN’05). 183--186.Google Scholar
- Levow, G.-A. 2006. The third international Chinese language processing bakeoff: Word segmentation and named entity recognition. In Proceedings of the 5th SIGHAN Workshop on Chinese Language Processing (SIGHAN’06). 108--117.Google Scholar
- Li, M., Gao, J., Huang, C.-N., and Li, J. 2003. Unsupervised training for overlapping ambiguity resolution in Chinese word segmentation. In Proceedings of the 2nd SIGHAN Workshop on Chinese Language Processing (SIGHAN’03). 1--7. Google ScholarDigital Library
- Li, S. 2005. Chinese word segmentation in ICT-NLP. In Proceedings of the 4th SIGHAN Workshop on Chinese Language Processing (SIGHAN’05). 187--188.Google Scholar
- Liang, P. 2005. Semi-supervised learning for natural language. M.S. thesis, Massachusetts Institute of Technology.Google Scholar
- Low, J. K., Ng, H. T., and Guo, W. 2005. A maximum entropy approach to Chinese word segmentation. In Proceedings of the 4th SIGHAN Workshop on Chinese Language Processing (SIGHAN’05). 161--164.Google Scholar
- Luo, X., Sun, M., and Tsou, B. K. 2002. Covering ambiguity resolution in Chinese word segmentation based on contextual information. In Proceedings of the 19th International Conference on Computational Linguistics (COLING’02). 1--7. Google ScholarDigital Library
- Malouf, R. 2002. A comparison of algorithms for maximum entropy parameter estimation. In Proceedings of the Conference on Natural Language Learning (CoNLL’02). 49--55. Google ScholarDigital Library
- Ng, H. T. and Low, J. K. 2004. Chinese part-of-speech tagging: One-at-a-time or all-at-once? Word-based or character-based? In Proceedings of the Conference on Empirical Methods in Natural Language Processing (EMNLP’04). 277--284.Google Scholar
- Nie, J.-Y., Jin, W., and Hannan, M.-L. 1994. A hybrid approach to unknown word detection and segmentation of Chinese. In Proceedings of the International Conference on Chinese Computing (ICCC’94). 326--335.Google Scholar
- Packard, J. 2000. The Morphology of Chinese: A Linguistics and Cognitive Approach. Cambridge University Press, Cambridge, UK.Google Scholar
- Palmer, D. D. 1997. A trainable rule-based algorithm for word segmentation. In Proceedings of the Association for Computational Linguistics (ACL’97). 321--328. Google ScholarDigital Library
- Peng, F., Feng, F., and McCallum, A. 2004. Chinese segmentation and new word detection using conditional random fields. In Proceedings of the 19th International Conference on Computational Linguistics (COLING’04). 562--568. Google ScholarDigital Library
- Ratnaparkhi, A. 1996. A maximum entropy part-of-speech tagger. In Proceedings of the Empirical Method in Natural Language Processing Conference (EMNLP’96). 133--142.Google Scholar
- Rosenfeld, B., Feldman, R., and Fresko, M. 2006. A systematic cross-comparison of sequence classifiers. In Proceedings of the SIAM International Conference on Data Mining (SDM’06). 563--567.Google Scholar
- Sha, F. and Pereira, F. 2003. Shallow parsing with conditional random fields. In Proceedings of the Conference of the North American Chapter of the Association for Computational Linguistics on Human Language Technology (ACL-HLT’03). 134--141. Google ScholarDigital Library
- Sproat, R. and Emerson, T. 2003. The first international Chinese word segmentation bakeoff. In Proceedings of the 2nd SIGHAN Workshop on Chinese Language Processing (SIGHAN’03). 133--143. Google ScholarDigital Library
- Sproat, R. and Shih, C. 1990. A statistical method for finding word boundaries in Chinese text. Comput. Proc. Chinese Oriental Lang. 4, 4, 336--351.Google Scholar
- Sproat, R. and Shih, C. 2002. Corpus-based methods in Chinese morphology and phonology. In Proceedings of the 19th International Conference on Computational Linguistics (COLING’02).Google Scholar
- Sproat, R., Shih, C., Gale, W., and Chang, N. 1996. A stochastic finite-state word-segmentation algorithm for Chinese. Comput. Linguist. 22, 3, 377--404. Google ScholarDigital Library
- Sun, C., Huang, C.-N., and Guan, Y. 2006. Combinative ambiguity string detection and resolution based on annotated corpus. In Proceedings of the 3rd Student Workshop on Computational Linguistics (SWCL’06).Google Scholar
- Sun, M., Shen, D., and Tsou, B. K. 1998. Chinese word segmentation without using lexicon and hand-crafted training data. In Proceedings of the 36th Annual Meeting of the Association for Computational Linguistics and 17th International Conference on Computational Linguistics (COLING-ACL’98). 1265--1271. Google ScholarDigital Library
- Sun, M. and Tsou, B. K. 2001. A review and evaluation on automatic segmentation of Chinese (in Chinese). Contemporary Linguist. 3, 1, 22--32.Google Scholar
- Tsai, J.-L. 2006. BMM-based Chinese word segmentor with word support model for the SIGHAN bakeoff 2006. In Proceedings of the 5th SIGHAN Workshop on Chinese Language Processing (SIGHAN’06). 130--133.Google Scholar
- Tsai, R. T.-H., Hung, H.-C., Sung, C.-L., Dai, H.-J., and Hsu, W.-L. 2006. On closed task of Chinese word segmentation: An improved CRF model coupled with character clustering and automatically generated template matching. In Proceedings of the 5th SIGHAN Workshop on Chinese Language Processing (SIGHAN’06). 108--117.Google Scholar
- Tseng, H., Chang, P., Andrew, G., Jurafsky, D., and Manning, C. 2005. A conditional random field word segmenter for SIGHAN bakeoff 2005. In Proceedings of the 4th SIGHAN Workshop on Chinese Language Processing (SIGHAN’06). 168--171.Google Scholar
- Wang, X., Lin, X., Yu, D., Tian, H., and Wu, X. 2006. Chinese word segmentation with maximum entropy and N-gram language model. In Proceedings of the 5th SIGHAN Workshop on Chinese Language Processing (SIGHAN’06). 138--141.Google Scholar
- Wu, A. and Jiang, Z. 2000. Statistically-enhanced new word identification in a rule-based Chinese system. In Proceedings of the 2nd Chinese Processing Workshop (ACL’00). 46--51. Google ScholarDigital Library
- Xue, N. 2003. Chinese word segmentation as character tagging. Comput. Linguist. Chinese Lang. Proc. 8, 1, 29--48.Google Scholar
- Xue, N. and Shen, L. 2003. Chinese word segmentation as LMR tagging. In Proceedings of the 2nd SIGHAN Workshop on Chinese Language Processing (SIGHAN’03). 176--179. Google ScholarDigital Library
- Yuan, Y. 1997. Statistics based approaches towards Chinese language processing. Ph.D. thesis, National University of Singapore.Google Scholar
- Zhang, H.-P. and Liu, Q. 2003. Chinese lexical analysis using hierarchical hidden markov model. In Proceedings of the 2nd SIGHAN Workshop on Chinese Language Processing (SIGHAN’03). 63--70. Google ScholarDigital Library
- Zhang, M., Zhou, G.-D., Yang, L.-P., and Ji, D.-H. 2006. Chinese word segmentation and named entity recognition based on a context-dependent mutual information independence model. In Proceedings of the 5th SIGHAN Workshop on Chinese Language Processing (SIGHAN’06). 154--157.Google Scholar
- Zhang, R., Kikui, G., and Sumita, E. 2006. Subword-based tagging by conditional random fields for Chinese word segmentation. In Proceedings of the Human Language Technology Conference/North American Chapter of the Association for Computational Linguistics (HLT/NAACL’06). 193--196. Google ScholarDigital Library
- ZHOU, G. D. 2005. A chunking strategy towards unknown word detection in Chinese word segmentation. In Proceeding of the 2nd International Joint Conference on Natural Language Processing (IJCNLP’05). R. Dale, K.-F. Wong, J. Su, and O. Y. Kwong, Eds. Lecture Notes in Computer Science, vol. 3651, 530--541. Google ScholarDigital Library
- Zhu, M.-H., Wang, Y.-L., Wang, Z.-X., Wang, H.-Z., and Zhu, J.-B. 2006. Designing special post-processing rules for SVM-based Chinese word segmentation. In Proceedings of the 5th SIGHAN Workshop on Chinese Language Processing (SIGHAN’06). 217--220.Google Scholar
Index Terms
- A Unified Character-Based Tagging Framework for Chinese Word Segmentation
Recommendations
Chinese Word Segmentation Based on Maximum Entropy
RSVT '19: Proceedings of the 2019 International Conference on Robotics Systems and Vehicle TechnologyChinese word segmentation has received extensive attention in recent years. The word segmentation method based on character-based tagging improves the performance of word segmentation greatly. This method transforms the word segmentation problem into a ...
An integrated approach to chinese word segmentation and part-of-speech tagging
ICCPOL'06: Proceedings of the 21st international conference on Computer Processing of Oriental Languages: beyond the orient: the research challenges aheadThis paper discusses and compares various integration schemes of Chinese word segmentation and part-of-speech tagging in the framework of true-integration and pseudo-integration. A true-integration approach, named ‘the divide-and-conquer integration', ...
Joint Decoding for Chinese Word Segmentation and POS Tagging Using Character-Based and Word-Based Discriminative Models
IALP '11: Proceedings of the 2011 International Conference on Asian Language ProcessingFor Chinese word segmentation and POS tagging problem, both character-based and word-based discriminative approaches can be used. Experiments show that these two approaches bring different errors and can complement each other. In this paper, we propose ...
Comments