关键词: 中文分词, 评分机制, 庖丁解牛

Abstract: Chinese word segmentation is one of main challenges for search engine. By analyzing the scoring mechanism of the document of Nutch, for the situation that word segmentation of Chinese word segmentation module of Nutch does not conform to Chinese language habit, this paper proposes to use Paodingjieniu Chinese word segmentation module based on dictionary to segment the data collected by Nutch, describes the method that how to realize Paodingjieniu Chinese word segmentation module on Nutch, then tests the word segmentation module. Experiments show that the word segmentation result of Paodingjieniue word segmentation module more conforms to Chinese language habit, and the coverage of terms are more balanced for documents, in addition, 20%~65% of the storage space of index file is saved.

Key words: Chinese word segmentation, scoring mechanism, Paodingjieniu