关键词: TF-IDF, 特征词提取, 词长, 文本预处理, 文本分类

Abstract: In the text feature word extraction algorithm,TF-IDF algorithm is the most common feature weight calculation method. On the basis of the traditional TF-IDF extract algorithm, a new keyword extraction algorithm based on the text word length is proposed.Using chinese phrase word segmentation technique to identify long words and ordinary words in text,the proposed TF-IDF-WL method is used to recompute weights for different lengths of words, and the keywords are sorted by weights. Experimental results show that the new feature word extraction algorithm can more accurately reflect the lexical length of the feature words.Compared with the traditional TF-IDF algorithm, the algorithm has greatly improved accuracy and recall rate.

Key words: TF-IDF, Keyword extraction, Word length, Text preprocessing, Text classification