关键词: 文本摘要, 评测方法, 语义损失率, 数据集偏差

Abstract: In the current field of text summarization automatic generation， the traditional ROUGE evaluation method has been repeatedly found by researchers that the gap between its evaluation results and artificial evaluation results is too large， but the gap has not been numerical and cannot be measured. Based on this situation， this paper uses multiple public Chinese summary datasets of different types and lengths to measure the degree of semantic loss generated by ROUGE in the evaluation by defining the calculation method of semantic loss rate. At the same time， it comprehensively considers the influence of summary length and internal factors of datasets on the generation of summary evaluation， and the specific values of errors between ROUGE evaluation and artificial evaluation are visualized finally. The experimental results show that the ROUGE evaluation score is weakly correlated with the artificial evaluation score. ROUGE method has a certain degree of semantic loss for different length datasets， and the length of the summary and the original annotation error of the datasets will also have an important impact on the final evaluation score. The calculation method of semantic loss rate defined in this paper can provide a certain reference for better selection of datasets and evaluation methods， provide a direction of thinking for improving evaluation methods， and also provide certain a guidance and help for the effectiveness of the final objective evaluation model.

Key words: text summarization, evaluation method, semantic-loss rate, dataset bias