Abstract: The fine-tuning method using the pre-trained language model has achieved good results in many natural language processing tasks represented by text classification, BERT model based on the Transformer framework as a typical representative especially. However, BERT uses the vector corresponding to ［CLS］ as the text representation directly, and does not consider the local features and global features of texts, which limits the classification performance of the model. Therefore, this paper proposes a text classification model that introduces a pooling operation, and uses pooling methods such as average pooling, maximum pooling, and K-MaxPooling to extract the representation vector of texts from the output matrix of BERT. The experimental results show that compared with the original BERT model, the text classification model with pooling operation proposed in this paper has better performance. In all text classification tasks in the experiment, its accuracy and F1-Score value are better than BERT model.

Key words: text classification, classification model, BERT, mean-pooling, max-pooling, K-MaxPooling