计算机与现代化 ›› 2024, Vol. 0 ›› Issue (07): 47-62.doi: 10.3969/j.issn.1006-2475.2024.07.008

• 人工智能 • 上一篇    下一篇

基于模态类别的多模态信息处理与融合综述


  

  1. (中国石油大学(华东)计算机科学与技术学院,山东 青岛 266580)
  • 出版日期:2024-07-25 发布日期:2024-08-08
  • 基金资助:
    山东省自然科学基金资助项目(ZR202211180156)

Survey on Multimodal Information Processing and Fusion Based on Modal Categories

  1. (College of Computer Science and Technology, China University of Petroleum (East China), Qingdao 266580, China)
  • Online:2024-07-25 Published:2024-08-08

摘要: 随着人工智能和深度学习技术的不断发展,多模态信息处理与融合领域的相关研究受到了研究者们的广泛关注。本文总结多模态信息处理的发展历史和里程碑式的工作,以及多模态融合策略和模型。根据模态类别的不同,分类整理多模态信息处理与融合的主流数据集。以模态类型作为分类标准,本文系统地梳理多模态信息处理与融合的研究进展,强调不同模态之间的区别,并将多模态信息处理与融合分为:视听处理与融合、声文处理与融合、视觉-文本处理与融合和视觉-音频-文本处理与融合4种类别,对不同输入模态的处理融合方法与模型进行详细的研究。最后针对多模态处理与融合领域的发展进行总结与展望。

关键词:  , 多模态处理; 多模态信息处理; 多模态融合; 深度学习

Abstract:  With the continuous advancement of artificial intelligence and deep learning technologies, research in the field of multimodal information processing and fusion has garnered widespread attention from researchers. This paper provides a comprehensive overview of the development history and milestone works of multimodal information processing, along with strategies and models for multimodal fusion. Based on different modalities,mainstream datasets for multimodal information processing and fusion are systematically classified and summarized. Using modality type as the classification criterion, this paper systematically reviews the research progress in multimodal information processing and fusion, emphasizing the distinctions between different modalities. Multimodal information processing and fusion are categorized into four types: audio-visual processing and fusion, audio-text processing and fusion, visual-text processing and fusion, and visual-audio-text processing and fusion. Detailed investigations are conducted on methods and models for processing and fusing different input modalities. Finally, a summary and outlook on the development of multimodal processing and fusion are provided.

Key words:  , multimodal processing; multimodal information processing; multimodal fusion; deep learning

中图分类号: