Feature-level Multimodal Fusion for Depression Recognition

doi:10.3969/j.issn.1006-2475.2023.10.003

Abstract

Abstract: Abstract: Depression is a common psychiatric disorder. However， the existing diagnostic methods for depression mainly rely on scales and interviews with psychiatrists， which are highly subjective. In recent years， researchers have devoted themselves to identifying depressed patients by EEG features or audio features， but no study has effectively combined EEG information with audio information， ignoring the correlation between audio and EEG data. Therefore， this study proposes a feature-level multimodal fusion model to improve the accuracy of depression recognition. We combine the audio and EEG modality information based on a fully connected neural network. Our experiments show that the accuracy of depression recognition using feature-level multimodal fusion model on the MODMA dataset reaches 81.58%， which is higher than that of using single-modality. The results indicate that the feature-level multimodal fusion model can improve the accuracy of depression recognition compared to single-modality. Our research provides a new perspective and method for depression recognition.

Key words: Key words: multimodal data fusion, depression detection, feature-level fusion, fully-connected neural networks

CLC Number:

TP399

GU Ming-xuan, FAN Bing-bing. Feature-level Multimodal Fusion for Depression Recognition[J]. Computer and Modernization, 2023, 0(10): 17-22.

References

［1］廖成菊，冯正直. 抑郁症情绪加工与认知控制的脑机制［J］. 心理科学进展， 2010，18（2）:282-287.
［2］祁荣，陈军，余邵民. 关于抑郁症的研究综述［J］. 心理月刊， 2020，15（17）:238-240.
［3］ CASSANO P， FAVA M. Depression and public health: An overview［J］. Journal of Psychosomatic Research， 2002，53（4）:849-857.
［4］祁娜，冯媛，王刚. 抑郁症客观评估方法的研究进展［J］. 神经疾病与精神卫生， 2020，20（5）:341-346.
［5］汪作为，彭代辉，刘晓华，等. 忧郁/快感缺失型抑郁症临床评估与诊治指导建议［J］. 临床精神医学杂志， 2021，31（1）:1-5.
［6］ KROENKE K， SPITZER R L， WILLIAMS J B W. The PHQ-9: Validity of a brief depression severity measure［J］. Journal of General Internal Medicine， 2001，16（9）:606-613.
［7］ VALSTAR M， GRATCH J， SCHULLER B， et al. AVEC 2016: Depression， mood， and emotion recognition workshop and challenge［C］// Proceedings of the 6th International Workshop on Audio/Visual Emotion Challenge. 2016:3-10.
［8］ OLBRICH S， ARNS M. EEG biomarkers in major depressive disorder: Discriminative power and prediction of treatment response［J］. International Review of Psychiatry， 2013，25（5）:604-618.
［9］ ZHOU Z H， FENG J. Deep forest: Towards an alternative to deep neural networks［C］// Proceedings of the 26th International Joint Conference on Artificial Intelligence. 2017:3553-3559.
［10］ ISLAM M R， KABIR M A， AHMED A， et al. Depression detection from social network data using machine learning techniques［J］. Health Information Science and Systems， 2018，6（1）. DOI: 10.1007/s13755-018-0046-0.
［11］ FINGELKURTS A A， FINGELKURTS A A， BAGNATO S， et al. EEG oscillatory states as neuro-phenomenology of consciousness as revealed from patients in vegetative and minimally conscious states［J］. Consciousness and Cognition， 2012，21（1）:149-169.
［12］蒲涛，许莉，蒲涛青，等. 不同严重程度抑郁症患者SPECT/CT脑血流灌注显像特点分析［J］. 中国CT和MRI杂志， 2022，20（8）:24-27.
［13］ LAVE J R， FRANK R G， SCHULBERG H C， et al. Cost-effectiveness of treatments for major depression in primary care practice［J］. Archives of General Psychiatry， 1998，55（7）:645-651.
［14］ ERGUZEL T T， OZEKES S， TAN O， et al. Feature selection and classification of electroencephalographic signals: An artificial neural network and genetic algorithm based approach［J］. Clinical EEG and Neuroscience， 2015，46（4）:321-326.
［15］ HOSSEINIFARD B， MORADI M H， ROSTAMI R. Classifying depression patients and normal subjects using machine learning techniques and nonlinear features from EEG signal［J］. Computer Methods and Programs in Biomedicine， 2013，109（3）:339-345.
［16］ ORGO L， BACHMANN M， KALEV K， et al. Resting EEG functional connectivity and graph theoretical measures for discrimination of depression［C］// Proceedings of the 2017 IEEE EMBS International Conference on Biomedical & Health Informatics （BHI）. 2017:389-392.
［17］ PENG H， XIA C， WANG Z H， et al. Multivariate pattern analysis of EEG-based functional connectivity: A study on the identification of depression［J］. IEEE Access， 2019，7:92630-92641.
［18］ BALANO J B， HUERTO V L， SANCHEZ S， et al. Determining the level of depression using BDI-II through voice recognition［C］// Proceedings of the 2019 IEEE 6th International Conference on Industrial Engineering and Applications （ICIEA）. 2019:387-391.
［19］ FLINT A J， BLACK S E， CAMPBELL-TAYLOR I， et al. Abnormal speech articulation， psychomotor retardation， and subcortical dysfunction in major depression［J］. Journal of Psychiatric Research， 1993，27（3）:309-319.
［20］任泽裕，王振超，柯尊旺，等. 多模态数据融合综述［J］. 计算机工程与应用， 2021，57（18）:49-64.
［21］ CAI H S， QU Z D， LI Z， et al. Feature-level fusion approaches based on multimodal EEG data for depression recognition［J］. Information Fusion， 2020，59:127-138.
［22］何俊，张彩庆，李小珍，等. 面向深度学习的多模态融合技术研究综述［J］. 计算机工程， 2020，46（5）:1-11.
［23］ KAHOU S E， PAL C， BOUTHILLIER X， et al. Combining modality specific deep neural networks for emotion recognition in video［C］// Proceedings of the 15th ACM on International Conference on Multimodal Interaction. 2013:543-550.
［24］ YANG L， JIANG D M， XIA X H， et al. Multimodal measurement of depression using deep learning models［C］// Proceedings of the 7th Annual Workshop on Audio/Visual Emotion Challenge. 2017:53-59.
［25］张迎辉，聂燕敏，孙波，等. 基于深度森林多模态数据决策级融合抑郁症评价方法［J］. 北京师范大学学报（自然科学版）， 2018，54（5）:606-611.
［26］ WU D， PIGOU L， KINDERMANS P J， et al. Deep dynamic neural networks for multimodal gesture segmentation and recognition［J］. IEEE Transactions on Pattern Analysis and Machine Intelligence， 2016，38（8）:1583-1597.
［27］ LAN Z Z， BAO L， YU S I， et al. Multimedia classification and event detection using double fusion［J］. Multimedia Tools and Applications， 2014，71（1）:333-347.
［28］ CAI H S， YUAN Z Q， GAO Y W， et al. A multi-modal open dataset for mental-disorder analysis［J］. Scientific Data， 2022，9（1）. DOI: 10.1038/s41597-022-01211-x.
［29］ DELORME A， MAKEIG S. EEGLAB: An open source toolbox for analysis of single-trial EEG dynamics including independent component analysis［J］. Journal of Neuroscience Methods， 2004，134（1）:9-21.
［30］ WIDMANN A， SCHROGER E， MAESS B. Digital filter design for electrophysiological data: A practical approach［J］. Journal of Neuroscience Methods， 2015，250:34-46.
［31］ STAM C J， NOLTE G， DAFFERTSHOFER A. Phase lag index: Assessment of functional connectivity from multi channel EEG and MEG with diminished bias from common sources［J］. Human Brain Mapping， 2007，28（11）:1178-1193.
［32］ ZENG L L， SHEN H， LIU L， et al. Identifying major depression using whole-brain functional connectivity: A multivariate pattern analysis［J］. Brain， 2012，135（5）:1498-1507.
［33］ VERGIN R， O'SHAUGHNESSY D. Pre-emphasis and speech recognition［C］// Proceedings of the 1995 Canadian Conference on Electrical and Computer Engineering. 1995，2:1062-1065.
［34］ SOHN J， KIM N S， SUNG W. A statistical model-based voice activity detection［J］. IEEE Signal Processing Letters， 1999，6（1）:1-3.
［35］ EYBEN F， WOLLMER M， SCHULLER B. OpenSMILE: The Munich versatile and fast open-source audio feature extractor［C］// Proceedings of the 18th ACM International Conference on Multimedia. 2010:1459-1462.
［36］罗涛，李剑峰，韩家辉，等. 一种基于多模态特征融合的骨质疏松评估方法［J］. 北京邮电大学学报， 2019，42（6）:84-90.
［37］ XU L， FU H Y， GOODARZI M， et al. Stochastic cross validation［J］. Chemometrics and Intelligent Laboratory Systems， 2018，175:74-81.

[1]	WAN Bing1, 2, 3, ZHAO Wentao4, PAN Duotao1, ZHAO Zhengtao2, 3, SUN Zhaoyang2, 3, YU Jiancheng2, 3 . Design of Semi-physical Simulation Test System for Unmanned Sailboat [J]. Computer and Modernization, 2024, 0(12): 91-99.
[2]	YANG Bo, WANG Hongjie, XU Shengchao, MAO Mingyang, JIANG Jinling, JIANG Darui. A Task Scheduling Method for Biological Gene Multi Sequence Alignment Algorithm [J]. Computer and Modernization, 2024, 0(07): 7-12.
[3]	CAO Ning1, XU Genqi2, ZHANG Wen3, XU Youwen1, HE Panqing1. Prediction Method of Mountain Flood Disaster Based on AFSPSO-ν-SVM [J]. Computer and Modernization, 2024, 0(05): 33-37.
[4]	XIONG Qing-zhi1, LI Xiang1, 2, PENG Fang-wei1, JIN An-an1. A Data-Driven Intelligent Analysis Platform for Ion Source Data [J]. Computer and Modernization, 2024, 0(02): 121-126.