Unpaired Cross-modal Hashing Method for Web News Data

doi:10.3969/j.issn.1006-2475.2024.03.009

Abstract

Abstract: Abstract： Most of the current cross-modal Hashing methods can only be trained when fully paired instances are provided， and are not suitable for a large number of unpaired data in the real world. In order to solve this problem， an unpaired cross-modal Hashing method for Web news data is proposted. Firstly， a feature fusion network is constructed to process the unpaired training data， the modal information is supplemented and improved， and the adversarial loss is used to strengthen the common representation of learning. Secondly， the affinity matrix optimizes the feature distribution of samples and the generated binary codes， so that the semantic relationship between samples is more explicit. Finally， we add a class prediction loss to enhance the discrimination ability of binary codes. Experiments on real network news datasets with paired scenes and unpaired scenes respectively， the results show that the proposed method can be extended to practical applications.

Key words: Key words： cross-modal Hashing, feature fusion, unpaired data, adversarial learning

CLC Number:

TP391

WU Zhao-meng1, ZHANG Cheng-gang2. Unpaired Cross-modal Hashing Method for Web News Data[J]. Computer and Modernization, 2024, 0(03): 54-60.

References

［1］徐文婉，周小平，王佳. 跨模态检索技术研究综述［J］. 计算机工程与应用， 2022，58（23）：12-23.
［2］ BRONSTEIN M M， BRONSTEIN A M， MICHEL F， et al. Data fusion through cross-modality metric learning using similarity-sensitive hashing［C］// Proceedings of the 2010 IEEE Computer Society Conference on Computer Vision and Pattern Recognition. 2010：3594-3601.
［3］ SONG J K， YANG Y， YANG Y， et al. Inter-media Hashing for large-scale retrieval from heterogeneous data sources［C］// Proceedings of the 2013 ACM SIGMOD International Conference on Management of Data. 2013：785-796.
［4］ SU S P， ZHONG Z S， ZHANG C. Deep joint-semantics reconstructing Hashing for large-scale unsupervised cross-modal retrieval［C］// Proceedings of the IEEE/CVF International Conference on Computer Vision. 2019：3027-3035.
［5］ DING G G， GUO Y C， ZHOU J L. Collective matrix factorization hashing for multimodal data［C］// Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 2014：2075-2082.
［6］ LIU S， QIAN S S， GUAN Y， et al. Joint-modal distribution-based similarity hashing for large-scale unsupervised deep cross-modal retrieval［C］// Proceedings of the 43rd International ACM SIGIR Conference on Research and Development in Information Retrieval. 2020：1379-1388.
［7］ ZHANG M J， LI J Z， ZHANG H X， et al. Deep semantic cross modal Hashing with correlation alignment［J］. Neurocomputing， 2020，381（8）：240-251.
［8］ XIE D， DENG C， LI C， et al. Multi-task consistency-preserving adversarial hashing for cross-modal retrieval［J］. IEEE Transactions on Image Processing， 2020，29：3626-3637.
［9］ ZHANG P F， LI Y， HUANG Z， et al. Aggregation-based graph convolutional hashing for unsupervised cross-modal retrieval［J］. IEEE Transactions on Multimedia， 2021，24：466-479.
［10］刘渭滨，邹智元，邢薇薇. 模式分类中的特征融合方法［J］. 北京邮电大学学报， 2017，40（4）：1-8.
［11］ SRIVASTAVA N， SALAKHUTDINOV R. Multimodal learning with deep boltzmann machines［J］. The Journal of Machine Learning Research， 2014，15（1）：2949-2980.
［12］ HU Z K， LIU X， WANG X Z， et al. Triplet fusion network Hashing for unpaired cross-modal retrieval［C］// Proceedings of the 2019 International Conference on Multimedia Retrieva. 2019：141-149.
［13］ GOODFELLOW I， POUGET-ABADIE J， MIRZA M， et al. Generative adversarial networks［J］. Communications of the ACM， 2020，63（11）：139-144.
［14］ WANG B K， YANG Y， XU X， et al. Adversarial cross-modal retrieval［C］// Proceedings of the 25th ACM International Conference on Multimedia. 2017：154-162.
［15］ ZHANG X， ZHOU S Y， FENG J S， et al. Hashgan： Attention-aware deep adversarial Hashing for cross modal retrieval［J］. arXiv preprint arXiv：171109347， 2017.
［16］ JIANG Q Y， LI W J. Deep cross-modal Hashing［C］// Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 2017：3270-3278.
［17］ GU W， GU X Y， GU J Z， et al. Adversary guided asymmetric Hashing for cross-modal retrieval［C］// Proceedings of the 2019 on International Conference on Multimedia Retrieval. 2019：159-167.
［18］ ZHANG J， PENG Y X， YUAN M K. SCH-GAN： Semi-supervised cross-modal Hashing by generative adversarial network［J］. IEEE Transactions on Cybernetics， 2018，50（2）：489-502.
［19］ RASIWASIA N， COSTA PEREIRA J， COVIELLO E， et al. A new approach to cross-modal multimedia retrieval［C］// Proceedings of the 18th ACM International Conference on Multimedia. 2010：251-260.
［20］ EVERINGHAM M， VAN GOOL L， WILLIAMS C K I， et al. The pascal visual object classes （voc） challenge ［J］. International Journal of Computer Vision， 2009，88（2）：303-338.
［21］ ZHONG F M， CHEN Z K， MIN G Y. Deep discrete cross-modal Hashing for cross-media retrieval［J］. Pattern Recognition， 2018，83：64-77.
［22］ SIMONYAN K， ZISSERMAN A. Very deep convolutional networks for large-scale image recognition［J］. arXiv preprint arXiv：14091556， 2014.
［23］ LE Q V， MIKOLOV T. Distributed representations of sentences and documents［J］. arXiv preprint arXiv：1405.4053，
2014.
［24］ LI K P， ZHANG Y L， LI K， et al. Visual semantic reasoning for image-text matching［C］// Proceedings of the IEEE/CVF International Conference on Computer Vision. 2019：390-406.
［25］ CHEN H， DING G G， LIU X D， et al. IMRAM： Iterative matching with recurrent attention memory for cross-modal image-text retrieval［C］// Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2020：12652-12660.
［26］ LU H Y， FEI N Y， HUO Y Q， et al. COTS： Collaborative two-stream vision-language pre-training model for cross-modal retrieval［C］// Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2022：15671-15680.
［27］ ZHENG Y， ZHANG X. Heterogeneous interactive learning network for unsupervised cross-modal retrieval［C］// Proceedings of the Asian Conference on Computer Vision. 2022.

[1]	ZHANG Simin, LIU Xinmei, YIN Junling, LI Baoling. PCB Defect Detection Method Based on Improved YOLOv7 [J]. Computer and Modernization, 2024, 0(12): 45-52.
[2]	WANG Haiyang, GONG Tongxin, YANG Jintao, CHEN Zailong. Short-term Load Forecasting in Industrial Parks with Multi-scale Time Coding [J]. Computer and Modernization, 2024, 0(12): 59-65.
[3]	MA Yu, YANG Yong, REN Ge, Palidan Tuerxun. Automated Essay Scoring Method Based on GCN and Fine Tuned BERT [J]. Computer and Modernization, 2024, 0(09): 33-37.
[4]	ZHENG Shangpo1, CHEN Defu1, LI Jianli2, LIN Guoxian2, WANG Xingping3. Pedestrian Tracking Algorithm Based on Improved YOLOv5s and DeepSORT [J]. Computer and Modernization, 2024, 0(08): 54-58.
[5]	PANG Mei, WANG Gong, ZHAN Yong, HUANG Zhefa. Underwater Trash Detection Method Based on Improved YOLOv5 [J]. Computer and Modernization, 2024, 0(07): 120-126.
[6]	FU Lingli, QIU Yu, ZHANG Xinchen. Retinal Vessel Segmentation Based on Improved U-Net with Multi-feature Fusion [J]. Computer and Modernization, 2024, 0(06): 76-82.
[7]	ZHU Fen, HE Lifeng, SUN Shuang, ZHANG Mengying, YU Jiajia. Pancreas Segmentation Model Based on Deformable Residual and Cascading Encoding [J]. Computer and Modernization, 2024, 0(06): 83-88.
[8]	NING Juan, ZHOU Qing-hua, ZENG Xiao-wei. Application of Improved YOLOv7 Algorithm in Detection of Capping Defects of Vials [J]. Computer and Modernization, 2023, 0(12): 82-86.
[9]	CHEN Jun-yi. Health Event Prediction Model Based on Dynamic and Static Features of Graph Nodes [J]. Computer and Modernization, 2023, 0(10): 39-44.
[10]	XING Shi-shuai, LIU Dan-feng, WANG Li-guo, PAN Yue-tao, MENG Ling-hong, YUE Xiao-han. Image Super-resolution Reconstruction Based on Spatial Attention Residual Network [J]. Computer and Modernization, 2023, 0(10): 45-52.
[11]	CHEN Jia-min, ZHANG Bo-quan, MAI Hai-peng. Hippocampus Segmentation Based on Feature Fusion [J]. Computer and Modernization, 2023, 0(08): 1-6.
[12]	WANG Hong, GE Hong. Cross Modal Hash Retrieval Based on Attention Mechanism and Semantic Similarity [J]. Computer and Modernization, 2023, 0(08): 44-53.
[13]	WANG Jie, PAN Feng, ZHANG Yan-sha, TAN Mian, YAN Xiao-bo, WANG Lin, . Surface Defect Classification of Aluminum Profiles with Weighted Non-local Modules [J]. Computer and Modernization, 2023, 0(05): 86-92.
[14]	ZHU Li-qing, LI Xiang, . Vehicle Detection of Remote Sensing Images Based on Improved YOLOv5 Algorithm [J]. Computer and Modernization, 2023, 0(05): 117-121.
[15]	ZHANG Xiao-dong, WANG Xu-ying, QIN Zi-xuan. Pump Detection Period Predicting of Pump Well Based on Feature Fusion [J]. Computer and Modernization, 2022, 0(12): 60-66.