Multi-object Tracking of UAV Based on Improved YOLOX and New Data Association Method

doi:10.3969/j.issn.1006-2475.2024.08.011

Abstract

Abstract: Multi-object tracking in UAV videos is a crucial computer vision task with extensive applications across various domains. To address the challenges of occlusions， small objects， and complex， varying backgrounds in UAV video scenes， an improved UAV multi-object tracking model is proposed. This paper improves the YOLOX network by integrating the Swin Transformer to enhance global information extraction capabilities and adding an additional detection head to boost the detection performance of small objects. Furthermore， this paper introduces the CBAM attention module to focus on informative features. In the data association stage， this paper adopts a new data association approach that retains all detection boxes， categorizing them into high-scoring and low-scoring detection boxes based on their confidence scores. The first association is performed between high-scoring detection boxes and tracking trajectories， while the second association is performed between unmatched trajectories and low-scoring detection boxes. Experimental results on the public datasets VisDrone2021 and UAVDT demonstrate that the proposed method exhibits relatively high superiority and robustness in UAV multi-object tracking scenarios.

Key words: multi-object tracking, unmanned aerial vehicles（UAV） videos, attention mechanism, data association

CLC Number:

TP391

FU Shugang1, 2, 3. Multi-object Tracking of UAV Based on Improved YOLOX and New Data Association Method[J]. Computer and Modernization, 2024, 0(08): 59-66.

References

［1］ PATRICIO D I， RIEDER R. Computer vision and artificial intelligence in precision agriculture for grain crops： A systematic review［J］. Computers and Electronics in Agriculture， 2018，153：69-81.
［2］ VOULODIMOS A， DOULAMIS N， DOULAMIS A， et al. Deep learning for computer vision： A brief review［J］. Computational Intelligence and Neuroscience， 2018，2018. DOI： 10.1155/2018/7068349.
［3］ JANAI J， GUNEY F， BEHL A， et al. Computer vision for autonomous vehicles： Problems， datasets and state of the art［J］. Foundations and Trends in Computer Graphics and Vision， 2020，12（1-3）：1-308.
［4］ FERNANDEZ-SANJURJO M， MUCIENTES M， BREA V M. Real-time multiple object visual tracking for embedded GPU systems［J］. IEEE Internet of Things Journal， 2021，8（11）：9177-9188.
［5］ CHEN T， PENNISI A， LI Z， et al. A hierarchical association framework for multi-object tracking in airborne videos［J］. Remote Sensing， 2018，10（9）. DOI： 10.3390/rs10091347.
［6］ KOK K Y， RAJENDRAN P. Differential-evolution control parameter optimization for unmanned aerial vehicle path planning［J］. PLoS One， 2016，11（3）. DOI： 10.1371/journal.pone.0150558.
［7］ DELIBASOGLU I. UAV images dataset for moving object detection from moving cameras［J］. arXiv preprint arXiv：2103.11460， 2021.
［8］ ZHU P F， WEN L Y， DU D W， et al. Detection and tracking meet drones challenge［J］. IEEE Transactions on Pattern Analysis and Machine Intelligence， 2022，44（11）：7380-7399.
［9］ AZIMI S M， KRAUS M， BAHMANYAR R， et al. Multiple pedestrians and vehicles tracking in aerial imagery： A comprehensive study［J］. arXiv preprint arXiv：2010.09689， 2020.
［10］ WEN L Y， DU D W， ZHU P F， et al. Detection， tracking， and counting meets drones in crowds： A benchmark［C］// Proceedings of the 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition. IEEE， 2021：7808-7817.
［11］ VARGA L A， KIEFER B， MESSMER M， et al. SeaDronesSee： A maritime benchmark for detecting humans in open water［C］// Proceedings of the 2022 IEEE/CVF Winter Conference on Applications of Computer Vision. IEEE， 2022：3686-3696.
［12］ KAMPFFMEYER M， SALBERG A B， JENSSEN R. Semantic segmentation of small objects and modeling of uncertainty in urban remote sensing images using deep convolutional neural networks［C］// Proceedings of the 2016 IEEE Conference on Computer Vision and Pattern Recognition Workshops. IEEE， 2016：680-688.
［13］ AHMED I， AHMAD M， AHMAD A， et al. Top view multiple people tracking by detection using deep SORT and YOLOv3 with transfer learning： Within 5G infrastructure［J］. International Journal of Machine Learning and Cybernetics， 2021，12（11）：3053-3067.
［14］ WANG Z Y， MIAO D Q， ZHAO C R， et al. A robust long-term pedestrian tracking-by-detection algorithm based on three-way decision［C］// Proceedings of the 2019 International Joint Conference on Rough Sets. Springer， 2019：522-533.
［15］ XIE Y F， HUANG Y， SONG T L. Iterative joint integrated probabilistic data association filter for multiple-detection multiple-target tracking［J］. Digital Signal Processing， 2018，72：232-243.
［16］ WANG Z D， ZHENG L， LIU Y X， et al. Towards real-time multi-object tracking［C］// Proceedings of the 16th European Conference on Computer Vision. Springer， 2020：107-122.
［17］ MUNJAL B， AFTAB A R， AMIN S， et al. Joint detection and tracking in videos with identification features［J］. Image and Vision Computing， 2020，100. DOI： 10.1016/j.imavis.2020.103932.
［18］ LIN X W， GUO Y A， WANG J Q. Global correlation network： End-to-end joint multi-object detection and tracking［J］. arXiv preprint arXiv：2103.12511， 2021.
［19］ SUN Z H， CHEN J， CHAO L， et al. A survey of multiple pedestrian tracking based on tracking-by-detection framework［J］. IEEE Transactions on Circuits and Systems for Video Technology， 2021，31（5）：1819-1833.
［20］ WU J L， CAO J L， SONG L C， et al. Track to detect and segment： An online multi-object tracker［C］// Proceedings of the 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition. IEEE， 2021：12347-12356.
［21］ WOJKE N， BEWLEY A， PAULUS D. Simple online and realtime tracking with a deep association metric［C］// Proceedings of the 2017 IEEE International Conference on Image Processing （ICIP）. IEEE， 2017：3645-3649.
［22］ GIRSHICK R， DONAHUE J， DARRELL T， et al. Rich feature hierarchies for accurate object detection and semantic segmentation［C］// Proceedings of the 2014 IEEE Conference on Computer Vision and Pattern Recognition. IEEE， 2014：580-587.
［23］ REN S Q， HE K M， GIRSHICK R， et al. Faster R-CNN： Towards real-time object detection with region proposal networks［J］. arXiv preprint arXiv：1506.01497， 2015.
［24］ LIN T Y， GOYAL P， GIRSHICK R， et al. Focal loss for dense object detection［C］// Proceedings of the 2017 IEEE International Conference on Computer Vision. IEEE， 2017：2999-3007.
［25］ ZHOU X Y， KOLTUN V， KRAHENBUHL P. Tracking objects as points［C］// Proceedings of the 16th European Conference on Computer Vision. Springer， 2020：474-490.
［26］ REDMON J， DIVVALA S， GIRSHICK R， et al. You only look once： Unified， real-time object detection［C］// Proceedings of the 2016 IEEE Conference on Computer Vision and Pattern Recognition. IEEE， 2016：779-788.
［27］ GE Z， LIU S T， WANG F， et al. YOLOX： Exceeding YOLO series in 2021［J］. arXiv preprint arXiv：2107.08430， 2021.
［28］ BOCHKOVSKIY A， WANG C Y， LIAO H Y M. YOLOv4： Optimal speed and accuracy of object detection［J］. arXiv preprint arXiv：2004.10934， 2020.
［29］ JOCHER G， STOKEN A， BOROVEC J， et al. Ultralytics/YOLOv5： v5.0-YOLOv5-P6 1280 models， AWS， Supervise.ly and YouTube integrations［DB/OL］. （2021-04-11）［2023-03-19］. https：//ui.adsabs.harvard.edu/abs/2021zndo...4679653J/abstract.
［30］ LU Z C， RATHOD V， VOTEL R， et al. RetinaTrack： Online single stage joint detection and tracking［C］// Proceedings of the 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition. IEEE， 2020：14656-14666.
［31］ ZHANG Y F， WANG C Y， WANG X G， et al. FairMOT： On the fairness of detection and re-identification in multiple object tracking［J］. International Journal of Computer Vision， 2021，129（11）：3069-3087.
［32］ CHU P， WANG J， YOU Q Z， et al. TransMOT： Spatial-temporal graph transformer for multiple object tracking［C］// Proceedings of the 2023 IEEE/CVF Winter Conference on Applications of Computer Vision. IEEE， 2023：4859-4869.
［33］ KALMAN R E. A new approach to linear filtering and prediction problems［J］. Journal of Basic Engineering， 1960，82（1）：35-45.
［34］ BEWLEY A， GE Z Y， OTT L， et al. Simple online and realtime tracking［C］// Proceedings of the 2016 IEEE International Conference on Image Processing （ICIP）. IEEE， 2016：3464-3468.
［35］ DU Y H， WAN J F， ZHAO Y Y， et al. GIAOTracker： A comprehensive framework for MCMOT with global information and optimizing strategies in VisDrone 2021［C］// Proceedings of the 2021 IEEE/CVF International Conference on Computer Vision Workshops. IEEE， 2021：2809-2819.
［36］ VASWANI A， SHAZEER N， PARMAR N， et al. Attention is all you need［J］. arXiv preprint arXiv：1706.03762， 2017.
［37］ WOO S， PARK J， LEE J Y， et al. CBAM： Convolutional block attention module［C］// Proceedings of the 2018 European Conference on Computer Vision. Springer， 2018：3-19.
［38］ WANG C Y， LIAO H Y M， WU Y H， et al. CSPNet： A new backbone that can enhance learning capability of CNN［C］// Proceedings of the 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops. IEEE， 2020：1571-1580.
［39］ HE K M， ZHANG X Y， REN S Q， et al. Spatial pyramid pooling in deep convolutional networks for visual recognition［J］. IEEE Transactions on Pattern Analysis and Machine Intelligence， 2015，37（9）：1904-1916.
［40］ LIU S， QI L， QIN H F， et al. Path aggregation network for instance segmentation［C］// Proceedings of the 2018 IEEE Conference on Computer Vision and Pattern Recognition. IEEE， 2018：8759-8768.
［41］ LIN T Y， DOLLAR P， GIRSHICK R， et al. Feature pyramid networks for object detection［C］// Proceedings of the 2017 IEEE Conference on Computer Vision and Pattern Recognition. IEEE， 2017：936-944.
［42］ DOSOVITSKIY A， BEYER L， KOLESNIKOV A， et al. An image is worth 16x16 words： Transformers for image recognition at scale［J］. arXiv preprint arXiv：2010.11929， 2020.
［43］ CARION N， MASSA F， SYNNAEVE G， et al. End-to-end object detection with transformers［C］// Proceedings of the 16th European Conference on Computer Vision. Springer， 2020：213-229.
［44］ LIU Z， LIN Y T， CAO Y， et al. Swin transformer： Hierarchical vision transformer using shifted windows［C］// Proceedings of the 2021 IEEE/CVF International Conference on Computer Vision. IEEE， 2021：9992-10002.
［45］ ZHANG Y F， SUN P Z， JIANG Y， et al. ByteTrack： Multi-object tracking by associating every detection box［C］// Proceedings of the 17th European Conference on Computer Vision. Springer， 2022：1-21.
［46］ CHEN G L， WANG W G， HE Z J， et al. VisDrone-MOT2021： The vision meets drone multiple object tracking challenge results［C］// Proceedings of the 2021 IEEE/CVF International Conference on Computer Vision Workshops. IEEE， 2021：2839-2846.
［47］ DU D W， QI Y K， YU H Y， et al. The unmanned aerial vehicle benchmark： Object detection and tracking［C］// Proceedings of the 2018 European Conference on Computer Vision. Springer， 2018：375-391.
［48］ CHEN L， AI H Z， ZHUANG Z J， et al. Real-time multiple people tracking with deeply learned candidate selection and person re-identification［C］// Proceedings of the 2018 IEEE International Conference on Multimedia and Expo （ICME）. IEEE， 2018. DOI： 10.1109/ICME.2018.8486597.
［49］ BOCHINSKI E， EISELEIN V， SIKORA T. High-speed tracking-by-detection without using image information［C］// Proceedings of the 2017 14th IEEE International Conference on Advanced Video and Signal Based Surveillance （AVSS）. IEEE， 2017. DOI： 10.1109/AVSS.2017.8078516.
［50］PIRSIAVASH H， RAMANAN D， FOWLKES C C. Globally-optimal greedy algorithms for tracking a variable number of objects［C］// Proceedings of the 2011 Conference on Computer Vision and Pattern Recognition （CVPR）. IEEE， 2011：1201-1208.
［51］ ZENG F G， DONG B， ZHANG Y A， et al. MOTR： End-to-end multiple-object tracking with transformer［C］// Proceedings of the 17th European Conference on Computer Vision. Springer， 2022：659-675.
［52］ MEINHARDT T， KIRILLOV A， LEAL-TAIXE L， et al. Trackformer： Multi-object tracking with transformers［C］// Proceedings of the 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition. IEEE， 2022：8834-8844.
［53］ MILAN A， ROTH S， SCHINDLER K. Continuous energy minimization for multitarget tracking［J］. IEEE Transactions on Pattern Analysis and Machine Intelligence， 2014，36（1）：58-72.
［54］ BAE S H， YOON K J. Robust online multi-object tracking based on tracklet confidence and online discriminative appearance learning［C］// Proceedings of the 2014 IEEE Conference on Computer Vision and Pattern Recognition. IEEE， 2014：1218-1225.
［55］ SUN S J， AKHTAR N， SONG H S， et al. Deep affinity network for multiple object tracking［J］. IEEE Transactions on Pattern Analysis and Machine Intelligence， 2021，43（1）：104-119.
［56］ XIANG Y， ALAHI A， SAVARESE S. Learning to track： Online multi-object tracking by decision making［C］// Proceedings of the 2015 IEEE International Conference on Computer Vision. IEEE， 2015：4705-4713.

[1]	HE Sida, CHEN Pinghua. Intent-based Lightweight Self-Attention Network for Sequential Recommendation [J]. Computer and Modernization, 2024, 0(12): 1-9.
[2]	ZHAO Chenyang, XUE Tao, LIU Junhua. Fashion Clothing Pattern Generation Based on Improved Stable Diffusion [J]. Computer and Modernization, 2024, 0(12): 15-23.
[3]	HUANG Tingpei1, MA Lubiao1, LI Shibao2, LIU Jianhang1. Gesture Recognition Method Based on WiFi and Prototypical Network [J]. Computer and Modernization, 2024, 0(12): 34-39.
[4]	ZHANG Xiaodong1, BAI Guangzhi1, LI Min1, LI Haoyang2. Oil and Gas Well Production Prediction Model Based on Empirical Wavelet Transform [J]. Computer and Modernization, 2024, 0(12): 53-58.
[5]	WANG Yanyuan, MAO Zhengchong. Detection and Recognition Algorithms for Chinese and English Scene Text Images [J]. Computer and Modernization, 2024, 0(12): 84-90.
[6]	LI Junchao1, YOU Fei1, ZHANG Chao2, SU Lele2, GONG Yan2 . BiLSTM-Attention Prediction Model and Error Analysis #br# Based on Novel Multi-objective Coati Optimization Algorithm [J]. Computer and Modernization, 2024, 0(11): 70-76.
[7]	ZHANG Yu1, 2, LI Jing1, 2, MA Ming1, 2, WANG Zhongxiang1, 2, SUN Yan1, 2. YOLOLW: A Novel Lightweight Object Detection Model [J]. Computer and Modernization, 2024, 0(11): 91-98.
[8]	QI Xian, LIU Daming, CHANG Jiaxin. Multi-view 3D Reconstruction Based on Improved Self-attention Mechanism [J]. Computer and Modernization, 2024, 0(11): 106-112.
[9]	YANG Jun1, HU Wei1, ZHU Wenfu2. Visual SLAM Loop Closure Detection Algorithm Based on Improved MobileNetV3 [J]. Computer and Modernization, 2024, 0(10): 21-26.
[10]	WEI Xuecheng1, JIANG Lingyun1, LI Yan2, HE Fei2. Improved Roadside Monocular View Small Target Detection Algorithm Based on YOLOv5 [J]. Computer and Modernization, 2024, 0(10): 27-34.
[11]	DU Mengjun1, LI Ang1, TONG Jun1, QIAN Jin1, KANG Kai1, WANG Ruoding1, JIN Wenxing2. Power Information Data Fusion Model Based on Improved Extreme Learning Algorithm [J]. Computer and Modernization, 2024, 0(10): 61-64.
[12]	YANG Shijun1, DI Guangyi1, GAO Jun1, CHEN Jianfei1, WANG Yaokun1, JI Xiaohan2. Sentiment Consistency Detection Based on Cross Modal Attention Fusion and#br# Information Perception [J]. Computer and Modernization, 2024, 0(10): 113-119.
[13]	HOU Congying, YANG Wengqing, WANG Zhao, CHENG Cong. Speech Enhancement Based on Time-frequency Self-attention Residual Temporal#br# Convolutional Networks [J]. Computer and Modernization, 2024, 0(09): 20-24.
[14]	ZHANG Ze1, ZHANG Jianquan2, 3, ZHOU Guopeng2, 3. Camera Module Defect Detection Based on Improved YOLOv8s [J]. Computer and Modernization, 2024, 0(09): 107-113.
[15]	ZHENG Shangpo1, CHEN Defu1, LI Jianli2, LIN Guoxian2, WANG Xingping3. Pedestrian Tracking Algorithm Based on Improved YOLOv5s and DeepSORT [J]. Computer and Modernization, 2024, 0(08): 54-58.