Optical Flow Estimation Based on Inverse Residual Attention

doi:10.3969/j.issn.1006-2475.2024.02.010

Abstract

Abstract: Abstract: Optical flow estimation is a basic task of video understanding and analysis. Many existing methods directly take occlusion as the outer point and eliminate it， so as to improve the ability of the model to calculate the optical flow， but it is also easy to cause the image gray discontinuity， leading to the failure of optical flow estimation. In addition， the problem of large displacement caused by high speed motion of objects has always been a difficulty in optical flow estimation. In order to solve the above problems， this paper proposes a generative adversarial learning framework based on reverse residual attention （FlowTranGAN， FTGAN） for optical flow estimation. The proposed framework enhances the spatial information of features by designing a reverse residual attention module to improve the matching degree between pixels. Besides， we use a discriminator based on U-Net to constrain the generator to reduce the error and discontinuity of optical flow estimation， and improve the generalization ability of the model. Experiment results on the KITTI-2015 dataset and MPI-Sintel dataset demonstrate the effectiveness and superiority of the proposed FTGAN.

Key words: Key words: optical flow estimation, reverse residual attention, generative adversarial learning, supervised learning

CLC Number:

TP391.41

LIANG Jian-ye1, CHEN Jun-hong1, FANG Gui-biao1, WU Xing-cai2, LIU Wen-yin1. Optical Flow Estimation Based on Inverse Residual Attention[J]. Computer and Modernization, 2024, 0(02): 64-68.

References

［1］ VIHLMAN M， VISALA A. Optical flow in deep visual tracking［C］// Proceedings of the 2020 AAAI Conference on Artificial Intelligence. 2020，34（7）:12112-12119.
［2］ XU R， LI X X， ZHOU B L， et al. Deep flow-guided video inpainting［C］// 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition （CVPR）. 2019:3718-3727.
［3］ BUTLER D J， WULFF J， STANLEY G B， et al. A naturalistic open source movie for optical flow evaluation［C］// Computer Vision – ECCV 2012. 2012:611-625.
［4］ HORN B K P， SCHUNCK B G. Determining optical flow［J］. Artificial Intelligence， 1981，17（1-3）:185-203.
［5］ CHEN Q F， KOLTUN V. Full flow: Optical flow estimation by global optimization over regular grids［C］// 2016 IEEE Conference on Computer Vision and Pattern Recognition （CVPR）. 2016:4706-4714.
［6］ ZACH， POCK， BISCHOF. A duality based approach for realtime TV-L 1 optical flow［C］// Pattern Recognition. 2007:214-223.
［7］ TEED Z， DENG J. RAFT: Recurrent all-pairs field transforms for optical flow［C］// Computer Vision-ECCV 2020. 2020:402-419.
［8］ SUI X C， LI S H， GENG X， et al. CRAFT: Cross-attentional flow transformer for robust optical flow［C］// 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition （CVPR）. 2022:17581-17590.
［9］ PARK N， KIM S. How do vision transformers work?［J］. arXiv preprint arXiv:2202.06709， 2022.
［10］ CHO K Y H Y， VAN M B， GULCEHRE C， et al. Learning phrase representations using RNN encoder-decoder for statistical machine translation［J］. arXiv preprint arXiv:1406.1078， 2014.
［11］ SCHÖNFELD E， SCHIELE B， KHOREVA A. A U-Net based discriminator for generative adversarial networks［C］// 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition （CVPR）. 2020:8204-8213.
［12］ MAYER N， ILG E， HAUSSER P， et al. A large dataset to train convolutional networks for disparity， optical flow， and scene flow estimation［C］// 2016 IEEE Conference on Computer Vision and Pattern Recognition （CVPR）. 2016:4040-4048.
［13］ KONDERMANN D， NAIR R， HONAUER K， et al. The HCI benchmark suite: Stereo and flow ground truth with uncertainties for urban autonomous driving［C］// 2016 IEEE Conference on Computer Vision and Pattern Recognition Workshops （CVPRW）. 2016:19-28.

［14］ MENZE M， GEIGER A. Object scene flow for autonomous vehicles［C］// 2015 IEEE Conference on Computer Vision and Pattern Recognition （CVPR）. 2015:3061-3070.
［15］ DOSOVITSKIY A， FISCHER P， ILG E， et al. FlowNet: Learning optical flow with convolutional networks［C］// 2015 IEEE International Conference on Computer Vision （ICCV）. 2015:2758-2766.
［16］ RANJAN A， BLACK M J. Optical flow estimation using a spatial pyramid network［C］// 2017 IEEE Conference on Computer Vision and Pattern Recognition （CVPR）. 2017:2720-2729.
［17］ JIANG S H， CAMPBELL D， LU Y， et al. Learning to estimate hidden motions with global motion aggregation［C］// 2021 IEEE/CVF International Conference on Computer Vision （ICCV）. 2021:9752-9761.
［18］ ILG E， MAYER N， SAIKIA T， et al. FlowNet 2.0: Evolution of optical flow estimation with deep networks［C］// 2017 IEEE Conference on Computer Vision and Pattern Recognition （CVPR）. 2017:1647-1655.
［19］ SUN D Q， YANG X D， LIU M Y， et al. PWC-Net: CNNs for optical flow using pyramid， warping， and cost volume［C］// 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2018:8934-8943.
［20］ HUI T W， TANG X， LOY C C. LiteFlowNet: A lightweight convolutional neural network for optical flow estimation［C］// 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2018:8981-8989.
［21］ YIN Z C， DARRELL T， YU F S. Hierarchical discrete distribution decomposition for match density estimation［C］// 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition （CVPR）. 2019:6037-6046.
［22］ HUI T W， TANG X， LOY C C. A lightweight optical flow CNN—Revisiting data fidelity and regularization［J］. IEEE Transactions on Pattern Analysis and Machine Intelligence， 2021，43（8）:2555-2569.
［23］ YANG G S， RAMANAN D. Volumetric correspondence networks for optical flow［J］. Advances in neural information processing systems， 2019:1-15.
［24］ ZHAO S Y， SHENG Y L， DONG Y， et al. MaskFlownet: Asymmetric feature matching with learnable occlusion mask［C］// 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition （CVPR）. 2020:6277-6286.
［25］ ZHANG F H， WOODFORD O J， PRISACARIU V， et al. Separable flow: Learning motion cost volumes for optical flow estimation［C］// 2021 IEEE/CVF International Conference on Computer Vision （ICCV）. 2021:10787-10797.
［26］ SUN D Q， YANG X D， LIU M Y， et al. Models matter， so does training: An empirical study of CNNs for optical flow estimation［J］. IEEE Transactions on Pattern Analysis and Machine Intelligence， 2020，42（6）:1408-1423.
［27］ ZENG Y H， FU J L， CHAO H Y， et al. Learning pyramid-context encoder network for high-quality image inpainting［C］// 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition （CVPR）. 2019:1486-1494.

[1]	XU Yue-wen1, LI Ming1, LI Li2. Image Classification of COVID-19 Based on Contrast Learning MocoV2 [J]. Computer and Modernization, 2024, 0(02): 81-87.
[2]	LIN Wei. Incremental News Recommendation Method Based on Self-supervised Learning and Data Replay [J]. Computer and Modernization, 2023, 0(12): 1-6.
[3]	QIAO Shan-bao, GAO Yong-bin, HUANG Bo, YU Wen-jun. Monocular Depth and Pose Estimation Based on Conditionally Convolution and Polarized Self-attention [J]. Computer and Modernization, 2023, 0(02): 34-39.
[4]	LIU Ying-jie, LAN Hai, WEI Xian. Semi-supervised Learning Method Based on Convolution and Sparse Coding [J]. Computer and Modernization, 2022, 0(11): 9-16.
[5]	ZHANG Dong-fang, CHEN Hai-yan, YUAN Li-gang. S2R2: Semi-supervised Feature Selection Based on Analysis of Relevance and Redundancy [J]. Computer and Modernization, 2021, 0(09): 113-120.
[6]	SHAO Shi-yun, ZHOU Yu, YANG Lei, ZHONG Mao-sheng, DAI Rui, ZHAO Jia-le. An Efficient Entity Identification Method for Electric Bidding Documents Based on Conditional Random Field [J]. Computer and Modernization, 2020, 0(12): 72-77.
[7]	ZHANG Ning, HUANG Ting-pei, CAI Li-ping, LI Shi-bao. A Multi-classification Approach to Discriminate Causes of Packet Losses and Errors for High Throughput Wireless LAN Based on Convolution Neural Network [J]. Computer and Modernization, 2020, 0(06): 14-.
[8]	MA Ji-ke, YIN Fei, ZHU Yong-jin, DOU Long-long, LI Jian. A Semi-supervised Learning Method for Operating State Identification of #br# Energy Measurement Devices [J]. Computer and Modernization, 2020, 0(03): 82-.
[9]	SUN Zhen, WANG Jing-dong, MAO Tian-yi, WEI Xue-ying. Segmentation of River Based on Self-supervised Learning [J]. Computer and Modernization, 2017, 0(10): 10-14.
[10]	LI Rong, WANG Hua-jun, XU Yan-hua, MENG De-jian. A Face Expression Recognition Method Based on Fusion of Supervised #br# Super-vector Encoding and Adaptive GMM Model [J]. Computer and Modernization, 2016, 0(2): 15-20.
[11]	WU Fei, MAO Yu-guang. A Novel Fuzzy C-Means Algorithm Based on k-d Tree [J]. Computer and Modernization, 2015, 0(11): 1-5+11.
[12]	NING Bo, SONG Yan. Person Recognition in Video Based on Unsupervised Method [J]. Computer and Modernization, 2014, 0(12): 49-53.
[13]	CONG Xue-yan. Error-based Hybrid Classification Algorithm [J]. Computer and Modernization, 2014, 0(1): 100-103,125.
[14]	WU Yong-cheng. Research and Advances on Machine Learning of Multi-view Unlabeled Data [J]. Computer and Modernization, 2013, 1(3): 96-98.
[15]	ZHANG Xia-huan;LI Yu-jian;ZHANG Chen-guang. Learning from Only Positive and Unlabeled Examples for Eyebrow Image Segmentation [J]. Computer and Modernization, 2012, 1(9): 127-133.