结合局部自注意力和深度优化的多视图重建

doi:10.3969/j.issn.1006-2475.2024.05.016

摘要/Abstract

摘要：
摘要：针对多视图三维重建中存在的内存和时间消耗过大、高分辨率重建完整性差等问题，提出一种基于深度学习的多视图重建网络。网络由特征提取模块、级联的Patchmatch模块和深度图优化模块组成。首先，设计U型的特征提取模块，提取多阶段特征图，并在每个阶段引入相对位置编码的局部自注意力层，捕捉图像中的局部细节和全局上下文，提升网络特征提取性能。其次，设计深度残差网络，通过密集连接和残差结构对特征进行融合，充分利用彩色图像先验知识来约束深度图，提升深度估计的准确性。在公开数据集DTU（Technical University of Denmark）上进行测试，实验结果表明，三维重建质量到了有效的提升，与PatchmatchNet相比在完整性上提升了6.1%，在整体性上提升了2.5%，与其他的 SOTA（State-Of-The-Art）方法相比，在完整性和整体性上都得到了较大提升。

关键词: 关键词：深度学习, 三维重建, 局部自注意力, 多视图立体, 深度估计

Abstract: Abstract: To address the issues of high memory and time consumption， low completeness and fidelity of high-resolution reconstruction in multi-view 3D reconstruction， we propose a deep learning-based multi-view reconstruction network. The network consists of a feature extraction module， a cascaded Patchmatch module and a depth map optimization module. First， we design a U-shaped feature extraction module to extract multi-stage feature maps， and introduce local self-attention layers with relative position encoding at each stage， which capture the local details and global context in the images， and enhance the feature extraction performance of the network. Second， we design a deep residual network to fuse the features， and fully utilize the color image prior knowledge to constrain the depth map， and improve the accuracy of depth estimation. We test our network on the public dataset DTU （Technical University of Denmark）， and the experimental results show that our network achieves significant improvement in 3D reconstruction quality. Compared with PatchmatchNet， our network improves the completeness by 6.1% and the overall by 2.5%. Compared with other SOTA （State-Of-The-Art） methods， our network also achieves better performance in both completeness and overall.

Key words: Key words: deep learning, 3D reconstruction, local self-attention mechanism, multi-view stereo, depth estimation

中图分类号:

TP391

叶森辉, 王蕾. 结合局部自注意力和深度优化的多视图重建[J]. 计算机与现代化, 2024, 0(05): 92-98.

YE Senhui, WANG Lei. Multi-view Reconstruction with Local Self-attention and Deep Optimization[J]. Computer and Modernization, 2024, 0(05): 92-98.

参考文献

［1］鄢化彪，徐方奇，黄绿娥，等. 基于深度学习的多视图立体重建方法综述［J］. 光学精密工程， 2023，31（16）:2444-2464.
［2］杨航，陈瑞，安仕鹏，等. 深度学习背景下的图像三维重建技术进展综述［J］. 中国图象图形学报， 2023，28（8）:2396-2409.
［3］ IZADI S， KIM D， HILLIGES O， et al. Kinectfusion: Real-time 3D reconstruction and interaction using a moving depth camera［C］// Proceedings of the 24th Annual ACM Symposium on User Interface Software and Technology. ACM， 2011:559-568.
［4］ WANG J Y， ZHONG Y R， DAI Y C， et al. Deep two-view structure-from-motion revisited［C］// Proceedings of the 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition. IEEE ， 2021:8953-8962..
［5］ WEN Q R， YANG J R， YANG X， et al. PatchDCT: Patch refinement for high quality instance segmentation［J］. arXiv preprint arXiv:2302.02693， 2023.
［6］ ZHANG Y， CHEN X. CoD-Fusion: Multi-view stereo via depth map fusion: A coordinate decent approach［J］. Neurocomputing， 2016，173:1919-1930.
［7］ YAO Y， LUO Z X， LI S W， et al. MVSNet: Depth inference for unstructured multi-view stereo［C］// Proceedings of the 2018 European Conference on Computer Vision（ECCV）. Springer， 2018:767-783.
［8］ YAO Y， LUO Z X， LI S W， et al. Recurrent MVSNet for high-resolution multi-view stereo depth inference［C］// Proceedings of the 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition. IEEE， 2019:5525-5534.
［9］ GU X D， FAN Z W， ZHU S Y， et al. Cascade cost volume for high-resolution multi-view stereo and stereo matching［C］// 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition（CVPR）. IEEE， 2020:10357-10366.
［10］ WANG F J H， GALLIANI S， VOGEL C， et al. Patchmatchnet: Learned multi-view patchmatch stereo［C］// Proceedings of the 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition. IEEE， 2021:14194-14203.
［11］ RONNEBERGER O， FISCHER P， BROX T. U-Net: Convolutional networks for biomedical image segmentation［C］// Proceedings of the 18th International Conference on Medical Image Computing and Computer-Assisted Intervention（MICCAI 2015）. Springer， 2015:234-241.
［12］ WU Y X， HE K M. Group normalization［C］// Proceedings of the 2018 European Conference on Computer Vision（ECCV）. Springer， 2018:3-19.
［13］ IOFFE S， SZEGEDY C. Batch normalization: Accelerating deep network training by reducing internal covariate shift［C］// 2015 International Conference on Machine Learning. PMLR， 2015:448-456.
［14］ BELLO I， ZOPH B， VASWANI A， et al. Attention augmented convolutional networks［J］. arXiv preprint arXiv:1904.09925， 2019.
［15］ RAMACHANDRAN P， PARMAR N， VASWANI A， et al. Stand-alone self-attention in vision models［C］// 32nd Conference on Neural Information Processing Systems （NeurIPS 2019）. IEEE， 2019:68-80.
［16］ SHAW P， USZKOREIT J， VASWANI A. Self-attention with relative position representations［J］. arXiv preprint arXiv:1803.02155， 2018.
［17］ HUI T W， LOY C C， TANG X O. Depth map super-resolution by deep multi-scale guidance［C］// European Conference on Computer Vision（ECCV 2016）. Springer， 2016:353-369.
［18］ JENSEN R， DAHL A， VOGIATZIS G， et al. Large scale multi-view stereopsis evaluation［C］// 2014 IEEE Conference on Computer Vision and Pattern Recognition. IEEE， 2014:406-413.
［19］ KNAPITSCH A， PARK J， ZHOU Q Y， et al. Tanks and temples: Benchmarking large-scale scene reconstruction［J］. ACM Transactions on Graphics， 2017，36（4）:1-13.
［20］ WANG X， WANG C， LIU B， et al. Multi-view stereo in the deep learning era: A comprehensive revfiew［J］. Displays， 2021. DOI: 10.1016/j.displa.2021.102102.
［21］ CAMPBELL N D F， VOGIATZIS G， HERNáNDEZ C， et al. Using multiple hypotheses to improve depth-maps for multi-view stereo［C］// Proceedings of the 10th European Conference on Computer Vision. Springer， 2008:766-779.
［22］ FURUKAWA Y， PONCE J. Accurate， dense， and robust multiview stereopsis［J］. IEEE Transactions on Pattern Analysis and Machine Intelligence， 2009，32（8）:1362-1376.
［23］ GALLIANI S， LASINGER K， SCHINDLER K. Massively parallel multiview stereopsis by surface normal diffusion［C］// Proceedings of the 2015 IEEE International Conference on Computer Vision. IEEE， 2015:873-881.
［24］ JI M Q， GALL J， ZHENG H T， et al. SurfaceNet: An end-to-end 3D neural network for multiview stereopsis［C］// Proceedings of 2017 IEEE International Conference on Computer Vision. IEEE， 2017:2326-2334.
［25］ CHEN R， HAN S F， XU J， et al. Point-based multi-view stereo network［C］// Proceedings of the 2019 IEEE/CVF International Conference on Computer Vision. IEEE， 2019:1538-1547.
［26］ YU Z H， GAO S H. Fast-MVSNet: Sparse-to-dense multi-view stereo with learned propagation and Gauss-Newton refinement［C］// 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition（CVPR）. IEEE， 2020:11016-11025.
［27］ YANG J Y， MAO W， ALVAREZ J M， et al. Cost volume pyramid based depth inference for multi-view stereo［C］// 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition（CVPR）. IEEE， 2020:11016-11025.
［28］ SCHÖNBERGER J L， ZHENG E， FRAHM J M， et al. Pixelwise view selection for unstructured multi-view stereo［C］// Proceedings of the 14th European Conference on Computer Vision（ECCV 2016）. Springer， 2016:501-518.

[1]	祁贤, 刘大铭, 常佳鑫. 基于改进自注意力机制的多视图三维重建[J]. 计算机与现代化, 2024, 0(11): 106-112.
[2]	游嘉靖1, 2, 何月顺1, 何璘琳1, 钟海龙1, 2. 基于AHP-CNN的加密流量分类方法[J]. 计算机与现代化, 2024, 0(04): 83-87.
[3]	陈震1, 姚京辉2, 苏成悦1. 基于改进U-Net的髋关节关键点检测算法[J]. 计算机与现代化, 2024, 0(02): 15-19.
[4]	黎世达, 项剑文. 一种提高图像识别模型鲁棒性的弱化强化方法[J]. 计算机与现代化, 2023, 0(10): 70-76.
[5]	李延满, 王必恒, 赵羚焱. 基于轻量化YOLOv5的安全帽检测[J]. 计算机与现代化, 2023, 0(10): 59-64.
[6]	张亚文, 林文忠, 韩晓东. 基于组合特征点和主成分分析的点云配准算法[J]. 计算机与现代化, 2023, 0(09): 59-63.
[7]	张文元, 海琳琦, 刘英杰, 张海波. 一种基于改进多光谱的稀疏角CB-XLCT成像方法[J]. 计算机与现代化, 2022, 0(05): 96-101.
[8]	苏桃, 邵盈鑫, 李纯斌, 吴静, 常秀红, 杨帆. 基于运动恢复结构方法的植株三维重建[J]. 计算机与现代化, 2020, 0(11): 94-99.
[9]	孟兆旭，蔡超. 环境重建三维格网数据下的快速航迹规划方法[J]. 计算机与现代化, 2017, 0(11): 17-22+34.
[10]	化春键1,2，方程骏1,2，陈莹3. 基于改进Census变换的自由曲面立体匹配方法[J]. 计算机与现代化, 2015, 0(7): 44-.
[11]	韦羽棉，尚赵伟. 基于Kinect的旋转刚体三维重建方法[J]. 计算机与现代化, 2014, 0(5): 89-93，98.
[12]	董小龙;郭太良. 一种基于激光散斑去相关技术的三维重建方法[J]. 计算机与现代化, 2012, 203(7): 83-87.
[13]	党光磊;韦巍. 基于MITK的医学图像系统设计[J]. 计算机与现代化, 2011, 1(2): 8-10.
[14]	沈文超;孙涌. 不规则扁平粒状物表面平整度识别与分选方法的研究 [J]. 计算机与现代化, 2010, 1(7): 115-119.