Multi-view IM-NET for Fine Reconstruction of 3D Object

doi:10.3969/j.issn.1006-2475.2025.12.007

Abstract

Abstract: Abstract: Implicit net (IM-NET) converts the task of reconstructing three-dimensional (3D) objects into a classification issue concerning whether sampled points in space are on or inside the object’s surface, effectively conserving computational and storage resources. However, IM-NET can only reconstruct 3D objects from a single view, and the limited amount of target information available in a single view, especially the missing information from occluded parts, leads to insufficient reconstruction accuracy. This paper extends IM-NET to adapt multi-view inputs by employing an attention module to fuse extracted features, thereby obtaining more complete target features and enhancing 3D reconstruction accuracy. Considering that the target surface generated by the Marching Cubes algorithm is not smooth enough when converting implicit expression into explicit mesh representation, this paper utilizes a mesh refinement algorithm to refine the reconstructed targets iteratively, achieving the refined reconstruction. Experimental results on the ShapNet dataset indicate that compared with single-view IM-NET and other multi-view reconstruction methods, the proposed multi-view IM-NET reconstructs 3D targets more complete and smooth, and the average intersection and union ratios of targets are significantly improved. Additionally, visualization effects show that the refined targets have richer details and smoother surfaces than those without refinement.

Key words: Key words: deep learning, 3D reconstruction, IM-NET, multi-view

CLC Number:

中图分类号:TP391

LIU Jianlong1, CEN Ying2, XU Bin3, JIAO Xuan4. Multi-view IM-NET for Fine Reconstruction of 3D Object[J]. Computer and Modernization, 2025, 0(12): 46-53.

References

[1] RICHTER S R, ALHAIJA H A, KOLTUN V. Enhancing photorealism enhancement[J]. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2023,45(2):1700-1715.
[2] XIANG L R, GAI J Y, BAO Y, et al. Field-based robotic leaf angle detection and characterization of maize plants using stereo vision and deep convolutional neural networks[J]. Journal of Field Robotics, 2023,40(5):1034-1053.
[3] DING C L, DAI Y F, FENG X M, et al. Stereo vision SLAM-based 3D reconstruction on UAV development platforms[J]. Journal of Electronic Imaging, 2023,32(1). DOI:
10.1117/1.JEI.32.013041.
[4] SUMETHEEPRASIT B, ROSALES MARTINEZ R, PAUL H, et al. Variable baseline and flexible configuration stereo vision using two aerial robots[J]. Sensors, 2023,23(3). DOI: 10.3390/S23031134.
[5] PETRAKIS G, ANTONOPOULOS A, TRIPOLITSIOTIS A, et al. Precision mapping through the stereo vision and geometric transformations in unknown environments[J]. Earth Science Informatics, 2023,16(2):1849-1865.
[6] 杨航,陈瑞,安仕鹏,等. 深度学习背景下的图像三维重建技术进展综述[J]. 中国图象图形学报, 2023,28(8):2396-2409.
[7] CHOY C B, XU D F, GWAK J Y, et al. 3D-R2N2: A unified approach for single and multi-view 3D object reconstruction[C]// Proceedings of the 14th European Conference on Computer Vision(ECCV 2016). Springer, 2016:628-644.
[8] TATARCHENKO M, DOSOVITSKIY A, BROX T. Octree generating networks:Efficient convolutional architectures for high-resolution 3D outputs[C]// Proceedings of 2017 IEEE International Conference on Computer Vision. IEEE, 2017:2088-2096.
[9] FAN H Q, SU H, GUIBAS L J. A point set generation network for 3D object reconstruction from a single image[C]// 2017 IEEE/CVF Conference on Computer Vision and Pattern Recognition(CVPR). IEEE, 2017:605-613.
[10] MELAS-KYRIAZI L, RUPPRECHT C, VEDALDI A. PC2:Projection-conditioned point cloud diffusion for single-image 3D reconstruction[C]// 2023 IEEE/CVF Conference on Computer Vision and Pattern Recognition(CVPR). IEEE, 2023:12923-12932.
[11] GROUEIX T, FISHER M, KIM V G, et al. A papier-mâché approach to learning 3D surface generation[C]// 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition(CVPR). IEEE, 2018:216-224.
[12] WANG N Y, ZHANG Y D, LI Z W, et al. Pixel2Mesh: Generating 3D mesh models from single RGB images[C]// Proceedings of the 2018 European Conference on Computer Vision(ECCV). Springer, 2018:55-71.
[13] WEN C, ZHANG Y D, LI Z W, et al. Pixel2Mesh++: Multi-view 3D mesh generation via deformation[C]// Proceedings of the 2019 IEEE/CVF International Conference on Computer Vision. IEEE, 2019:1042-1051.
[14] LIN K, WANG L J, LIU Z C. End-to-end human pose and mesh reconstruction with Transformers[C]// 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition(CVPR). IEEE, 2021:1954-1963.
[15] CHEN Z Q, ZHANG H. Learning implicit fields for generative shape modeling[C]// 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition(CVPR). IEEE, 2019:5939-5948.
[16] MESCHEDER L, OECHSLE M, NIEMEYER M, et al. Occupancy networks: Learning 3D reconstruction in function space[C]// 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition(CVPR). IEEE, 2019:4460-4470.
[17] XU Q G, WANG W Y, CEYLAN D, et al. DISN: Deep implicit surface network for high-quality single-view 3D reconstruction[J]. arXiv preprint arXiv:1905.10711, 2019.
[18] MILDENHALL B, SRINIVASAN P P, TANCIK M, et al. NeRF: Representing scenes as neural radiance fields for view synthesis[J]. Communications of the ACM, 2021, 65(1):99-106.
[19] YANG Z Y, GAO X Y, ZHOU W, et al. Deformable 3D gaussians for high-fidelity monocular dynamic scene reconstruction[C]// 2024 IEEE/CVF Conference on Computer Vision and Pattern Recognition(CVPR). IEEE, 2024:20331-20341.
[20] SAITO S, HUANG Z, NATSUME R, et al. PIFu: Pixel-aligned implicit function for high-resolution clothed human digitization[C]// Proceedings of the 2019 IEEE/CVF International Conference on Computer Vision. IEEE, 2019:2304-2314.
[21] SAITO S, SIMON T, SARAGIH J, et al. PIFuHD: Multi-level pixel-aligned implicit function for high-resolution 3D human digitization[C]// 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition(CVPR). IEEE, 2020:84-93.
[22] PENG S Y, NIEMEYER M, MESCHEDER L, et al. Convolutional occupancy networks[C]// Proceedings of the 2020 European Conference on Computer Vision(ECCV). Springer, 2020:523-540.
[23] TANG J P, LEI J B, XU D, et al. SA-ConvONet: Sign-agnostic optimization of convolutional occupancy networks[C]// Proceedings of the 2021 IEEE/CVF International Conference on Computer Vision. IEEE, 2021:6504-6513.
[24] BOULCH A, MARLET R. POCO: Point convolution for surface reconstruction[C]// 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition(CVPR). IEEE, 2022:6302-6314.
[25] LORENSEN W E, CLINE H E. Marching Cubes: A high resolution 3D surface construction algorithm[J]. ACM SIGGRAPH Computer Graphics, 1987,21(4):163-169.
[26] HE K M, ZHANG X Y, REN S Q, et al. Deep residual learning for image recognition[C]// 2016 IEEE/CVF Conference on Computer Vision and Pattern Recognition(CVPR). IEEE, 2016:770-778.
[27] YANG B, WANG S, MARKHAM A, et al. Robust attentional aggregation of deep feature sets for multi-view 3D reconstruction[J]. International Journal of Computer Vision, 2020,128(1):53-73.
[28] CHANG A X, FUNKHOUSER T, GUIBAS L, et al. ShapeNet:An information-rich 3D model repository[J]. arXiv preprint arXiv:1512.03012, 2015.