Application of Neural Rendering Based Visual Synthesis in Construction Scene

doi:10.3969/j.issn.1006-2475.2023.12.013

Abstract

Abstract: Abstract: When neural radiation field （NeRF） is applied to outdoor construction scenes， due to the difficulty in capturing the lighting in outdoor scenes and the large difference between the foreground and background ranges of construction scenes， blurring and artifacts will appear in novel views. Through analysis， an improved visual synthesis method is proposed. First， camera parameters are obtained from RGB images through SFM algorithm to represent outdoor construction scenes. Then， the vector generated from the pre training encoder is introduced and added to the rendering network to reduce the impact of light. Finally， the foreground and background in the image are separated for volume rendering， which improves the effect of visual synthesis. Based on the outdoor construction scene data set， compared with three methods， the results show that the proposed method improves the peak signal-to-noise ratio and structural similarity by 12.2% and 10.9% respectively compared with the best method. It appears that the novel views generated by the proposed method in outdoor construction scenes have better finesse.

Key words: Key words: NeRF, visual synthesis, construction scenario, illumination, range

CLC Number:

TP391

ZHANG Zai-cheng, LI Jian. Application of Neural Rendering Based Visual Synthesis in Construction Scene[J]. Computer and Modernization, 2023, 0(12): 76-81.

References

［1］ TEWARI A， FRIED O， THIES J， et al. State of the art on neural rendering［J］. Computer Graphics Forum， 2020，39（2）:701-727.
［2］ SHUM H Y， HE L W. Rendering with concentric mosaics［C］// Proceedings of the 26th Annual Conference on Computer Graphics and Interactive Techniques. 1999:299-306.
［3］ NIEMEYER M， MESCHEDER L， OECHSLE M， et al. Differentiable volumetric rendering: Learning implicit 3D representations without 3D supervision［C］// Proceedings of the 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2020:3501-3512.
［4］ SITZMANN V， ZOLLHOFER M， WETZSTEIN G. Scene representation networks: Continuous 3D-structure-aware neural scene representations［C］// Proceedings of the 33rd International Conference on Neural Information Processing Systems. 2019:1119-1130.
［5］ MILDENHALL B， SRINIVASAN P P， TANCIK M， et al. NeRF: Representing scenes as neural radiance fields for view synthesis［C］// Proceedings of the 2020 European Conference on Computer Vision. 2020:405-421.
［6］常远，盖孟. 基于神经辐射场的视点合成算法综述［J］. 图学学报， 2021，42（3）:376-384.
［7］ MARTIN-BRUALLA R， RADWAN N， SAJJADI M S M， et al. NeRF in the wild: Neural radiance fields for unconstrained photo collections［C］// Proceedings of the 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2021:7206-7215.
［8］ ZHANG K， RIEGLER G， SNAVELY N， et al. NeRF++: Analyzing and improving neural radiance fields［J］. arXiv preprint arXiv:2010.07492， 2020.
［9］ DAVIS A， LEVOY M， DURAND F. Unstructured light fields［J］. Computer Graphics Forum， 2012，31（2pt1）:305-314.
［10］ BROCK A， LIM T， RITCHIE J M， et al. Generative and discriminative voxel modeling with convolutional neural networks［J］. arXiv preprint arXiv:1608.04236， 2016.
［11］ ACHLIOPTAS P， DIAMANTI O， MITLIAGKAS I， et al. Learning representations and generative models for 3D point clouds［C］// Proceedings of the 35th International Conference on Machine Learning. 2018:40-49.
［12］ GROUEIX T， FISHER M， KIM V G， et al. AtlasNet: A papier-mache approach to learning 3D surface generation［J］. arXiv preprint arXiv:1802.05384， 2018.
［13］ MICHALKIEWICZ M， PONTES J K， JACK D， et al. Implicit surface representations as layers in neural networks［C］// Proceedings of the 2019 IEEE/CVF International Conference on Computer Vision. 2019:4743-4752.
［14］ HARTLEY R， ZISSERMAN A. Multiple View Geometry in Computer Vision［M］. Cambridge University Press， 2003.
［15］ HIEP V H， KERIVEN R， LABATUT P， et al. Towards high-resolution large-scale multi-view stereo［C］// Proceedings of the 2009 IEEE Conference on Computer Vision and Pattern Recognition. 2009:1430-1437.
［16］ HEDMAN P， PHILIP J， PRICE T， et al. Deep blending for free-viewpoint image-based rendering［J］. ACM Transactions on Graphics （TOG）， 2018，37（6）. DOI: 10.1145/3272127.3275084.
［17］ SCHONBERGER J L， FRAHM J M. Structure-from-motion revisited［C］// Proceedings of the 2016 IEEE Conference on Computer Vision and Pattern Recognition. 2016:4104-4113.
［18］ WAECHTER M， MOEHRLE N， GOESELE M. Let there be color! Large-scale texturing of 3D reconstructions［C］// Proceedings of the 2014 European Conference on Computer Vision. 2014:836-850.
［19］ CHEN W Z， LING H， GAO J， et al. Learning to predict 3D objects with an interpolation-based differentiable renderer［C］// Proceedings of the 2019 International Conference on Neural Information Processing Systems. 2019.
［20］ LI T M， AITTALA M， DURAND F， et al. Differentiable Monte Carlo ray tracing through edge sampling［J］. ACM Transactions on Graphics （TOG）， 2018，37（6）. DOI: 10.1145/3272127.3275109.
［21］ KUTULAKOS K N， SEITZ S M. A theory of shape by space carving［J］. International Journal of Computer Vision， 2000，38（3）:199-218.
［22］ MILDENHALL B， SRINIVASAN P P， ORTIZ-CAYON R， et al. Local light field fusion: Practical view synthesis with prescriptive sampling guidelines［J］. ACM Transactions on Graphics （TOG）， 2019，38（4）. DOI: 10.1145/3306346.3322980.
［23］ WOOD D N， AZUMA D I， ALDINGER K， et al. Surface light fields for 3D photography［C］// Proceedings of the 27th Annual Conference on Computer Graphics and Interactive Techniques. 2000:287-296.
［24］ ZHU J Y， ZHANG R， PATHAK D， et al. Toward multimodal image-to-image translation［C］// Proceedings of the 31st International Conference on Neural Information Processing Systems. 2017:465-476.
［25］ MCREYNOLDS T， BLYTHE D. Advanced Graphics Programming Techniques Using OpenGL［M］. Silicon Graphics， 1998.
［26］ ATTAL B， LING S， GOKASLAN A， et al. MatryODShka: Real-time 6DoF video view synthesis using multi-sphere images［C］// Proceedings of the 2020 European Conference on Computer Vision. 2020:441-459.

[1]	WANG Hong-jie, XU Sheng-chao, YANG Bo, MAO Ming-yang, JIANG Jin-ling. Automatic Arrangement Method of Cloud Network Security Service Chain Based on SRv6 Technology [J]. Computer and Modernization, 2024, 0(01): 1-5.
[2]	ZHANG Mei-di, YU Shun-yuan. Low-light Image Enhancement Based on Adaptive Local Gamma Correction [J]. Computer and Modernization, 2023, 0(08): 74-78.
[3]	WANG Xin, WU Jun-hui, . FPGA-based Interactive Control System for Molecular Dynamics Simulation [J]. Computer and Modernization, 2022, 0(09): 25-31.
[4]	JI Ping, GUO Ying. Underwater Localization Algorithm of Range Correction Based on Long Short-Term Memory [J]. Computer and Modernization, 2022, 0(02): 52-57.
[5]	WU Zhong-de, LUO Xiao-fang, HOU Zeng-xuan, DUAN Peng-xuan, LI Nan-nan. Visualization of Radar Detection Range Based on 3D Earth Scene Under Interference Condition [J]. Computer and Modernization, 2020, 0(10): 90-96.
[6]	GUI Yi-qi, TIAN Xing-chen. A POI Recommendation Algorithm Combined with Expert Trust [J]. Computer and Modernization, 2019, 0(12): 10-.
[7]	LIANG Jun-xiu, XU Jian-qiu. Range Pattern Match Query Based on Spatial-temporal Label Trajectories [J]. Computer and Modernization, 2018, 0(08): 79-.
[8]	REN Fei, WANG Jiarun, LI Xiaojuan, YIN Hui. Construction and Optimization Implementation of 3D Radar Detection Range Based on OSG [J]. Computer and Modernization, 2018, 0(04): 111-.
[9]	WANG Huajun, LI Rong, XU Yanhua, MENG Dejian. A Face Recognition Algorithm Under Illumination Variation Based on #br# Local Phasetexture Representation [J]. Computer and Modernization, 2015, 0(12): 84-.
[10]	YANG Yunping1,2, LI Muhua1,2, ZHOU Zhipeng1,2. Design and Implementation of Orange Planting Expert System [J]. Computer and Modernization, 2014, 0(9): 124-127.
[11]	TIAN Pei-yun;ZHENG Bi-bo;CHEN Wei-qing. Research on Texture Classification Based on Color Local Angle Binary Patterns [J]. Computer and Modernization, 2013, 1(4): 86-89,9.
[12]	KONG Yong;DU Xin-yu. ]Research on TENA Object Model [J]. Computer and Modernization, 2013, 1(2): 85-89.
[13]	LYU Qiong-shuai, XIONG Shu-feng . null [J]. Computer and Modernization, 2013, 12(12): 23-26.
[14]	LI Nan. A Method for Face Recognition Under Variable Illumination Conditions [J]. Computer and Modernization, 2012, 203(7): 104-106,.
[15]	PAN Hao;YANG Yi-min. Color Recognition Method Based on Illumination Changed Compensation [J]. Computer and Modernization, 2012, 1(200): 201-04.