A Real-time Video Stitching System in Large Parallax Scene

doi:10.3969/j.issn.1006-2475.2020.04.004

Abstract

Abstract: With the development of virtual reality technology and live video technology, panoramic video live broadcast has received extensive attention. Traditional panoramic products are prone to image distortion when dealing with large parallax scenes, and they are difficult to balance real-time requirements. In order to solve these problems, a two-way video splicing system is designed. Firstly, a mosaic background model is established. Then, the image registration is completed by combining the ORB (Oriented FAST and Rotated BRIEF) with APAP (As-Projective-As-Possible) based on moving DLT (Direct Linear Transformation) algorithm. A method based on minimum energy detection is improved to find the optimal seam to avoid ghosting and misalignment caused by moving foreground. Finally, by splicing the parameter index table obtained in the stage of the model calculation, the overlapping regions are fused to complete the real-time splicing of video frames. The experimental results show that the proposed system can handle large parallax scenes and has good real-time stitching effect.

Key words: video stitching, APAP, large parallax, optimal seam

CLC Number:

TP393.4

YANG Dan1,2, CHEN Jun1,2, ZHU Xiao-yong1. A Real-time Video Stitching System in Large Parallax Scene[J]. Computer and Modernization, doi: 10.3969/j.issn.1006-2475.2020.04.004.

References

［1］江铁，朱桂斌，孙奥. 全景图像拼接技术研究现状综述［J］. 重庆工商大学学报(自然科学版), 2012,29(12):60-65.
［2］罗莹,宋利,解蓉,等. 全景媒体的系统架构研究综述［J］. 电信科学, 2018,34(2):88-98.
［3］ EL-SABAN M, IZZ M, KAHEEL A. Fast stitching of videos captured from freely moving devices by exploiting temporal redundancy［C］// 2010 IEEE International Conference on Image Processing. 2010:1193-1196.
［4］ XU W. Panoramic Video Stitching［D］. USA：University of Colorado at Boulde, 2012.
［5］ JIANG W, GU J W. Video stitching with spatial-temporal content-preserving warping［C］// 2015 IEEE Conference on Computer Vision and Pattern Recognition. 2015:42-48.
［6］薛佳乐,赵萌,张哲,等. 针对大视差图像拼接的显性子平面配准［J］. 中国图象图形学报, 2018,23(3):323-332.
［7］ GAO J H, KIM S J, BROWN M S. Constructing image panoramas using dual-homography warping［C］// 2011 IEEE Conference on Computer Vision and Pattern Recognition. 2011:49-56.
［8］ LIN W Y, LIU S, MATSUSHITA Y, et al. Smoothly varying affine stitching［C］// 2011 IEEE Conference on Computer Vision and Pattern Recognition. 2011:345-352.
［9］ LI J, XU W, ZHANG J G, et al. Efficient video stitching based on fast structure deformation［J］. IEEE Transactions on Cybernetics, 2015,45(12):2707-2719.
［10］HUANG K C, CHIEN P Y, CHIEN C A, et al. A 360-degree panoramic video system design［C］// Technical Papers of 2014 International Symposium on VLSI Design, Automation and Test. 2014:1-4.
［11］GUO H, LIU S C, HE T, et al. Joint video stitching and stabilization from moving cameras［J］. IEEE Transactions on Image Processing, 2016(99):1.
［12］HASHMI M F, PAL R, SAXENA R, et al. A new approach for real time object detection and tracking on high resolution and multi-camera surveillance videos using GPU［J］. Journal of Central South University, 2016,23(1):130-144.
［13］RUBLEE E, RABAUD V, KONOLIGE K, et al. ORB: An efficient alternative to SIFT or SURF［C］// 2011 International Conference on Computer Vision. 2011:2564-2571.
［14］ZARAGOZA J, CHIN T J, TRAN Q H, et al. As-projective-as-possible image stitching with moving DLT［C］// Proceedings of 2013 IEEE Conference on Computer Vision and Pattern Recognition. 2013:2339-2346.
［15］刘畅,金立左,费树岷,等. 固定多摄像头的视频拼接技术［J］. 数据采集与处理, 2014,29(1):126-133.
［16］LOWE D G. Object recognition from local scale-invariant features［C］// Proceedings of the 7th IEEE International Conference on Computer Vision. 1999:1150-1157.
［17］BAY H, TUYTELAARS T, GOOL L V. SURF: Speeded up robust features［C］// Proceedings of the 9th European Conference on Computer Vision. 2006:404-417.
［18］ROSSI M M, SILVATTI A P, DIAS F A S, et al. Improved accuracy in 3D analysis using DLT after lens distortion correction［J］. Computer Methods in Biomechanics and Biomedical Engineering, 2015,18(9):993-1002.
［19］史金霞,王铮. 一种拼接缝消除方法［J］. 现代电子技术, 2005,28(13):116-117.
［20］陈淑清. 一种改进的图像无缝拼接算法［J］. 武夷学院学报, 2016,35(6):65-69.
［21］王嶺,周军. 图像美感质量评价方法研究［J］. 网络新媒体技术, 2018,7(3):19-24.
［22］CANNY J. A computational approach to edge detection［J］. IEEE Transactions on Pattern Analysis and Machine Intelligence, 1986,8(6):679-698.
［23］杨艳春,李娇,王阳萍. 图像融合质量评价方法研究综述［J］. 计算机科学与探索, 2018,12(7):1021-1035.
［24］朱炼,孙枫,夏芳莉,等. 图像融合研究综述［J］. 传感器与微系统, 2014,33(2):14-18.
［25］CHANG C H, SATO Y, CHUANG Y. Shape-preserving half-projective warps for image stitching［C］// 2014 IEEE Conference on Computer Vision and Pattern Recognition. 2014:3254-3261.
［26］PAN X, WANG G. Parallax-tolerant image stitching based on mesh optimization［C］// 2017 IEEE 2nd Advanced Information Technology, Electronic and Automation Control Conference. 2017:414-420.

[1]	LI Si-jie, TANG Qing-shan, GAO Ying-hua. Parallax Image Stitching Algorithm Based on GMS and Improved Optimal Seam [J]. Computer and Modernization, 2022, 0(12): 95-101.
[2]	MO Wei, TANG Qing-shan, HUANG Tao. A High Real-time Video Fusion Algorithm Based on Multi Direction Perception [J]. Computer and Modernization, 2021, 0(10): 81-87.