Computer and Modernization ›› 2022, Vol. 0 ›› Issue (01): 41-53.
Previous Articles Next Articles
Online:
2022-01-24
Published:
2022-01-24
CAO Yu, LI Xiao-hui, LIU Zhong-lin, JIA He, FEI Zhi-wei. Review of Big Data Workflow Orchestration and Management System in Cloud Environment[J]. Computer and Modernization, 2022, 0(01): 41-53.
[1] LIU J, PACITTI E, VALDURIEZ P, et al. A survey of data-intensive scientific workflow management[J]. Journal of Grid Computing, 2015,13(4):457-493. [2] PANDEY S, BUYYA R. A survey of scheduling and management techniques for data-intensive application workflows[M]// Data Intensive Distributed Computing: Challenges and Solutions for Large-scale Information Management. 2012:156-176. [3] 田倬璟,黄震春,张益农. 云计算环境任务调度方法研究综述[J]. 计算机工程与应用, 2021,57(2):1-11. [4] ADHIKARI M, AMGOTH T, SRIRAMA S N. A survey on scheduling strategies for workflows in cloud environment and emerging trends[J]. ACM Computing Surveys, 2020,52(4):68.1-68.36. [5] KAUR S, BAGGA P, HANS R, et al. Quality of service (QoS) aware workflow scheduling (WFS) in cloud computing: A systematic review[J]. Arabian Journal for Science and Engineering, 2019,44(4):2867-2897. [6] POOLA D, SALEHI M A, RAMAMOHANARAO K, et al. Chapter 15: A taxonomy and survey of fault-tolerant workflow management systems in cloud and distributed computing environments[M]// Software Architecture for Big Data and the Cloud. 2017:285-320. [7] LIU J, PACITTI E, VALDURIEZ P. A survey of scheduling frameworks in big data systems[J]. International Journal of Cloud Computing, 2018,7(2):103-128. [8] RANJAN R, GARG S, KHOSKBAR A R, et al. Orchestrating bigdata analysis workflows[J]. IEEE Cloud Computing, 2017,4(3):20-28. [9] BARIKA M, GARG S, ZOMAYA A Y, et al. Orchestrating big data analysis workflows in the cloud: Research challenges, survey, and future directions[J]. ACM Computing Surveys, 2020,52(5):95.1-95.41. [10]AMSTUTZ P, CRUSOE M R, TIJANIC N, et al. Common Workflow Language, v1.0[S]. 2016. [11]VAN DER AALST W M P, TER HOFSTEDE A H M. YAWL: Yet another workflow language[J]. Information Systems, 2005,30(4):245-275. [12]ADAMS M, HENSE A V, TER HOFSTEDE A H M. YAWL: An open source business process management system from science for science[J]. SoftwareX, 2020,12. DOI:10.1016/j.softx.2020.100576. [13]GWL. A Workflow Management Language Extension for GNU Guix[EB/OL]. [2021-07-23]. https://www.guixwl.org/. [14]CLOUDSLANG. Orchestration as Code[EB/OL]. [2021-07-24]. https://cloudslang-docs.readthedocs.io/en/latest/cloudslang/cloudslang_dsl_reference.html. [15]BRANDT J, BUX M, LESER U. Cuneiform: A functional language for large scale scientific data analysis[C]//2015 EDBT /ICDT Workshops. 2015:7-16. [16]BRANDT J, REISIG W, LESER U. Computation semantics of the functional scientific workflow language Cuneiform[J]. Journal of Functional Programming, 2017,27. DOI:10.1017/S0956796817000119. [17]AHMAD S G, LIEW C S, RAFIQUE M M, et al. Data-intensive workflow optimization based on application task graph partitioning in heterogeneous computing systems[C]// 2014 IEEE 4th International Conference on Big Data and Cloud Computing. 2014:129-136. [18]AHMAD S G, LIEW C S, RAFIQUE M M, et al. Optimization of data-intensive workflows in stream-based data processing models[J]. The Journal of Supercomputing, 2017,73(9):3901-3923. [19]ZHANG J H, CHEN J, ZHAN J, et al. Graph partition-based data and task co-scheduling of scientific workflow in geo-distributed datacenters[J]. Concurrency and Computation: Practice and Experience, 2019,31(24). DOI:10.1002/cpe.5245. [20]LI C L, TANG J H, MA T, et al. Load balance based workflow job scheduling algorithm in distributed cloud[J]. Journal of Network and Computer Applications, 2020,152. DOI:10.1016/j.jnca.2019.102518. [21]LI C L, ZHANG Y H, HAO Z Q, et al. An effective scheduling strategy based on hypergraph partition in geographically distributed datacenters[J]. Computer Networks, 2020,170. DOI:10.1016/j.comnet.2020.107096. [22]NIU M, CHENG B, CHEN J L. GTAA: A geo-aware task allocation approach in cloud workflow[C]// 2019 IEEE International Conference on Web Services (ICWS). 2019:449-451. [23]ADHIKARI M, AMGOTH T. An intelligent water drops-based workflow scheduling for IaaS cloud[J]. Applied Soft Computing, 2019,77:547-566. [24]ALKHANAK E N, LEE S P. A hyper-heuristic cost optimisation approach for scientific workflow scheduling in cloud computing[J]. Future Generation Computer Systems, 2018,86:480-506. [25]ZHOU X M, ZHANG G X, SUN J, et al. Minimizing cost and makespan for workflow scheduling in cloud using fuzzy dominance sort based HEFT[J]. Future Generation Computer Systems, 2019,93:278-289. [26]CHEN Z G, ZHAN Z H, LIN Y, et al. Multiobjective cloud workflow scheduling: A multiple populations ant colony system approach[J]. IEEE Transactions on Cybernetics, 2019,49(8):2912-2926. [27]CHOUDHARY A, GUPTA I, SINGH V, et al. A GSA based hybrid algorithm for bi-objective workflow scheduling in cloud computing[J]. Future Generation Computer Systems, 2018,83:14-26. [28]XIE Y, ZHU Y W, WANG Y G, et al. A novel directional and non-local-convergent particle swarm optimization based workflow scheduling in cloud-edge environment[J]. Future Generation Computer Systems, 2019,97:361-378. [29]ELSHERBINY S, ELDAYDAMONY E, ALRAHMAWY M, et al. An extended intelligent water drops algorithm for workflow scheduling in cloud computing environment[J]. Egyptian Informatics Journal, 2018,19(1):33-55. [30]ANWAR N, DENG H F. A hybrid metaheuristic for multi-objective scientific workflow scheduling in a cloud environment[J]. Applied Sciences, 2018,8(4). DOI:10.3390/app8040538. [31]ARABNEJAD H, BARBOSA J G. List scheduling algorithm for heterogeneous systems by an optimistic cost table[J]. IEEE Transactions on Parallel and Distributed Systems, 2014,25(3):682-694. [32]CHENG M Y, PRAYOGO D. Symbiotic organisms search: A new metaheuristic optimization algorithm[J]. Computers & Structures, 2014,139:98-112. [33]WANG Z J, ZHAN Z H, YU W J, et al. Dynamic group learning distributed particle swarm optimization for large-scale optimization and its application in cloud workflow scheduling[J]. IEEE Transactions on Cybernetics, 2020,50(6):2715-2729. [34]SINGH V, GUPTA I, JANA P K. An energy efficient algorithm for workflow scheduling in IaaS cloud[J]. Journal of Grid Computing, 2020,18(3):357-376. [35]MANASRAH A M, BA ALI H. Workflow scheduling using hybrid GA-PSO algorithm in cloud computing[J]. Wireless Communications and Mobile Computing, 2018. DOI: 10.1155/2018/1934784. [36]SAEEDI S, KHORSAND R, BIDGOLI S G, et al. Improved many-objective particle swarm optimization algorithm for scientific workflow scheduling in cloud computing[J]. Computers & Industrial Engineering, 2020,147.DOI:10.1016/j.cie.2020.106649. [37]ISMAYILOV G, TOPCUOGLU H R. Neural network based multi-objective evolutionary algorithm for dynamic workflow scheduling in cloud computing[J]. Future Generation Computer Systems, 2020,102:307-322. [38]DEB K, PRATAP A, AGARWAL S, et al. A fast and elitist multiobjective genetic algorithm: NSGA-II[J]. IEEE Transactions on Evolutionary Computation, 2002,6(2):182-197. [39]WU H, HUA X Y, LI Z, et al. Resource and instance hour minimization for deadline constrained DAG applications using computer clouds[J]. IEEE Transactions on Parallel and Distributed Systems, 2016,27(3):885-899. [40]THENNARASU S R, SELVAM M, SRIHARI K. A new whale optimizer for workflow scheduling in cloud computing environment[J]. Journal of Ambient Intelligence and Humanized Computing, 2021,12(3):3807-3814. [41]KASHLEV A, LU S Y. A system architecture for running big data workflows in the cloud[C]// 2014 IEEE International Conference on Services Computing. 2014:51-58. [42]KASHLEV A, LU S Y, MOHAN A. Big data workflows: A reference architecture and the dataview system[J]. Services Transactions on Big Data (STBD), 2017,4(1):1-19. [43]DESSALK Y D, NIKOLOV N, MATSKIN M, et al. Scalable execution of big data workflows using software containers[C]// The 12th International Conference on Management of Digital EcoSystems. 2020:76-83. [44]BARIKA M, GARG S, ZOMAYA A, et al. Online scheduling technique to handle data velocity changes in stream workflows[J]. IEEE Transactions on Parallel and Distributed Systems, 2021,32(8):2115-2130. [45]BARIKA M, GARG S, CHAN A, et al. Scheduling algorithms for efficient execution of stream workflow applications in multicloud environments[J]. IEEE Transactions on Services Computing, 2019. DOI: 10.1109/TSC.2019.2963382.〖HJ1.42mm〗 [46]ABAZARI F, ANALOUI M, TAKABI H, et al. MOWS: Multi-objective workflow scheduling in cloud computing based on heuristic algorithm[J]. Simulation Modelling Practice and Theory, 2019,93:119-132. [47]WANG Y W, GUO Y F, GUO Z H, et al. CLOSURE: A cloud scientific workflow scheduling algorithm based on attack-defense game model[J]. Future Generation Computer Systems, 2020,111:460-474. [48]XU X L, MO R C, DAI F, et al. Dynamic resource provisioning with fault tolerance for data-intensive meteorological workflows in cloud[J]. IEEE Transactions on Industrial Informatics, 2020,16(9):6172-6181. [49]KHALDI M, REBBAH M, MEFTAH B, et al. Fault tolerance for a scientific workflow system in a cloud computing environment[J]. International Journal of Computers and Applications, 2020,42(7):705-714. [50]XIE G Q, ZENG G, LI R F, et al. Quantitative fault-tolerance for reliable workflows on heterogeneous IaaS clouds[J]. IEEE Transactions on Cloud Computing, 2020,8(4):1223-1236. [51]YAO G S, DING Y S, HAO K R. Using imbalance characteristic for fault-tolerant workflow scheduling in cloud systems[J]. IEEE Transactions on Parallel and Distributed Systems, 2017,28(12):3671-3683. [52]DING Y S, YAO G S, HAO K R. Fault-tolerant elastic scheduling algorithm for workflow in cloud systems[J]. Information Sciences, 2017,393:47-65. [53]ALAEI M, KHORSAND R, RAMEZANPOUR M. An adaptive fault detector strategy for scientific workflow scheduling based on improved differential evolution algorithm in cloud[J]. Applied Soft Computing, 2021,99. DOI: 10.1016/j.asoc.2020.106895. [54]CONTAINERS AT GOOGLE[EB/OL]. [2021-07-24]. https://cloud.google.com/containers/. [55]PINEDA-MORALES L, COSTAN A, ANTONIU G. Towards multi-site metadata management for geographically distributed cloud workflows[C]// 2015 IEEE International Conference on Cluster Computing. 2015:294-303. [56]PINEDA-MORALES L, LIU J, COSTAN A, et al. Managing hot metadata for scientific workflows on multisite clouds[C]// 2016 IEEE International Conference on Big Data. 2016:390-397. [57]LIU J, PINEDA-MORALES L, PACITTI E, et al. Efficient scheduling of scientific workflows using hot metadata in a multisite cloud[J]. IEEE Transactions on Knowledge and Data Engineering, 2019,31(10):1940-1953. [58]LIU X F, ZHAN Z H, ZHANG J. Neural network for change direction prediction in dynamic optimization[J]. IEEE Access, 2018,6:72649-72662. [59]JIANG M, HU W Z, QIU L M, et al. Solving dynamic multi-objective optimization problems via support vector machine[C]// 2018 10th International Conference on Advanced Computational Intelligence. 2018:819-824. |
[1] | QIU Ling1, 2, SONG Zhi1, 2, LYU Shuang1, 2, YANG Xue1, 2. Application of Data Synchronization Technology in External Services of Meteorological Big Data Cloud Platform [J]. Computer and Modernization, 2024, 0(07): 76-81. |
[2] | XIONG Qing-zhi1, LI Xiang1, 2, PENG Fang-wei1, JIN An-an1. A Data-Driven Intelligent Analysis Platform for Ion Source Data [J]. Computer and Modernization, 2024, 0(02): 121-126. |
[3] | HE Yu-peng, TAO Yong, WANG Bing-heng, ZHAO Ying-nan. Research Status and Prospect of Edge Computing in Smart Distribution Network [J]. Computer and Modernization, 2023, 0(08): 87-92. |
[4] | YANG Bo, XU Sheng-chao. Security Protection Method of Cloud Network Special Line Scenarios Based on SRv6#br# Service Chain [J]. Computer and Modernization, 2023, 0(08): 107-111. |
[5] | ZHOU Ming-sheng, ZHANG Wen. A Smart Park Management Platform for Multi-source Data [J]. Computer and Modernization, 2023, 0(05): 68-74. |
[6] | MAO Ming-yang, XU Sheng-chao. Communication Terminal Attack Behavior Identification Algorithm Based on Trusted Cloud Computing#br# [J]. Computer and Modernization, 2022, 0(11): 37-42. |
[7] | QIU Jin-shui, ZHUANG Hui-fu, JIN Tao. Design of Intelligent Retrieval System for Massive Plant Images [J]. Computer and Modernization, 2022, 0(10): 62-67. |
[8] | SHAN Ke, ZHANG Yi-ming, LIU Rui-xia, . Research and Design of Science and Technology Service Resource Pool Oriented to Central Plains Urban Agglomeration [J]. Computer and Modernization, 2022, 0(07): 91-96. |
[9] | HUANG AN-qi, MIAO Fang, YANG Wen-hui, NI Ya-ting, JIANG Yuan. Design of Structured Data Registration Engine Based on Data Architecture [J]. Computer and Modernization, 2022, 0(05): 82-89. |
[10] | LI Qian-shi, WANG Shu-ying, ZENG Wen-qu. Research and Application of Flexible Workflow Path Change [J]. Computer and Modernization, 2021, 0(11): 44-49. |
[11] | ZHANG Xiao-fang, FENG Hui-fang. Dynamic Optimal Path Planning Based on Trajectory Big Data [J]. Computer and Modernization, 2021, 0(11): 82-88. |
[12] | LI Ming, CHEN Ji-fu, YI Xiao-rong, LIU Shu-ming. An Environment Monitoring System for Dongting Lake Based on JFinal Framework [J]. Computer and Modernization, 2021, 0(10): 41-48. |
[13] | ZHANG Xiao-min. An Attribute-based Multi-keyword Searchable Scheme Based on Bloom Filters [J]. Computer and Modernization, 2021, 0(08): 104-111. |
[14] | DENG Bin-tao, XU Sheng-chao. A Differential Evolution K-mediods Clustering Algorithm Based on Dynamic Gemini Population [J]. Computer and Modernization, 2021, 0(07): 54-59. |
[15] | WEI Yun-dong. Intelligent Talent Recommendation Method Based on Big Data Technology [J]. Computer and Modernization, 2021, 0(07): 60-64. |
Viewed | ||||||
Full text |
|
|||||
Abstract |
|
|||||