计算机与现代化

• 数据库与数据挖掘 • 上一篇    下一篇

基于微服务的石油大数据挖掘平台

  

  1. (1.中国石油大学(华东)计算机与通信工程学院,山东青岛266580;2.北京科技大学计算机与通信工程学院,北京100083;
    3.中国传媒大学信息与通信工程学院,北京100024)
  • 收稿日期:2018-11-05 出版日期:2019-05-14 发布日期:2019-05-14
  • 作者简介:郭燚(1990-),女,山东滨州人,硕士研究生,研究方向:大数据处理,数据挖掘,E-mail: guoyi909@foxmail.com; 张卫山(1970-),男,上海人,教授,硕士生导师,博士,研究方向:人工智能及其应用,大数据处理,数据挖掘; 徐亮(1991-),男,山东济南人,博士研究生,研究方向:大数据处理,数据挖掘; 翟佳(1989-),女,河北石家庄人,博士研究生,研究方向:散射与逆散射,机器学习。
  • 基金资助:
    国家自然科学基金资助项目(61309024); 山东省重点科研项目(2017GGX10140)

A Micro-service-based Oil Big Data Mining Platform

  1. (1. College of Computer and Communication Engineering, China University of Pertroleum, Qingdao 266580, China;
    2. College of Computer and Communication Engineering, Beijing University of Science and Technology, Beijing 100083, China;
    3. College of Information and Communication Engineering, Communication University of China, Beijing 100024, China)
  • Received:2018-11-05 Online:2019-05-14 Published:2019-05-14

摘要: 为推进大数据技术在油田领域的快速融合和应用,提出一种覆盖大数据处理整个生命周期的多功能大数据处理平台。平台融合各类大数据分析框架和机器学习框架,设计面向油田领域,能够支持实时和离线处理的数据挖掘功能。基于Docker容器封装各类计算框架和算法服务,并基于Kubernetes框架完成容器的编排与调度。在系统的架构方式上采用基于微服务的架构方式,将不同技术栈的应用独立分解为单个服务模块,以此来保证业务系统服务的可靠性、可扩展性。这使得企业数据分析人员能够专注于业务数据分析问题,而不必花费大量时间学习框架部署和其他大型数据挖掘技术细节。

关键词: 微服务, 大数据挖掘; Docker

Abstract:  In order to promote the rapid fusion and application of big data technology in oilfield, a multi-functional big data processing platform covering the whole life cycle of big data processing is proposed. The platform combines various big data analysis frameworks and machine learning frameworks to design data mining functions that can support real-time and off-line processing in the oilfield. Based on Docker containers, it encapsulates all kinds of computing frameworks and algorithms, and  based on Kubernetes framework, it completes container arrangement and scheduling. In order to ensure the reliability and extensibility of the business system services, the system adopts the micro-service-based architecture, which decomposes the application of different technology stacks into a single service module independently. This allows enterprise data analysts to focus on business data analysis issues without spending a lot of time learning the details of framework deployment and other big data mining technologies.

Key words: micro-services, big data mining, Docker

中图分类号: